I’ve worried for a few months that the massive increase in money spent on math research makes it harder to track the mathematical abilities of frontier models. Wish I had written this up publicly before the OAI/NS incident!

Tristan Buckmaster’s write up (insinuating that OAI likely tried to scoop his work with Levent Alpoge on NS): https://cims.nyu.edu/~tristanb/statement.pdf

Before LLMs, progress on prestige mathematics was not very economically valuable. Since reasoning models have been developed at frontier labs, high scores on mathematics benchmark are plausibly valuable for marketing (and attracting talent). Therefore, mathematics plausibly becomes a target for Silicon Valley entrepreneurs instead of (or as well as) the ordinary chalk-dusted professors. That’s quite a cultural shift. It also means that a lot more money pours into math research. Fable 5.1 estimates 40-50%, which is lower than I expected (for now).

Greater optimization pressure towards math progress may be sufficient to get many more theorems proven, EVEN WITHOUT superhuman math models. Arguably, humanity just hasn’t been going that hard on math until now (a very few autistic cloistered ~monks work on it for fun, which is charming but unlikely to be the global optimum). Frontier labs are incentivized to disrupt math, but to give as much credit as possible to their models rather than employees or external collaborators.

This suggests that mathematics will become a lower-trust, more competitive community. Over the long term and taken in isolation, this would probably weaken math progress (except that superhuman models vastly speed it up).

Math benchmarks also become less reliable, which is an instance of Goodharting. However, I expect that progress will be so fast that this doesn’t matter too much, because the models will prove vastly more theorems than all of humanity very soon.