The future of human mathematics

Cross-posted from Twitter; written for mathematicians.

About a year ago, I wrote that superhuman math AI will plausibly arrive in the late 2020s/early 2030s. It now looks like if anything, I was underestimating the pace of progress. In this post, I will try to say something about the sociological implications for mathematics.

I don't want to focus too much on the predicament of the mathematical community today; instead, I'll try to take a longer-term view, and say something about where I think human mathematics might eventually end up. For the sake of argument, I'll assume that within the next decade or so, AI systems are developed that are vastly more capable than humans at every mathematical task (including proving theorems, building theories, and explaining mathematics to humans), and that there nevertheless remains a community of human mathematicians, alive and well, and with funding to pursue mathematics as they please. What would such a community be doing?

Of course, in this hypothetical, mathematicians are not pursuing mathematics for any productive purpose; that is handled by the machines. They are pursuing mathematics for its own sake. There is ... (read more)

At ARC, we have been using "AI use levels" to make it easy to disclose whether and how AI was used to create a piece of content. They were designed with mathematics in mind, but they should be useful beyond that. I am sharing them in case others find them to be a useful standard.

• AI0 (no non-trivial AI use): no AI use at all, except perhaps in trivial or tangential ways, e.g. explaining background material, finding relevant literature, or rewriting prose at the level of individual phrases.
• AI1 (fully human digested and composed): AI had a non-trivial role in producing mathematical ideas, but these were entirely digested by a human, who wrote everything by hand (again except for trivial writing assistance). In particular, a human actively made all decisions about how to structure the write-up, how to define terminology, how to organize proofs, and so on.
• AI2 (fully human verified and edited): AI may have been heavily involved in producing both the mathematical ideas and the written content, but a human went through the content line by line, checked that it was correct and comprehensible, and made at least a minimal attempt to edit the content for clarity.
• AI3 (human–AI mixture): may con

Superhuman math AI will plausibly arrive significantly before broad automation

I think it's plausible that for several years in the late 2020s/early 2030s, we will have AI that is vastly superhuman at formal domains including math, but still underperforms humans at most white-collar jobs (and so world GDP growth remains below 10%/year, say – still enough room for AI to be extraordinarily productive compared to today).

Of course, if there were to be an intelligence explosion on that timescale, then superhuman math AI would be unsurprising. My main point is that superhuman math AI still seems plausible even disregarding feedback loops from automation of AI R&D. On the flip side, a major catastrophe and/or coordinated slowdown could prevent both superhuman math AI and broad automation. Since both of these possibilities are widely discussed elsewhere, I will disregard both AI R&D feedback loops and catastrophe for the purposes of this forecast. (I think this is a very salient possibility on the relevant timescale, but won't justify that here.)

My basic reasons for thinking vastly superhuman math AI is a serious possibility in the next 4–8 years (even absent AI R&D feedback loo... (read more)

Two facts about text watermarking that seem to be frequently misunderstood:

1. There exist watermarking schemes such that it is completely impossible for users to ever distinguish between watermarked and non-watermarked text. (A tempting but invalid argument is that no such scheme can exist because watermarking consumes entropy.)
2. The above property does not hold of watermarking schemes that labs claim to actually use (though I would still expect well-tuned versions to make essentially no practical difference to users).

When I say "completely impossible", I mean it in the everyday sense, i.e., not possible to perform using any machine that could feasibly be built, assuming the integrity of the scheme remains intact (so stealing the secret key doesn't count). I do not mean "impossible with infinite computational resources".

On my first point: watermarking does reduce the information-theoretic entropy of generated text in order to insert its cryptographic footprint. However, what matters is not information-theoretic entropy, but computational entropy (often called "pseudoentropy"): roughly, the maximum entropy of a distribution that is indistinguishable to a computationally bounded observer.... (read more)

Task duration as a Bradley–Terry score: an alternative to the constant hazard rate model

@Toby_Ord writes about the constant hazard rate model for task duration: a long task can be thought of as a sequence of many short subtasks of fixed difficulty, each of which must be completed to complete the overall task. This explains the approximately sigmoidal relationship between log(task horizon length) and the probability that a given model successfully completes the overall task.

I think this is a useful conceptual framing that explains the data reasonably well. But there is at least one alternative that explains the data about as well, which is to think of the task duration as being similar to a Bradley–Terry score, i.e., an exponential of an Elo rating.

The underlying intuition is that, in addition to having a larger number of subtasks, a longer task also has a higher probability of having a particularly hard subtask. We can crudely approximate the difficulty of a long task by the difficulty of its hardest subtask.

Concretely, consider any fixed random number distribution (e.g. uniform over [0,1]), representing the difficulty of a subtask. Assign to each task $T$ a positive intege... (read more)

Against superexponential fits to current time horizon measurements

I think is unreasonable to put non-trivial weight (e.g. > 5%) on a superexponential fit to METR's 50% time horizon measurements, or similar recently-collected measurements.

To be precise about what I am claiming and what I am not claiming:

• I am not claiming that these measurements will never exhibit a superexponential trend. In fact, I think a superexponential trend is fairly likely eventually, due to feedback loops from AI speeding up AI R&D. I am claiming that current measurements provide almost no information about such an eventuality, and naively applying a superexponential fit gives a poor forecast.
• I am not claiming that is very unlikely for the trend to be faster in the near future than in the near past. I think a good forecast would use an exponential fit, but with wide error bars on the slope of the fit. After all, there are very few datapoints, they are not independent of each other, and there is measurement noise. I am claiming that extrapolating the rate at which the trend is getting faster is unreasonable.
• My understanding is that AI 2027's forecast is heavily driven by putting substantial weight on s