This is a special post for quick takes by Adam Kaufman. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Suppose there are 100 tasks which we expect to be similarly hard for AIs to do, in expectation, but with high variance; some will in fact turn out to be much easier than others.
If I tell you that task X, which you think is around the 50th percentile, has been accomplished by date Y, it would be reasonable for you to expect that many (~half) of the other tasks had already been solved by that date. And if date Y was earlier than you’d have guessed for when half of the tasks had been solved, you'd update towards thinking they’d all be solved earlier than you expected.
But now suppose instead that I tell you that task X was solved first, before all the others, and this happened on date Y. In this case you need to make an update about task X being easier than you expected, and when you revise your prediction of when the rest of the tasks will be solved it must take this into account instead of treating X as 50th percentile.
I feel like when AIs do something impressive, people often make an update about timelines similar to the update they’d have made in the first scenario ("wow, the AIs can construct frontier math proofs so much sooner than I expected!"), but fail to make a corresponding update about how much easier task X turned out to be than they were expecting. I might speculate that this is because people aren’t explicitly tracking a reference class of similarly hard tasks. For example, suppose someone isn't carefully thinking about things like how much easier they expect proving certain math results to be than piloting robots. So they hear “AIs autonomously proved some impressive math results”, and instead of making the update that the math results were in fact easier to derive than they'd thought relative to building robots, they’ll instead make an overly strong update that the robots are coming sooner than they thought.
I think that one activity which might be valuable is to write down, in advance, a bunch of concrete things AIs might be able to do in the future, so that people can make predictions about which order they’ll happen in and how much longer some will take than others. With this list in mind, it will be easier to make more sane updates when, inevitably, some of the accomplishments happen sooner than they expect, because all of the things which happened later or were harder than expected will also be more salient.
A related thought I stumbled across when thinking about this:
When people try to make bets about how fast AI progress will be, they have to pick tasks which are easy to check/measure in order to resolve the bet. For example, the date by which AIs will get IMO gold. But these are also precisely the tasks which the current ML paradigm is disproportionately good at accomplishing (relative to tasks which are harder for a computer to check). Hence, these tasks tend to be solved earlier than you’d expect if you thought they were representative of a broader class of impressive things in general (including those that are harder to verify, or construct verifiable cousins of).