(Cross-posted from Twitter)
The AI safety community seems very confused about what it is forecasting timelines *to*.
RSI is already happening, so “timelines to RSI” is nonsensical. “Superhuman AI researchers” (SAR) or “top-expert-dominating AI” (TED-AI) are a bit more specific, but often get conflated with “intelligence explosion” or “superintelligence”. It is *possible* that the former will lead very quickly to the latter, but on an emotional level people seem to be treating forecasts to the former as deadlines beyond which they can’t predict or plan. (Also, even the latter terms have become too low-resolution for useful steering—e.g. I’d say we’re already in a slow-ish intelligence explosion.)
Forecasting when an AI takeover happens is even vaguer, because nobody has a compelling way of pinning down the threshold. In AI 2027, for instance, December 2027 is “the last month in which humans had any plausible chance of exercising control over their own future”, but this isn’t observable to any human. From some people’s perspectives it’s *already* very implausible today that humans will end up in control—so again we end up forecasting timelines to something that may have already happened.
This is all very bad for collective epistemics. So my requests are:
(Even these are gonna get kinda fuzzy and ill-defined as we move close to them, but they’re the best I have for now. More suggestions welcome!)
And yes, you might think “by that time humans will already have been disempowered”. But since we are so confused about what disempowerment actually looks like, we need tangible metrics like these to keep ourselves anchored. Otherwise everyone will keep throwing around numbers without really understanding their implications.
Also, one of my biggest takeaways from the last decade is that AIs can do crazy things while producing surprisingly few real-world implications. AIs can now solve Millennium problems or autonomously hack into major govts, yet still have little real-world power. So I expect that we will hit some of the crazy capabilities I listed above while humans are still broadly in charge. That’s kinda hard to picture today, but so was our current situation a decade or two ago.
Lastly: one guess for why people have bandwagoned so hard on short timelines is that it creates a sense of urgency which motivates action. Concretely, the entire AI safety community was far too scared of conflict with the AGI companies until fear of short timelines recently spurred more direct action. I suggest decoupling these, so that your willingness to get in conflicts with the AGI companies doesn’t feel dependent on your timelines.
I agree that timelines should be given to concrete events. Fundamentally I think the most interesting endpoint is more like "the end of the intelligence explosion;" we probably both agree that's much better than forecasting the "beginning" since it's already plausibly underway (and in a broader sense it's been underway for centuries or millennia). An intelligence explosion is a convenient scenario for forecasting because it would imply that many very dramatic capabilities arrive in rapid succession. In that context I think almost anything will work as a fo... (read more)