Around 8 years ago Dario Amodei, Jan Leike and a few others predicted that we’d probably have AGI by now. Since then, AI capabilities have progressed so fast that many people updated that the short timelines advocates were right. And yes, their expectations about the speed of AI progress were far more directionally correct than almost anyone expected. But we don’t in fact have AGI (even under relatively weak definitions, like a reliable drop-in replacement for central examples of white-collar jobs), and the world would look very different if we did.

Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they'll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.

I’ll write about this more extensively soon, but I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild. I previously noted that we’re seeing a pretty sharp divergence between the measured capabilities of models and their real-world impacts, which should suggest that something weird is going on. My sense is that most AI safety people are brushing this aside with the idea that automating ML research is all you need for superintelligence, without thinking very much about where and how the automated ML research would actually lead to atoms moving around in the physical world.

I worry about the next decade feeling like a Shepard tone: an auditory illusion of a sound that's continually going up. In such a scenario, people would continually be panicking and grasping for big levers, without any clear point where they stop and recognize that their actual expectations were miscalibrated. In my upcoming post I plan to instead argue that “there’s plenty of room at the top”—i.e. that LLMs could become superhuman at a wide range of skills (like math research, or long-term strategic planning, or running companies) without triggering a singularity. The other term I really like is Vlad Nesov’s “prosaic RSI”, which he discusses in this comment and this post.

Now, ideally we'd assign appropriate credences to each of these possibilities, and choose actions accordingly. But the kind of bandwagoning I'm seeing is often less about credences or expected utility calculations and more about an emotional stance—people go straight from "an intelligence explosion soon is plausible" to "okay, what do I do about it?" This shortform would be better if I linked to more examples of the mindset I'm talking about, but I'm too lazy to dig them up—also, many were in-person conversations. This post is the one which most directly inspired the shortform, but I don't want to index too much on that. I'd appreciate links to places where people have reasoned through how they're orienting to these ideas (either well or badly).