People don't want to talk about positive visions of the future, because it is not timely and because it's not the pressing problem. Preventing AI doom already seems so unlikely that caring about what happens in case we succeed feels meaningless.

I agree that it seems very unlikely. But I think we still need to care about it, to some extent, even if only for psychological and strategic reasons. And I think this neglect is itself contributing to the very dynamics that make success less likely.

The Desperation Engine

Some people — or, arguably, many people — go to work on AI capabilities because they see it as kind of "the only hope."

"So what now, if we pause AI?", they ask.

The problem is that even with paused AI, the future looks grim. Institutional decay continues, aging continues, regulations, social media brain rot, autocracies on the rise, maybe also climate change. The problems that made people excited about ASI as a solution don't go away just because you stopped building ASI. And so the prospect of a pause feels, to many technically-minded people who care about the long-term trajectory of civilization, not like safety but like despair — like choosing to die slowly instead of rolling the dice.

From what I see, at least on the level of individuals, not organizations, at least implicitly, not articulated openly, this is the desperation engine that contributes the race. If people are less desperate, they will be less willing to risk everything with ASI. Consider e/accs, or at least some part of them. It's hard for me to analyze them as a whole, but it looks like at least some non-negligible part of them is not simply trolls but genuine transhumanism-pilled people, and their radical obsession with accelerating AI is a response to desperation regarding technological stagnation and the state of civilizational hopelessness and apathy.

Techno-optimism sentiment is not inherently anti-AI-pause and shouldn't be anti-AI-pause. Indeed, many pro-AI-pause people say they want AI pause precisely because they want glorious transhuman future.

Paths Through the Pause

What would a positive future actually look like in the world where we succeed at preventing the development of misaligned ASI? I can imagine at least two positive futures from there:

Path 1: Augment humans to solve alignment. Use biological enhancement — cognitive augmentation, brain-computer interfaces, genetic engineering, pharmacological interventions — to make humans smart enough and wise enough to eventually solve alignment properly, and only then build superintelligence with confidence.

Path 2: Classical transhumanism without the singularity. Just abandon the idea of an AI singularity, at least for a while, and work on classical transhumanism — life extension, disease eradication, cognitive enhancement, space exploration — assisted by weak AGIs and narrow biological AI models. Not the cosmic endgame of filling the light cone, but the nearer-term project of making human civilization dramatically better and more resilient, buying time and building the institutional and epistemic infrastructure that would eventually be needed to handle ASI safely.

There are, however, problems with both.

Path 1 is still probably risky, and no one knows how hard it is to augment humans well enough for them to reliably solve alignment. It may turn out that the gap between "enhanced human" and "the kind of intelligence needed to solve alignment" is itself vast. And there are alignment-adjacent risks in cognitive augmentation itself — you're modifying the thing that does the valuing.

Path 2 seems unlikely as an at least moderately long-term stable situation, precisely because we now see how easily superintelligences can be created. If the world continues even with an AI pause, and civilization becomes smarter, and hardware and AI software progress is not fully halted (only the frontier), the capability to build ASI will grow, and eventually if will happen, even if accidentally.

Still: do you see any other cool paths which the techno-optimist crowd would find appealing? I am seriously asking. This article is partly a call to think about this.

Why Bother Thinking About It

Does all of this sound like daydreaming? Well, it does.

I think it is still useful to have this positive mental image in front of you.

Firstly, the strategic case. It is clear that the world requires radical transformations to become functional and for technological progress to persist in a benevolent manner. While it is indeed not timely to spend significant effort right now on addressing the question of how to fix the world — because firstly we need to prevent the world from literally dying — it is timely to spend some effort on demonstrating that a better world is possible, that the problems are fixable, that there are other ways to bet on a better future than building ASI here and now.

This is, I believe, a real strategic intervention in the AI risk landscape, not just feel-good rhetoric. If the pause camp can say "here is a concrete, appealing alternative pathway to the future you want," that is a stronger position than "stop building the dangerous thing and then... we'll figure something out." The My motivation and theory of change for working in AI post on LessWrong made a closely related argument: the more we humans can make the world better right now, the more we can alleviate what might otherwise be a desperate dependency upon superintelligence to solve all of our problems — and the less compelling it will be to take unnecessary risks.

Secondly, the psychological case. Me personally, when I imagine a good future ahead, I feel (and arguably am) much more productive than when I just focus plainly on preventing AI doom while keeping the world as it is. I believe of course that just keeping the current world as it is would be better than risking the current ASI race, and yet not everyone could agree with that (among potential allies), and the motivation can definitely be increased if we are fighting not only for the current world, but also for a future better world.

Note that people may have different motivations: it may be the case that some fight the best when they have nothing to lose. But others fight better when they have something to protect. Both types of people exist, and a movement that only speaks to the first type is leaving motivation on the table. So the positive vision of the future is, for some, not a distraction from the work of preventing doom; it can be the thing that makes the work of preventing doom psychologically sustainable.

And thirdly, the planning case. If a real pause happens, then we actually need to work on these futures, and we need to have a plan for that. I agree that it sounds a bit... premature, but still.

The Gap

The narrative that we are responsible for 10^gazillion future sentient beings in galaxy superclusters is quite common in longtermist circles. But the question is: are there realistic, tangible, concretely imaginable pathways to this?

People have of course thought a lot about good futures. There is rich transhumanist literature. In the Sequences themselves, Fun Theory is a nice example.

But almost all of these pieces either come from older times and are outdated techno-scientifically, or they describe a positive future conditional on aligned ASI existing, or they simply don't address the question of how exactly we get from our specific civilizational state with all its problems and bottlenecks, which must be explicitly acknowledged, towards better futures. Fun Theory describes properties of a desirable future world, but doesn't bridge from where we are. Amodei's essays are an example of modern writings and are inspiring for some, but, even leaving alignment-level disagreements aside, they are entirely conditional on building powerful AI safely — they do not address what a good future looks like if we don't build ASI, or if we delay it significantly.

What we are missing, specifically, are positive visions for the pause scenario. Visions that are not "the status quo is fine, let's just not die" (which is motivationally weak for the transhumanist-pilled crowd) and not "aligned ASI will fix everything" (which presupposes the thing we're probably incapable of doing). Rather: "here are concrete, tangible pathways to a dramatically better civilization that do not require solving alignment first, and here is how they address the problems that make people desperate enough to gamble with ASI."

Looks like Roots of Progress does something in this avenue — working on a positive vision of progress and the future without the AI singularity.

But I think we need more versions of this, which are aware of the alignment problem and the risks, and that explicitly addresses the desperation dynamics I described above.

A More General Story

People have the need to escape the state of desperation. People miss the promise of a better world. And yes — this is a bigger story than AI doom.

AI doom revealed, to some of us (to many of us?) the scale of dysfunctionality of our civilization. But by the law of earlier failure, AI doom is only part of the story: explicitly or implicitly, we understand that a civilization that allowed the current AI situation to happen has all kinds of rather fundamental flaws, and we can't escape the feeling that we are trapped within these flaws.

This means that positive visions of the future, if they are to be taken seriously, cannot just be technological wishlists. They need to grapple with the institutional, political, and cultural failures that brought us here. A vision that says "and then we cure aging with narrow AI" without addressing why we currently can't coordinate on existential risk is not a complete vision. This is hard, and I don't claim to have the answers. But I think the question needs to be posed explicitly.

Many popular LessWrong posts have this recurring topic of desperation and need for hope. Requiem for the hopes of a pre-AI world is a veteran transhumanist reflecting on decades of watching those hopes erode. Turning 20 in the probable pre-apocalypse is about the feeling from a younger generation. And my own Requiem for a Transhuman Timeline, where I was especially moved by this comment. Let me share an excerpt from it:

I miss the innocence of anticipating the glorious future. Even calling it "transhumanist" feels strange, like a child talking about adulthood as "trans-child". It once felt inevitable, and beautiful, and I watched as it became slowly more shared.

I dearly wish the culture here would loosen their fixation on "Don't hit the doom tree!" and target something positive as an alternative. What does success look like? What vision can we call ourselves and humanity into?

There are yet still such visions. But first the collective needs to stop seeking to die. And I've lost faith that it will let its fixation go.

I miss the promise of the stars.

There is clearly a demand for this kind of thinking and writing that is not being satisfied.

One could argue of course: well, the recipe for making a pro-progress eudaimonic civilization is already written somewhere, let's say in the Sequences. Even if so, there remains the question of why no one can take and cook with this recipe. But yes, probably just rereading and reiterating already written pieces on the topic can be helpful, I think! In any case, I consider it plainly obvious that, for one reason or another, there is demand for that which is not satisfied.

What I Am and Am Not Saying

I am not suggesting the epistemic-violating trick "let's imagine it goes well, that will help us."

What I am saying is: even if we believe that success is unlikely, it is still worth thinking, to some extent, about what happens in the case of success and what we can achieve in that case, and how.

So, I encourage you to think about better futures, in case we succeed with preventing the development of misaligned ASI, because:

• It may make you more productive and psychologically resilient
• It may make you feel better (which is not nothing — burnout is a real threat to the safety community)
• It may attract more people to the pause/moratorium camp, by offering them something to move toward rather than only something to move away from
• If a real pause happens, then we actually need to work on these futures, and having thought about them in advance will matter

And I am non-rhetorically asking: what would make the pause feel not like a retreat, but like a different kind of advance?