A Slow Guide to Confronting Doom
15MondSemmel
2Ruby
3MondSemmel
2Ruby
3MondSemmel
1Sam Iacono
6Ruby
3MondSemmel
1Sam Iacono
0MondSemmel
1Sam Iacono
11teradimich
11Declan Molony
10d_el_ez
9arisAlexis
2Ruby
7mattmacdermott
5Declan Molony
4Drake Morrison
3Stephen McAleese
New Comment
I personally first deeply felt the sense of "I'm doomed, I'm going to die soon" almost exactly a year ago, due to a mix of illness and AI news. It was a double-whammy of getting both my mortality, and AGI doom, for the very first time.
Re: mortality, it felt like I'd been immortal up to that point, or more accurately a-mortal or non-mortal or something. Up to this point I hadn't anticipated death happening to me as anything more than a theoretical exercise. I was 35, felt reasonably healthy, was familiar with transhumanism, had barely witnessed any deaths in the family, etc. I didn't feel like a mortal being that can die very easily, but more like some permanently existing observer watching a livestream of my life: it's easy to imagine turning off the livestream, but much harder to imagine that I, the observer, will eventually turn off.
After I felt like I'd suddenly become mortal, I experienced panic attacks for months.
Re: AGI doom: even though I've thought way less about this topic than you, I do want to challenge this part:
And doomed for no better reason than because people were incapable of not doing something.
Just as I felt non-mortal because of an anticipated transhumanist future or something, so too did it feel like the world was not doomed, until one day it was. But did the probability of doom suddenly jump to >99% in the last few years, or was the doom always the default outcome and we were just wrong to expect anything else? Was our glorious transhumanist future taken from us, or was it merely a fantasy, and the default outcome was always technological extinction?
Are we in a timeline where a few actions by key players doomed us, or was near-term doom always the default overdetermined outcome? Suppose we go back to the founding of LessWrong in 2009, or the founding of OpenAI in 2015. Would a simple change, like OpenAI not being founded, actually meaningfully change the certainty of doom, or would it have only affected the timeline by a few years? (That said, I should stress that I don't absolve anyone who dooms us in this timeline from their responsibility.)
From my standpoint now in 2025, AGI doom seems overdetermined for a number of reasons, like:
- Humans are the first species barely smart enough to take over the planet, and to industrialize, and to climb the tech ladder. We don't have dath ilan's average IQ of 180 or whatever. The arguments for AGI doom are just too complicated and time-consuming to follow, for most people. And even when people follow them, they often disagree about them.
- Our institutions and systems of governance have always been incapable of fully solving mundane problems, let alone extinction-level ones. And they're best at solving problems and disasters they can actually witness and learn from, which doesn't happen with extinction-level problems. And even when we faced a global disaster like Covid-19, our institutions didn't take anywhere sufficient steps to prevent future pandemics.
- Capitalism: our best, or even ~only, truly working coordination mechanism for deciding what the world should work on is capitalism and money, which allocates resources towards the most productive uses. This incentivizes growth and technological progress. There's no corresponding coordination mechanism for good political outcomes, incl. for preventing extinction.
- In a world where technological extinction is possible, tons of our virtues become vices:
- Freedom: we appreciate freedoms like economic freedom, political freedom, and intellectual freedom. But that also means freedom to (economically, politically, scientifically) contribute to technological extinction. Like, I would not want to live in a global tyranny, but I can at least imagine how a global tyranny could in principle prevent AGI doom, namely by severely and globally restricting many freedoms. (Conversely, without these freedoms, maybe the tyrant wouldn't learn about technological extinction in the first place.)
- Democracy: politicians care about what the voters care about. But to avert extinction you need to make that a top priority, ideally priority number 1, which it can never be: no voter has ever gone extinct, so why should they care?
- Egalitarianism: resulted in IQ denialism; if discourse around intelligence was less insane, that would help discussion of superintelligence.
- Cosmopolitanism: resulted in pro-immigration and pro-asylum policy, which in turn precipitated both a global anti-immigration and an anti-elite backlash.
- Economic growth: the more the better; results in rising living standards and makes people healthier and happier... right until the point of technological extinction.
- Technological progress: I've used a computer, and played video games, all my life. So I cheered for faster tech, faster CPUs, faster GPUs. Now the GPUs that powered my games instead speed us up towards technological extinction. Oops.
- And so on.
Yudkowsky had a glowfic story about how dath ilan prevents AGI doom, and that requires a whole bunch of things to fundamentally diverge from our world. Like a much smaller population; an average IQ beyond genius-level; fantastically competent institutions; a world government; a global conspiracy to slow down compute progress; a global conspiracy to work on AI alignment; etc.
I can imagine such a world to not blow itself up. But even if you could've slightly tweaked our starting conditions from a few years or decades ago, weren't we going to blow ourselves up anyway?
And if doom is sufficiently overdetermined, then the future we grieve for, transhumanist or otherwise, was only ever a mirage.
So gotta keep in mind that probabilities are in your head (I flip a coin, it's already tails or heads in reality, but your credence should still be 50-50). I think it can be the case that we were always doomed even if weren't yet justified in believing that.
Alternatively, it feels like this pushes up against philosophies of determinism and freewill. The whole "well the algorithm is a written program and it'll choose what is chooses deterministically" but also from the inside there are choices.
I think a reason to have been uncertain before and update more now is just that timelines seem short. I used to have more hope because I thought we had a lot more time to solve both technical and coordination problems, and then there was the DL/transformers surprise. You make a good case and maybe 50 years more wouldn't make a difference, but I don't know, I wouldn't have as high p-doom if we had that long.
I know that probabilities are in the map, not in the territory. I'm just wondering if we were ever sufficiently positively justified to anticipate a good future, or if we were just uncertain about the future and then projected our hopes and dreams onto this uncertainty, regardless of how realistic that was. In particular, the Glorious Transhumanist Future requires the same technological progress that can result in technological extinction, so I question whether the former should've ever been seen as the more likely or default outcome.
I've also wondered about how to think about doom vs. determinism. A related thorny philosophical issue is anthropics: I was born in 1988, so from my perspective the world couldn't have possibly ended before then, but that's no defense whatsoever against extinction after that point.
Re: AI timelines, again this is obviously speaking from hindsight, but I now find it hard to imagine how there could've ever been 50-year timelines. Maybe specific AI advances could've come a bunch of years later, but conversely, compute progress followed Moore's Law and IIRC had no sign of slowing down, because compute is universally economically useful. And so even if algorithmic advances had been slower, compute progress could've made up for that to some extent.
Re: solving coordination problems: some of these just feel way too intractable. Take the US constitution, which governs your political system: IIRC it was meant to be frequently updated in constitutional conventions, but instead the political system ossified and the last meaningful amendment (18-year voting age) was ratified in 1971, or 54 years ago. Or, the US Senate made itself increasingly ungovernable with the filibuster, and even the current Republican-majority Senate didn't deign to abolish it. Etc. Our political institutions lack automatic repair mechanisms, so they inevitably deteriorate over time, when what we needed was for them to improve over time instead.
I'm just wondering if we were ever sufficiently positively justified to anticipate a good future, or if we were just uncertain about the future and then projected our hopes and dreams onto this uncertainty, regardless of how realistic that was.
I think that's a very reasonable question to be asking. My answer is I think it was justified, but not obvious.
My understanding is it wasn't taken for granted that we had a way to get more progress with simply more compute until deep learning revolution, and even then people updated on specific additional data points for transformers, and even then people sometimes say "we've hit a wall!"
Maybe with more time we'd have time for the US system to collapse and be replaced with something fresh and equal to the challenges. To the extent the US was founded and set in motion by a small group of capable motivated people, it seems not crazy to think a small to large group such people could enact effective plans with a few decades.
One more virtue-turned-vice for my original comment: pacifism and disarmament: the world would be a more dangerous place if more countries had more nukes etc., and we might well have had a global nuclear war by now. But also, more war means more institutional turnover, and the destruction and reestablishment of institutions is about the only mechanism of institutional reform which actually works. Furthermore, if any country could threaten war or MAD against AI development, that might be one of the few things that could possibly actually enforce an AI Stop.
For me, S2 explicitly I can't justify being quite that confident, maybe 90-95%, but emotionally 9:1 odds feels very like "that's what's happening".
I'm not that invested in defending the p>99% thing; as Yudkowsky argues in this tweet:
If you want to trade statements that will actually be informative about how you think things work, I'd suggest, "What is the minimum necessary and sufficient policy that you think would prevent extinction?"
I see the business-as-usual default outcome as AI research progressing until unaligned AGI, resulting in an intelligence explosion, and thus extinction. That would be the >99% thing.
The kinds of minimum necessary and sufficient policies I can personally imagine which might possibly prevent that default outcome, would require institutions laughably more competent than what we have, and policies utterly outside the Overton window. Like a global ban on AI research plus a similar freeze of compute scaling, enforced by stuff like countries credibly threatening global nuclear war over any violations. (Though probably even that wouldn't work, because AI research and GPU production cannot be easily detected via inspections and surveillance, unlike the case of producing nuclear weapons.)
I think your defense of the >99% thing is in your first comment where you provided a list of things that cause doom to be “overdetermined”- meaning you believe that any one of those things is sufficient enough to ensure doom on its own (which seems nowhere near obviously true to me?).
Ruby says you make a good case, but considering what you’re trying to prove, (I.e. near-term “technological extinction” is our nigh-inescapable destiny) I don’t think it’s an especially sufficient case, nor is it treading any new ground. Like yeah, the chances don’t look good, and it would be a good case (as Ruby says) if you were just arguing for a saner type of pessimism, but to say it’s overdetermined to the point where it’s a >99% chance that not even an extra 50 years could move just seems crazy to me, whether you feel like defending it or not.
As far as the policy thing goes, I don’t really know what the weakest thing I could see doing that could avert an apocalypse would be. Although, something I’d like to see would be some kind of coordination regarding setting standards for testing or minimum amounts of safety research and then have compliance reviewed by a board maybe, with both legal and financial penalties to be administered in case of violations.
Probably underwhelming to you, but then as far as concrete policy goes it’s not something I think about a ton, and I think we’ve already established my views are less extreme than yours. And absent of any of my idea being remotely feasible, that still wouldn’t get me up to >99%. Something that would get me there would be actually seeing the cloud of poison spewing death drones (or whatever) flying towards me. Heck, even if I had a crystal ball right now and saw exactly that, I still wouldn’t see previously having a >99% credence as justifiable.
Am I just misunderstanding you here?
My initial comment isn't really arguing for the >99% thing. Most of that comes from me sharing the same so-called pessimistic (I would say realistic) expectations as some LWers (e.g. Yudkowsky's AGI Ruin: A List of Lethalities) that the default outcome of AI progress is unaligned AGI -> unaligned ASI -> extinction, that we're fully on track for that scenario, and that it's very hard to imagine how we'd get off that track.
meaning you believe that any one of those things is sufficient enough to ensure doom on its own (which seems nowhere near obviously true to me?)
No, I didn't mean it like that. I meant that we're currently (in 2025) in the >99% doom scenario, and I meant it seemed to me like we were overdetermined (even back in e.g. 2010) to end up in that scenario (contra Ruby's "doomed for no better reason than because people were incapable of not doing something"), even if some stuff changed, e.g. because some specific actors like our leading AI labs didn't come to exist. Because we're in a world where technological extinction is possible and the default outcome of AI research, and our civilization is fundamentally unable to grapple with that fact. Plus a bunch of our virtues (like democracy, or freedom of commerce) turn from virtue to vice in a world where any particular actor can doom everyone by doing sufficient technological research; we have no mechanism whereby these actors are forced to internalize these negative externalities of their actions (like via extinction insurance or some such).
that not even an extra 50 years could move
I don't understand this part. Do you mean an alternative world scenario where compute and AI progress had been so slow, or the compute and algorithmic requirements for AGI had been so high, that our median expected time for a technological singularity would be around the year 2070? I can't really imagine a coherent world where AI alignment progress is relatively easier to accomplish than algorithmic progress (e.g. AI progress yields actual feedback, whereas AI alignment research yields hardly any feedback), so wouldn't we then in 2067 just be in the same situation as we are now?
Although, something I’d like to see would be some kind of coordination regarding setting standards for testing or minimum amounts of safety research and then have compliance reviewed by a board maybe, with both legal and financial penalties to be administered in case of violations.
I don't understand the world model where that prevents any negative outcomes. For instance, AI labs like OpenAI currently argue that they should be under zero regulations, and even petitioned the US government to be exempted from regulation; and the current US government itself cheerleads race dynamics and is strictly against safety research. Even if some AI labs voluntarily submitted themselves to some kinds of standards, that wouldn't help anyone when OpenAI and the US government don't play ball.
(Not to mention that the review board would inevitably be captured by interests like anti-AI-bias stuff, since there's neither sufficient expertise nor a sufficient constituency for anti-extinction policies.)
Something that would get me there would be actually seeing the cloud of poison spewing death drones (or whatever) flying towards me. Heck, even if I had a crystal ball right now and saw exactly that, I still wouldn’t see previously having a >99% credence as justifiable.
That's a disbelief in superintelligence. You need to deflect the asteroid (prevent unaligned ASI from coming into being) long before it crashes into earth, not only when it's already burning up in the atmosphere. From my perspective, the asteroid is already almost upon us (e.g. see the recent AI 2027 forecast), you're just not looking at it, or you're not understanding what you're seeing.
Most of that comes from me sharing the same so-called pessimistic (I would say realistic) expectations as some LWers (e.g. Yudkowsky's AGI Ruin: A List of Lethalities) that the default outcome of AI progress is unaligned AGI -> unaligned ASI -> extinction, that we're fully on track for that scenario, and that it's very hard to imagine how we'd get off that track.
Ok, but I don’t read see those LWers also saying >99%, so what do you know that they don’t which allows you to justifiably hold that kind of confidence?
That's a disbelief in superintelligence.
For what it’s worth, after rereading my own comment I can see how you might think that. With that said, I do think super intelligence is overwhelmingly likely to be a thing.
I sympathize with this line of thinking, but I've never understood something like P(doom)>0.8.
The analogies with cancer or poison seem a bit odd, because we're trying to estimate the probability of an event that has never happened before. Without relying on anything like physical laws, without anything close to consensus. Even among the people who proposed the key ideas of the AI Risk discussions, not all were confident pessimists.
We have too many unknowns. We don't know when superintelligence will appear. We can't predict how governments and corporations will treat AI in the coming years. We don't know what will happen if someone tries to use a sufficiently advanced AI for automated safety research. Or narrow AI might change the situation in the world before superintelligence appears. Our civilization could collapse for any number of reasons.
And I don't think we can say for sure what superintelligence will do to humans.
In reckoning with my feelings of doom, I wrote a post in which I drew upon Viktor Frankl's popular book Man's Search for Meaning. Here's an excerpt from that post that discusses why hope is not cheap, but necessary for day-to-day life:
Frankl, a psychologist and holocaust survivor, wrote about his experience in the concentration camps. He observed two types of inmates—some prisoners collapsed in despair, while others were able to persevere in the harsh conditions:
"Psychological observations of the prisoners have shown that only the men who allowed their inner hold on their moral and spiritual selves to subside eventually fell victim to the camp’s degenerating influences.
"We who lived in concentration camps can remember the men who walked through the huts comforting others, giving away their last piece of bread. They [were] few in number, but they offer sufficient proof that everything can be taken from a man but one thing: the last of the human freedoms—to choose one’s attitudes in any given set of circumstances.
A depressed person honestly believes there’s no hope that things will improve. Consequently, they collapse into a vegetative state of despair. It follows, therefore, that hope is literally the prerequisite for action.
So if I didn’t implicitly believe there’s a tomorrow worth living, then I wouldn’t have written this post.
I concluded that post with this:
If a storm disturbs the rock garden of a Zen monk, the next day he goes to work to restore its beauty.
This does sound OCD in that all the psychic energy is going into rationalizing doom slightly differently in the hopes that this time you'll get some missing piece of insight that will change your feelings. Like I think we can accrue a sort of moral OCD that if too many people behavior as if p(doom)=low we must become the ascetic who doesn't just believe it's high but who has many mental rituals meant to enforce believing it's high as many hours a day as possible.
ERP (exposure/response prevention) is gold standard for OCD, not ACT or DBT. I mean, there's overlap, because ACT emphasizes cognitive defusion which is basically what ERP produces, and dialectics can be exposures to that thing you're avoiding. And there's a more OCD-treating way to do "radical acceptance," it's less like focusing on the truth of the unpleasant thing and "watch" the thoughts go by like "leaves on a stream."
I think an exposure for you might be more "p(doom) is close to 0, I was mistaken", and in rationalist-speak privately steelman this for 1-5 minutes, but also you know like a "butterfly thought," where you don't want to crush the butterfly. And just to emphasize this is a private exercise, if you "actually" believe p(doom) is 0 for 2 minutes, it's really not going to hurt you. The more I make it a rationalist exercise the less it's an ERP and the more it's just more rationalization, but I feel I need to "steelman" just exiting the space of constant high focus on high p(doom) at all. An ERP might look like this example but a therapist might also try to keep it more "low key," like pick an easy exposure to skill-build on, like if you're afraid of everyone dying and also spiders, therapy might target spiders as just easier to practice with.
For me the ACT perspective is things I care about include: 1. job 2. family 3. doom. What I don't include in this list is 4. figuring out the most important thing and only doing that. Yes it's compartmentalizing and I haven't endowed "doom" with any ability to leave its box and it's actually still #3 not 1 or 2. I mean this is a work-in-progress for me too but this sort of enumerated-values approach has been materially helpful every day.
I think having a huge p(doom) vs a much smaller one would change this article substantially. If you have 20-30 or 50% doom you can still be positive. In all other cases it sounds like a terminal illness. But since the number is subjective living your life like you know you are right is certainly wrong. So I take most of your article and apply it in my daily life and the closest to this is being a stoic but by any means I don't believe that it would take a miracle for our civilization to survive. It's more than that and it's important.
But since the number is subjective living your life like you know you are right is certainly wrong
I don't think this makes sense. Suppose you have a subjective belief that a vial of tasty fluid is lethal poison 90%, you're going to act in accordance with that belief. Now if other people think differently from you, and you think they might be right, maybe you adjust your final subjective probability to something else, but at the end of the day it's yours. That it's subjective doesn't rule it out being pretty extreme.
If what you mean is you can't be that confident given disagreement, I dunno, I wish I could have that much faith in people.
If what you mean is you can't be that confident given disagreement, I dunno, I wish I could have that much faith in people.
In another way, being that confident despite disagreement requires faith in people — yourself and the others who agree with you.
I think one reason I have a much lower p(doom) than some people is that although I think the AI safety community is great, I don’t have that much more faith in its epistemics than everyone else’s.
1Don't look away
But more than just "what do I believe", I think it's of equal or greater importance what you pay attention to. A person can correctly believe that we face doom yet try to just not think about it. In effect, if you never think about doom, are you any better off than if you didn't believe in it?
The movie Don't Look Up did a good job of capturing the feeling of doom and how the global citizenry might react to a apocalyptic event. Many people in the movie chose to live in blissful ignorance.
I discovered LessWrong a year ago and never read the AI-related material. I had a feeling I wouldn't like it so I avoided it. Now that I'm in the thick of it (as of a month ago), I'm reminded of this text my Christian friend sent me 5 years ago:
She was, of course, referring to religion, but it's an excellent series of questions that can equally be applied to AGI-related doom.
Who can I talk to about my doom? I tried discussing the implications with two of my married friends yesterday: the husband was receptive to the topic, but the wife refused the engage in the discussion because it was too stressful.
I tried talking to my parents about it, but they're older and don't understand AI.
I thought about trying talk therapy for the first time, but if the therapist is uninformed about AGI, I don't want to introduce them to new stress and existential angst.
This is a great essay and I find myself agreeing with a lot of it. I like how it fully accepts the possibility of doom while also being open to and prepared for other possibilities. I think becoming skilled at handling uncertainty, emotions, and cognitive biases and building self-awareness while trying to find the truth are all skills aspiring rationalists should aim to build and the essay demonstrates these skills.
1