This is a linkpost for https://worldspiritsockpuppet.substack.com/p/doom-as-a-bad-method-not-a-utopia
New Comment
Perhaps this is partly what you get if you promote reasoning in terms of expected values too hard, too generally (instead of "saner", less extremistan-conducive decision rules, or even heuriatics whatever "normal people use").
I don't know how much this has to do being overly neoclassical-econ-brained, but it's a live option for me that it does have non-trivial amount, whether it's in terms of actually skewing people's "honest thinking" or by giving them more concepts to produce palatable justifications for why they are riding the cancerous wave.
Expected values are good, but you can use them to trample other intuitions people have which are better, such as:
- Don't do something that's "bold", which is to say, something that's sticking your neck out in a way other people don't want you to and they aren't doing--or to say it in a more rationalist-friendly way, beware the unilateralist's curse.
- Don't do something that might cause a whole bunch of harm.
- Don't
I guess "expected values" should be packaged with "BTW, some people are systematically biased to put middling numbers on things (probabilities, utilities); if you're such a person, you can't trust your expected value calculations to not be way way off the true utility; maybe try tapping into these other intuitions, and then rationalizing (or if you prefer, "reexplaining") them in terms of expected value.".
I do think that expected values are generally valuable input into decision-making and sometimes good as the primary decision criterion.
The case that we're talking about here one where one branch is not just low-value, but an irrecoverable loss ("doom", extinction-level stuff), and that's one of the places, where expected values tend to mislead the most. Cf. The St Petersburg Paradox, non-ergodicity (as Aprillion says), etc. The natural/obvious fix (unclear to me how principled/holistically good) is to apply expected value reasoning to the policy level, but it's hard and I think it's uncommon to do that.
(Not sure whether/how much you disagree, but I realized I had kinda assumed this to be obvious in the context, so thought it worth clarifying.)
Unsure if we disagree, but maybe? Specifically, I'm saying something rather unsophisticated, which is like:
In general, when people try to do expected utility calculations, they feel uncomfortable using large positive or negative utilities.
This includes as a special case that irreversible outcomes don't get counted properly. Normal intuitions account for this better than someone trying to use expected value calculations but with this middling-number bias.
Now I understood what you meant by "middling numbers".
I haven't thought about this in these terms, but yeah seems true.
Building AI now instead of later is a very bad tradeoff in terms of expected value. IMO the problem is that people are doing EV reasoning wrong, not that they're doing it in the first place.
I think this makes good points. Definitely we shouldn't talk about p(doom) without an assumption about how we approach making AGI. I think the assumption that we'll just race flat out is breaking down, making more approaches more likely.
But your core point doesn't apply to most people, beyond a little slowdown.
If the the drive takes so long you'll die on the way if you don't drive 200 mph, it's not a mistake to risk it.
Humanity isn't remotely longtermist on average, nor are they utilitarian. They primarily care about themselves, and a few loved ones and friends. So the pitch "we can get there safely in 40 years" means asking most decision-makers (who tend to be over 40) to die and let their parents die. Sure they could save their kids and maybe some of them would if they understood the stakes. But there's a powerful incentive to convince themselves it's not too dangerous to try to save themselves.
This is less common in the rationalist community, but rationality does not demand utilitarianism (at least not clearly).
This tradeoff is also less common in the AI developer community just because they tend to be young.
I realize that's not why most people who oppose slowdown do. I hope we get to a situation where this distinction does become relevant, because that would mean everyone is seeing the situation more clearly. Which would improve our odds of navigating it.
This is what came to mind for me too.
Even confronted directly with a mention of "long run" as in "...How positive or negative do you expect the overall impact of this to be on humanity, in the long run?" I would wager most participants are not deeply considering the aggregate value of humanity's cosmic endowment.
Of course, no one can really feel that kind of scale intuitively, but I think it is important to believe and act as if one could. While clearly much smaller in magnitude and undoubtably a sunk cost, I find thinking about our ancestors much more salient and easy to imagine.
I have this narrative where hundreds of million years of brutal evolutionary contest and tens of thousands of generations of humans (most of whom existed in what we would consider terrible poverty) finally culminates in modern humanity's real chance to transcend these conditions and fill the observable universe with wonder...
...Then our generation just fumbles it and makes all of that suffering essentially in vain.
It's such a depressing story that I stop caring about my own shot at LEV. I care much less even about my two children's shot. I don't know if other people would find this motivating, should there be some way to impart it to them outside of LW.
I think humanity might be longtermist by proxy. Most parents care more about their children, and their ability to succeed in the future, than themselves.
Not to mention the majority of humanity that deeply believes in keeping things human-centric, and find TESCREAL ideas repulsive.
driving at 200mph to your new job if you think there’s only a 10% chance you’ll die in a fiery crash on the way there, and a 30% chance this job will radically improve your life
except the metaphor is more like ...
Situation: we jumped out of an airplane without a parachute.
PauseAI: we'll just get back on the airplane!
"Optimists": maybe some sort of parachute/glider-like structure made of tied-together shirts would raise the probability that at least one of us survives our now inevitable encounter with the first technological singularity ground?
Flat disagree, if I thought that ASI meant a 50% chance of destroying everything versus a 50% chance of utopia I'd go for it without much further thought.
My problem is that I think ASI means there's a 100-epsilon % chance of destroying everything and an epsilon% chance of utopia/hell. And that seems like a bad gamble.
I am a little worried that I don't understand where the fat middle of your picture comes from. Are there really vast numbers of people working in AI who think AI will be no big deal?
I think this post boils down to the fact that you need to compare outcome likelihoods of a plan with outcome likelihoods of other plans (or the counterfactual of not pursuing the plan), not against a hypothetical where "nothing ever happens".
As the post argues, if following a plan implies 30% of utopia and 10% of extinction, whether you actually prefer it or not really depends on what you think the counterfactual outcome distribution is if you don't take the plan (or if you take the best alternative plan instead of this one).
The natural implication is that pessimists should be a lot more likely to take such a plan. To these people, gambling on AI is worth it because it will save everything, otherwise something else bad will happen that will doom humanity with much greater than 10% likelihood! (e.g. getting outraced by less responsible people, WW3, covert AI projects, etc)
In contrast, optimists should probably reject such a plan because they broadly think that better plans will become available the longer they wait, e.g. humanity will eventually figure out better alignment techniques that reduce the extinction risk closer and closer to 0%.
I don't think anything in my comment contradicts yours. I'm talking strictly about risk/benefit to humanity and not to the actor themselves
That makes sense; it was easy to read your comment as applying to individual actors, so I appreciate the clarification. (And for extra clarity on my part, I wasn't meaning to making an accusation against your character even in my initial reading of your statement, which is analytical and neutral. My comment was very pointed, but was not intended to be pointed at you.)
whether you actually prefer it or not really depends on what you think the counterfactual outcome distribution is if you don't take the plan (or if you take the best alternative plan instead of this one).
Only in some very very rare scenarios would the best alternative plan be nothing, so the "if you don't take the plan" part is irrelevant in the great majority of cases.
Treating it as more important than it is leads to errors of reasoning such as "we must do something, and this is something, therefore we must do this".
This is indeed true if your goal is to find the best plan.
The part of my comment about comparing a plan to doing nothing is just because sometimes you're just trying to decide whether what you are doing is net-positive / net-negative.
I think it's just too many exponentials that are crossing here and too many complex dynamic systems to model - everyone will be working with a different set of intuitions and heuristics they can't necessarily explain very well and will overfit the data they have to fit the inherent bias. How much someone will collapse or survive depends on their level of epistemic humility (which even at top levels of AI research are not equally distributed).
I think your second point on sloppiness is an important one though. We compress so much in public discourse these days - and on a topic as complex as this - this compression hurts and reinforces entrenched positions. I wrote about that a while back (https://algorithmictradeoff.substack.com/p/why-ai-arguments-talk-past-each-other). This can make even well calibrated and good-meaning debaters collapse because they reach the time-exhaustion point before they are able to accurately map each others' mental models.
What I thought this post was going to be about from the title:
The doom-utopia tradeoff is fake. To get to utopia, you need to avoid doom first. Everyone’s interests are aligned here.
You don’t get to work at all if your 200mph car crashes. As a utopian, you should be obsessed with making sure the car won’t crash, so you can get to utopia.
“Avoiding doom” is a method for attaining utopia, not sitting on a tradeoff curve with utopia.
I agree that all probabilities are relative to method, but doesn't this mean that the probability of utopia is also method contingent? If that's the case, then it is about what the trade-off is between methods of development and their distribution of outcomes. I would think there is some basis here for seeing variance manifest is longer tails on both sides, meaning that it's not necessarily that utopia and doom trade off, but that they are both produced by variance.
The question then becomes more of risk preferences than anything else and, reading this thread and the comments, I am struck by how much the assumed state of global welfare tends to bear on these preferences. In some ways, we could already be living in a utopia by past standards (this is probably even more true for the AI researchers surveyed), so why is the risk tolerance not lower for doom? I almost feel as if the "utopia" is merely a nod at collectivism from people whose incentives lean more towards the individual (glory, significance, curiosity).
Advanced AI is generally expected to have some very high variance outcomes—it might herald everything good, it might destroy humanity. For instance, here are 800 random AI researchers’ expectations about how good the future is, lined up:
From my 2023 survey
As you can see, most AI researchers put a serious chunk of probability on very different overall outcomes: maybe doom, maybe utopia. This is common. Most people I know who think there is a serious chance of the destruction of humanity from AI also believe that if humanity isn’t destroyed, things might be insanely good.
I often hear people talk as if this means we are in a trade-off where the question is whether the good outweighs the bad. For instance, they look at the people above who think there’s a 10% chance of extinction and a 30% chance of utopia and round this off to ‘net positive on AI’.
That seems like a kind of wild error. Like considering yourself optimistic regarding driving at 200mph to your new job if you think there’s only a 10% chance you’ll die in a fiery crash on the way there, and a 30% chance this job will radically improve your life.
The things you should be comparing are driving at 200mph and driving at a normal speed! The things you should be comparing are attempting to attain advanced AI by the current route, and by other routes!
We can debate whether all the other routes are bad or impossible somehow, for instance if constraining projects that risk loss of human control risks sending humanity into an irrecoverable ruin. But I don’t think having ruled out such things is why people are usually thinking in trade-off terms.
Rather I think this error comes from a few things:
If you are bullish on some kind of advanced AI utopia, you should generally be lesskeen to try to achieve it via a careless route that leaves you at high risk of dying and losing it on the way there.