I think there's a fallacy in going from "slight caring" to "slight percentage of resources allocated."

Suppose that preserving earth now costs one galaxy that could be colonized later. Even if that one galaxy is merely one billionth of the total reachable number, it's still an entire galaxy ("our galaxy itself contains a hundred billion stars..."), and its usefulness in absolute terms is very large.

So there's a hidden step where you assume that the AIs that take over have diminishing returns (strong enough to overcome the hundred-billion-suns thing) on all their other desires for what they're going to do with the galaxies they'll reach, allowing a slight preference for saving Earth to seem worthwhile. Or maybe they have some strong intrinsic drive for variety, like how if my favorite fruit is peaches I still don't buy peaches every single time I go to the supermarket.

If I had no such special pressures for variety, and simply valued peaches at $5 and apples at $0.05, I would not buy apples 1% of the time, or dedicate 1% of my pantry to apples. I would just eat peaches. Or even if I had some modest pressure for variety but apples were only my tenth favorite fruit, I might be satisfied just eating my five favorite fruits and would never buy apples.

You describe this as "there are some reasons why small amounts of motivation don't suffice (given above) which are around 20% likely", but I think that's backwards. Small amounts of motivation by default don't suffice, but there's some extra machinery AIs could have that would make them matter.

Or to try a different analogy: Suppose a transformer model is playing a game where it gets to replace galaxies with things, and one of the things it can replace a galaxy with is "a preserved Earth." If actions are primitive, then there's some chance of preserving Earth, which we might call "the exponential of how much it cares about Earth, divided by the partition function" making a Boltzmann-rational modeling assumption. But if a model with similar caring it has to execute multi-step plans to replace each galaxy, then the probability of preserving Earth goes down dramatically, because it will have chances to change its mind and do the thing it cares for more (using Boltzmann-rationality assumption the other way). So in this toy example, a slight "caring" in the sense of what the model will say it would pick when quickly asked isn't represented when you look at the distribution of results of many-step plans.

If small motivations do matter, I think you can't discount "weird" preferences to do other things with Earth than preserve it. "Optimize Earth according to proxy X, which will kill all humans but really grow the economy / save the ecosystem / fill it with intelligent life / cure cancer / make it beautiful / maximize law-abiding / create lots of rewarding work for a personal assistant / really preserve Earth". Such motivations sound like they'd be small unless fairly directly optimized for, but the AI is supposed to be acting on small motivations, why not those bad ones, rather than the one we want?

@So8res^[1] responded on twitter, copying his response here for completeness:

I appreciate your emphasis the bottom line that (IIUC) we agree upon: rogue superintelligence would permanently ruin the future and (at best) relegate humanity to something like zoo animals.

As I understand your arguments, they boil down to (a) maybe the AI will care a little and (b) maybe the AI will make backups of humans and sell them to aliens.

My basic response to (a) is: weak prefs about humans (if any) would probably get distorted, and existing ppl probably wouldn't be the optimum (and if they were, the result probably wouldn't be pretty). cf

My basic response to (b) is: I'm skeptical of your apparent high probabilities (>50%?) on this outcome, which looks somewhat specific and narrow to me. I also expect most implementations to route through a step most people would call "death".

In case (b), the traditional concept of "death" gets dodgy. (What if copies of you are sold multiple times? What if you only start running in an alien zoo billions of years later? What if the aliens distort your mind before restoring you?) I consider this topic a tangent.

Mostly I consider these cases to be exotic enough and death-flavored enough that I don't think "maybe AIs will sell backups of us to aliens" merits a caveat when communicating the basic danger. But I'm happy to acknowledge the caveat, and that some people think it's likely.

1. ^{^}

   If Nate cross posts himself, I'll delete this comment.

It's unclear what fraction of people die due to takeover because this is expedient for the AI, but it seems like it could be the majority of people and could also be almost no one. If AIs are less powerful, this is more likely (because AIs would have a harder time securing a very high chance of takeover without killing more humans).

Yeah, this (extinction to facilitate takeover) seems like the most plausible pathway to total or near-total extinction by far. An AI that is only a little bit smarter than humanity collectively has to worry about humans making a counter-move - launching missiles or building a competing AI or various kinds of sabotage. If you're a rogue AI, engineering a killer virus (something that smart humans can already do or almost do, if they wanted to) as soon as you or humanity has built out sufficient robotics infrastructure, makes all the subsequent parts of your takeover / expansion plan much less contingent and more straightforward to reason about. (And I think the analogy to historical relatively-bloodless coups here is a pretty weak counter / faint hope - for one, because human coup instigators generally still need humans to rule over, whereas AIs wouldn't.)
 

If there are a large number of different rogue AIs, it becomes more likely that one of them would benefit from massive fatalities (e.g. due to a pandemic) making this substantially more likely.

I don't see how the number of AIs makes a big difference here, rather than the absolute power level of the leading AI? An extinction or near-extinction event seems beneficial to just about any unaligned AI that is not all-powerful enough to not have to worry about humanity at all.

Put another way, the scenarios where extinction doesn't happen due to takeover only feel plausible in scenarios where a single AI fooms so fast and so hard that it can leave humanity alive without really sweating it. But if I understand the landscape of the discourse / disagreement here, these fast and discontinuous takeoff scenarios are exactly the ones that you and some others find the least plausible.

I don't feel like "slightly caring" imagery coherently describes any plausible feature of caring of misaligned AIs.

I think that if Earth-originated AI gets to choose between state of the world without humans and exactly the same world but with humans added, supermajority of AIs is going to choose the latter, because of central role humans are likely to take in training data. But "humans are more valuable than no humans ceteris paribus" doesn't say anything about whether configuration of matter that contains humans is literally the best possible configuration. Take "there are galaxies of computronium and solar system with humans" vs "there is one solar system of computronium more, no humans".

To put things on scale: on napkin estimate, plausible lower bound on achievable density of computations is ${10}^{35}$ ops/sec*kg. If we assume that for minimally comfortable live humans will need 100 times their weight and one human has weight 80kg, we will get that one second of existence of single human is worth ~${10}^{39}$ operations, which is approximately 3 billion years of computations on all currently existing computers. I think it sends idea of non-uploaded humans existing after misaligned takeover straight out of window. 

So I think this topic isn't exactly cruxy for most decisions (except things routing through "is the IABIED thesis in particular wrong?". But, wanted to leave a quick note about where I ended up after thinking about it awhile longer:

tl;dr

• "for trade" doesn't seem like it should be >50% likely to avoid something-like-death (and IMO, the only reason I'm not at <1% is model uncertainty)
  • (for Pascal Wager Counterargument reasons)
• "because they're actually nice" doesn't seem like it should be >25%-ish likely (and I'd personally go much lower than 25%)
  • (for "uncertainty about how their values will structurally work" reasons)

(for time reasons I didn't doublecheck exactly what numbers you assigned to things and was going off memory, I may have misremembered and maybe you're not saying things that different from what I say here but it seemed like you were. If your numbers were similar to what I wrote above maybe this isn't a disagreement, just an elaboration)

Game Theory

Agreed there's some game theoretic reason to preserve humanity just-in-case, but, I don't see why it'd route through "forced uploading or stasis." 

There might be entities that might pay a premium for humanity that was fully preserved, and got access to a full star system. 

But there might also be entities that care about humanity, but, actively prefer a pristine stasis copy, rather than a "stunted" copy that evolved on it's own for millions of years in suboptimal conditions.^[1]" Or, lots of entities that prefer stasis copies because they are just intellectually interested in us rather than caring about us. Pascal's Wager has to account for multiple gods wanting multiple things.

So it seems wrong to put a >50% (or really even 10%) on "AI preserves humanity somehow for Trade Reasons" resulting in "we straightforwardly get to survive."

 Just Being Nice because maybe it's very slightly aligned or niceness is very slightly schelling.

The general thrust of:

• look, it really needs to be such a teeny amount nice for it to end up nice
• it would be pretty weird if somehow it had nonzero propensity to be nice and the amount was 'less than Leave Earth Alone'
• it would be pretty weird if it had zero niceness

seems at least plausibly pretty compelling. The main thing that felt sus about your other comments here was defining 'pretty nice' in terms of 'they will spend a fraction of their resources on us, no matter what, even if there are lots of other things they care about more', and then giving that a >50% likelihood of being how niceness would play out. (Tho I might have misread you there)

That's one way things could shake out. Two other ways are:

1.  They care about us nonzero, but there are nigh-infinite things they also care about about-as-much-as-us-or-more, enough that the many galaxies of resources they lose by slowing down aren't clearly just more important
2. They care about us, but not in a 'fraction-of-resources' sense, but rather in a 'they'd save us all-else-equal, but, things are not equal, and even ignoring the many galaxies out there there's just something they'd rather do with one spare solar system more than us.

(I'd maybe include #3 "random other reasons we haven't thought about yet")

#1 and #2 are similar, but I intend to make fairly different points with them.

The structure of #1 is taking at face value them attempting to give us some small fraction of their resources, just, even when they do that, the teeny-percentage-cost of us is outweighed by there being vast numbers of things they also want to give a similar percentage to, and the number of galaxies they lose by slowing down a few weeks or months just dwarfs the amount they are able to devote.

The structure of #2 is "they just aren't even trying to give us a fraction of their wealthy, they're just weighing each block of resources and asking 'what's the best thing to do with this?' and we show up under consideration but below the threshold.

It seems very weird to be confident in your frame #0 of "they earnestly care about us nonzero, which they frame as a fraction of their wealth, and there are no competing things they care about" as noticeably more likely than the other two, without more argumenation than you've made thus far.

It seems like the structure of the "but it's so cheap" argument is like "the EV of them caring a little is pretty high, because it's so cheap" but that is different than "the chances of them structurally caring this way is >50%"

1. ^{^}

   (I guess at least a partial counterpoint is "of the aliens that care about our values/freedom, rather than us as intellectually curiosities, maybe they care a lot about us all getting preserved, and maybe it's about as hard to put us all in stasis as to just leave us alone." Seems seems like more a stretch and shouldn't be the mainline guess)

We have a ~25% chance of extinction

Maybe add the implied 'conditional on AI takeover' to the conclusion so people skimming don't come away with the wrong bottom line? I had to go back through the post to check whether this was conditional or not.

Other entities that care trade with the AI (or with entities that the AI trades with) to keep humans alive. This includes acausal trade, ECL, simulations/anthropic capture (to be able to effectively acausally trade with decision theory naive AIs), and causal trade with aliens that the AI ends up encountering. It's unclear how simulations/anthropic capture works out. It seems plausible that this happens in a way which doesn't result in all relevant (acausal) trades happening. It seems plausible that other entities (including humans in other Everett branches) pretty universally don't want to bail humans (in this branch) out because they have better things to spend their resources on, but even a tiny fraction of entities spending some non-trivial fraction of resources could suffice. This depends on there being some beings with power who care about things like human survival despite alignment difficulties, but this feels very likely to me (even if alignment is very hard, there may well be more competent aliens or AIs who care a small amount about this sort of thing). Note that this requires the AI to care about at least some of these mechanisms which isn't obvious.

How much of your plausibility-of-caring-slightly routes through this, vs the "actually just slightly intrinisically nice?" thing.

In the previous discussion, you said:

I'm happy to consider non-consensual uploading to be death and I'm certainly happy to consider "the humans are modified in some way they would find horrifying (at least on reflection)" to be death. I think "the humans are alive in the normal sense of alive" is totally plausible and I expect some humans to be alive in the normal sense of alive in the majority of worlds where AIs takeover.

Making uploads is barely cheaper than literally keeping physical humans alive after AIs have fully solidified their power I think, maybe 0-3 OOMs more expensive or something, so I don't think non-consensual uploads are that much of the action. (I do think rounding humans up into shelters is relevant.)

I'm surprised at the "only 3 OOMs more expensive." I haven't done a calculation but that seems really implausible. Maybe I am not imagining how big an OOM is properly.

In particular, comparing "keep humans alive indefinitely, multigenerationally" vs "store them on ice without evening running them, turn them back on when/if you trade them." 

The sort of entity that might pay extra for "keep them actually alive instead of on-ice"... probably also cares about them being alive and getting some kind of standard of living or something. (fyi it's not even obvious most people would prefer being alive in minimally-satisfying-shelters vs stored, or run in simulation at a higher standard of living)

Insofar as this option is loadbearing for "you put lowish odds on them killing everyone", I think a crux is not finding it very plausible that the AI would choose "keep fully alive in a way that is clearly better than death" vs "store digitally" or "upload." It's just a very narrow target for the game theory to work out such that this exact thing is what turns out to matter.

Curated. This is a topic that is getting a ton of attention at the minute, and the post does a decent job at laying out relevant considerations and summarizing some prior debate.

(I disagree with the conclusion but it seems positive to me to curate something with a different conclusion than the recent book on the issue.)

One difficulty I have with these posts is I never have any idea how seriously to take any of the percentages or even the overall conclusion. My overall update is to doubt there’s a reasonable threshold of saying “we don’t know enough to put a number on this”.

If the AI keeps humans as uploads, it does not necessarily mean that they are running. They could be paused, ready to get unpaused when there is a reason to do so (e.g. the AI meets another space AI which values biological minds). That is effectively the same as being dead, because the probability of meeting an alien AI that cares about humans is tiny. Paused uploads are cheaper than running uploads.

Preserving earth (as in, not causing catastrophic environmental damage due to industrial expansion) is more expensive than keeping physical humans alive which is more expensive than only keeping humans alive as uploads.

Not saying this is likely, but if the AI is not speciesist in favor of humans (it does care about humans, but not more than e.g. whales or chimpanzees), then plans which end up protecting a large majority of humans look a whole lot more expensive overall. 

Interesting and good breakdown.

I place much higher odds on the "death due to takeover" for a pretty specific reason. We seem to have an excellent takeover mechanism in place which kills all or most of us: nukes. We have a gun pointed at our collective heads, and it's deadlier to humans than AGIs.

Igniting a nuclear exchange and having just enough working robots, power sources, and industrial resources (factories, etc) to rebuild seems like a pretty viable route to fast takeover.

Igniting that exchange could be done via software intrusion or perhaps more easily by spoofing human communications to launch. This is basically the only use of deepfakes that really concerns me, but it concerns me a lot.

This becomes increasingly concerning to me in a multipolar scenario, which in turn seems all too likely at this point. Then every misaligned AGI is incentivized to take over as quickly as possible. A risky plan becomes more appealing if you have to worry that another AGI with different goals may launch their coup at any point.

This logic also applies if we solve alignment and have intent-aligned AGIs in a multipolar scenario with different human masters. I guess it also favors everyone who can, getting themselves or a backup to a safe location.

This is also conditional on progress in robotics vs. timelines; even with short timelines it seems like robotics will probably be far enough along for clumsy robots to build better robots. But here my knowledge of robotics, EMP effects, etc, fails. It does seem like a nontrivial chance that triggering a nuclear exchange is the easiest/fastest route to takeover.

One very well-informed individual told me nobody knows if any humans would survive a nuclear winter; but that's probably less important than whether the "winning" AGI/human wanted them to survive.

The number of likely deaths given takeover seems higher than your estimate if that logic about nukes as a route to takeover mostly goes through.

But the question of extinction still hinges largely on whether AGI has any interest in humanity surviving. I think you're assuming it will have a distribution of interests like humans do; I don't think that's a safe assumption at all, even given LLM-based AGI and noting that LLMs do really seem to have a distribution of motivations, including kindness toward humans. 

I think how they reason about their goals once they're capable of doing so is very hard to predict, so I'd give it more like a 50% chance they wind up being effectively maximizers for whatever their top motivation happens to be. There seems to be some instrumental pressure toward reflective stability, and that might favor one motivation winning outright vs. sharing power within that mind. I wrote a little about that in Section 5 and more in the remainder of LLM AGI may reason about its goals and discover misalignments by default, but it's pretty incomplete.

Like the rest of alignment, it's uncertain. LLMs have some kindness, but that doesn't mean that LLM-based AGI will retain it. If we could be confident they would, we'd be a lot better set to solve alignment.