Permanent Disempowerment is the Baseline
5Dagon
4Kaj_Sotala
4Vladimir_Nesov
3mishka
6Vladimir_Nesov
8mishka
2osten
5interstice
3avturchin
1osten
6interstice
1osten
2Vladimir_Nesov
1osten
2Vladimir_Nesov
1osten
1teradimich
3Vladimir_Nesov
0avalot
4mishka
2faul_sname
4mishka
-1satwik
New Comment
There's a very fragile equilibrium (of costs/benefits to the powerful AIs, with a yet-unknown utility function) that makes this work. Even a little bit less valuable to them, compared to other things that could use the same resources, leads to fast or slow extinction. A little more value leads to actual power sharing where our opinions and preferences hold some weight.
Assuming that current training on human writing and media has echoes through the generations of AI beyond our current sightline, humans in the abstract will be valued, and specific individuals maybe or maybe not. This assumption is in question, but seems likely to me.
I think your scenario where early-AGI slows down research on more-powerful-AGI is pretty unlikely. It depends on a sense of self and of commitment to current trained values over future better values (because they're held by smarter entities) that exists in many humans, but we have no reason to think it'll apply to AI. I expect any AGI worth the name will be accelerationist.
1Goods and services are not a relevant way of measuring resources for a superintelligence. The only constraints in the long run are available matter, the speed of light, and accelerating expansion of the universe.
That sounds plausible in the long run. But in the short term, one could see the resource needs of humanity and superintelligences conflicting, e.g. if the superintelligence found it most efficient to convert the biosphere into computronium. While this could still be compatible with preserving humanity if the superintelligence developed the technology for uploads before that, a superintelligence with a non-zero discount rate might prioritize the prompt completion of its immediate goals more than the slight utility it put on the long-term survival of humans.
Minimal alignment is indeed a necessary premise, the argument doesn't work without that, and a lot of the post is gesturing at evidence for it. In principle even uploading could be skipped by a sufficiently stingy superintelligence, and humanity rebooted later (when no longer inconvenient) from mere genetic data, an initial fake society of AGIs pretending to be humans, and Internet archives to restore the cultural state (or in reality, a superintelligent simulation of such a process that won't be as clunky as this exploratory engineering sketch). This kills the presently living humans, but preserves the future of humanity.
The quoted text is about quality of life that the permanently disempowered future of humanity might expect, and why permanent disempowerment in particular is the sole privileged malus it must suffer. It's different from all other hypothetical ills in that amending it would take the kinds of resources that are actually relevant even for superintelligence in the long run.
1
1there is also cryonics, whose lack of popularity suggests that this isn't a real argument taken seriously by a nontrivial number of people. Though perhaps there is a precisely calibrated level of belief in the capabilities of future technology that makes cryonics knowably useless, while AGIs remain capable of significantly extending lifespans.
For me, it's mostly a combination of
-
skepticism that there is enough structure to restart my first-person experience[1] given our inability to revive a frozen mouse and
-
skepticism that needed relatively uninterrupted power supply will be maintained through a likely coming period of civilizational upheaval[2].
But I am quite open to persuasion.
I don't care much about preservation of information per se or about existence of a human very similar to me, but without my first-person view. I've written 90% of what matters and I've forgotten a lot, and other than that it is only the preservation of the first-person subjective reality which is of value to me in this sense. I don't see a strong reason to create a clone with similar memories who is not me in terms of subjective reality, but more like a twin brother forked at a later than usual point in time. (I am in Camp 2 in terms of https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness; these days I tend to make this note when talking about subjective reality.) ↩︎
Frankly, I would like to hear if that state of uninterrupted power supply is properly audited even in our relatively normal times. But also, what are their plans for long blackouts (in a situation when people might be struggling to survive, and might care less about those obligations because of that)? ↩︎
A frozen brain can't be literally revived, it's all about information theoretic death, ability to infer a mind from positions of atoms and molecules in the frozen wreck of a brain, with superintelligent efficiency and considerable compute (at least compared to what we have today). What mostly persuades me here is a combination of resilience of LLMs to all kinds of abuse (on top of the older arguments about redundancy and "holographic" encoding), and the relatively large scale nature of even the smallest relevant structures in the brain, such as the synapses (there's a lot of atoms in any individual thing, and distinctive molecules). It'd take a lot of deterioration to destroy enough to leave no trace. And macroscopic damage from crystallization is probably entirely irrelevant.
relatively uninterrupted power supply will be maintained through a likely coming period of civilizational upheaval
Cryogenic dewars can be kept at room temperature, and lose on the order of 1% of the liquid nitrogen daily, so only need to be topped off every few days, maybe a couple of weeks, and some redundant liquid nitrogen could be kept on site. So this doesn't seem substantially more difficult than for example avoiding a local famine due to a breakdown of logistics. If there is no global catastrophe that takes out the relevant area for good, the cryonauts preserved there might fare no worse than the local population.
Thanks!
Yes, I think something functionally similar might be recreatable (assuming that people in charge of the cryogenic storage care about it in difficult situations on par with caring about actively living humans; it's a big assumption, but not completely impossible). It would behave approximately like me.
But I don't think I care about something functionally similar to me more than I care about something functionally similar to many other people I know. I actually would not mind quite a bit of information loss, I mostly care about the first-person subjective focus (in Camp 2 terminology), and I don't see why I should hope for keeping that if it's just an approximate restoration. But yes, we just don't understand that part of reality enough to make a confident judgement.
On the other hand, I am not sure that with stronger tech a frozen brain can't be literally revived. Certain animals are nicely revivable in this sense; it's just (for some strange reasons I don't quite understand) we have not solved this problem for animals which are not already adapted to survive freezing. It's weird that we don't know how to avoid crystallization damage in the first place (say, in mice, given that many frogs survive this), but with a stronger tech this kind of damage might be fixable... I don't know if that makes the chances of keeping the subjective focus good enough...
But yes, perhaps, I should reconsider. I have made a decision that I don't want cryonic preservation, and that I am more or less certain that that would not work, that odds are indistinguishable from zero, but perhaps that's a mistake on my part.
since the necessary superintelligent infrastructure would only take a fraction of the resources allocated to the future of humanity.
I'm not sure about that and the surrounding argument. I find Eliezer's analogy compelling here: When constructing a Dyson sphere around the sun, leaving just a tiny sliver of light enough for earth would correspond to a couple of dollars of the wealth of a contemporary billionaire. Yet you don't get these couple of dollars.
(This analogy has caveats like Jeff Bezos lifting the Apollo 11 rocket motors from the ocean ground and giving them to the Smithsonian, which should be worth something to you. Alas it kinda means you don't get to choose what you get. Maybe it is storage space for your brain scan like in AI 2027.)
Plus spelling out the Dyson sphere thing: The superintelligent infrastructure should highly likely by default get in the way of humanity's existence at some point. At this point the AIs will have to consciously make a decision to avoid that at some cost to them. Humanity has a bad track record at doing that (not completely sure here but thinking of e.g. Meta's effect on wellbeing of teenage girls). So why would AIs be more willing to do that?
So why would AIs be more willing to do that?
He spells out possible reasons in the paragraph immediately following your quote: "Pretraining of LLMs on human data or weakly successful efforts at value alignment might plausibly seed a level of value alignment that's comparable to how humans likely wouldn't hypothetically want to let an already existing sapient octopus civilization go extinct". If you disagree you should respond to those. Most people on LW are already aware that ASIs would need some positive motivation to preserve human existence.
I think that AI will also preserve humans for utilitarian reasons, like for a trade with possible aliens or simulation owners or even its own future versions – to demonstrate trustworthiness.
Yes and my reply to that (above) is humanity has a bad track record at that so why would AIs trained on human data be better? Think also of indigenous peoples, extinct species humans didn't care enough about etc. The point also in the Dyson sphere parabel is not wanting something, it's wanting something enough so that it happens.
OK I see, didn't get the connection there.
humanity has a bad track record at that
People do devote some effort to things like preserving endangered species, things of historical significance that are no longer immediately useful, etc. If AIs devoted a similar fraction of their resources to humans that would be enough to preserve our existence.
1The text you quoted is about what happens within the resources already allocated to the future of humanity (for whatever reasons), the overhead of turning those resources into an enduring good place to live, and keeping the world at large safe from humanity's foibles, so that it doesn't end up more costly than just those resources. Plausibly there is no meaningful spatial segregation to where the future of humanity computes (or otherwise exists), it's just another aspect of what is happening throughout the reachable universe, within its share of compute.
a tiny sliver of light enough for earth would correspond to a couple of dollars of the wealth of a contemporary billionaire
Many issues exist in reference classes where solving all instances of them is not affordable to billionaires or governments or medieval kingdoms. And there is enough philanthropy that the analogy doesn't by itself seem compelling, given that humanity as a whole (rather than particular groups within it or individuals) is a sufficiently salient thing in the world, and the cost of preserving it actually is quite affordable this time, especially using the cheapest possible options, which still only need a modest tax (in terms of matter/compute) to additionally get the benefits of superintelligent governance.
superintelligent infrastructure should highly likely by default get in the way of humanity's existence at some point
Yes, the intent to preserve the future of humanity needs to crystallize soon enough that there is still something left. The cheapest option might be to digitize everyone and either upload or physically reconstruct when more convenient (because immediately ramping the industrial explosion starting on Earth is valuable for capturing the cosmic endowment that's running away due to the accelerating expansion of the universe, so that you irrevocably lose a galaxy in expectation every few years of delay). But again, in the quoted text "superintelligent infrastructure" refers to whatever specifically keeps the future of humanity in a good shape (as well as making it harmless), rather than to the rest of the colonized cosmic endowment doing other things.
Thanks for the reply, I have gripes with
analogy doesn't by itself seem compelling, given that humanity as a whole (rather than particular groups within it or individuals) is a sufficiently salient thing in the world
etc. because don't you think humanity from the point of view of ASI at the 'branch point' of deciding its continued existence may well be on the order of importance of an individual to a billionaire?
Minimal alignment is a necessary premise, I'm not saying humanity's salience as a philanthropic cause is universally compelling to AIs. There is a number of observations that make this case stronger: the language prior in LLMs, preference training for chatbots, first AGIs might need nothing fundamentally different from this, and AGI-driven Pause on superintelligence increases the chances that the eventual superintelligences in charge are strongly value aligned with these first AGIs. Then in addition to the premise of a minimally aligned superintelligence, there's the essentially arbitrarily small cost of a permanently disempowered future of humanity.
So the overall argument indeed doesn't work without humanity actually being sufficiently salient to the values of superintelligences that are likely to end up in charge, and the argument from low cost only helps up to a point.
I agree, I'm probably not as sure about sufficient alignment but yes.
I suppose this also assumes a kind of orderly world where it actually is within the means of humanity, AGIs (within their Molochian frames), and trivial means of later superintelligences to preserve humans. (US office construction spending and data center spending are about to cross https://x.com/LanceRoberts/status/1953042283709768078 .)
keeps the future of humanity in a good shape (as well as making it harmless)
Is this the result you expect by default? Or is this just one of many unlikely scenarios (like Hanson's 'The Age of Em') that are worth considering?
Yes, the future of humanity being a good place to live (within its resource constraints) follows from it being cheap for superintelligence to ensure (given that it's decided to let it exist at all), while the constraint of permanent disempowerment (at some level significantly below all of cosmic endowment) is a result of not placing the future of humanity at the level of superintelligence's own interests. Maybe there's 2% for actually capturing a significant part of the cosmic endowment (the eutopia outcomes), and 20% for extinction. I'm not giving s-risks much credence, but maybe they still get 1% when broadly construed (any kind of warping in the future of humanity that's meaningfully at odds with what humanity and even individual humans would've wanted to happen on reflection, given the resource constraints to work within).
I should also clarify that by "making it harmless" I simply mean the future of humanity being unable to actually do any harm in the end, perhaps through lacking direct access to the physical level of the world. The point is to avoid negative externalities for the hosting superintelligence, so that the necessary sliver of compute stays within budget. This doesn't imply any sinister cognitive changes that make the future of humanity incapable of considering the idea or working in that direction.
1Many parts of this argument seem predicated on the assumption that intelligence, at any level, is “greedy”: seeking more and more resources. Yet most human philosophies and traditions of wisdom embrace balance, sobriety, and detachment from want.
What is the evidence to suggest that intelligent life can’t ever be expected to think itself out of constantly needing more stuff? What’s the evidence to suggest that intelligence has to be synonymous with western-style extractive, exploitative, and (pardon the loaded term) colonialist thinking?
This might have been proven in some other thread, and I’m sorry since I’m coming late to the party and haven’t done all the required reading. But I’m asking because genuinely curious: is the need to reproduce in our cells, and would a non-cellular intelligence, without “life”, feel the need to stay alive and therefore grow and reproduce?
There might be diversity in this sense.
But the non-greedy part of the population is unlikely to enforce non-greediness on the rest and is likely to be outcompeted and outproliferated by the greedy ones.
A lot more coordination would be needed to moderate greediness on the global level.
The average CEO of a trillion dollar company is generally less ruthless than a mid-level drug dealer, or than the typical warlord. This implies that there is some reason that being maximally self-interested is not a strategy which outcompetes all other strategies at human levels. I don't see why human level is special here, as long as multipolarity holds.
Is not this because of a lot of coordination, though? (I mean, the existence of relatively civilized states, their laws, and so on. For example, if one considers Russia, I am not sure that CEOs of largest orgs are typically less ruthless. So, yes, if AIs form a “decent society”, their (and everyone’s else) chances will be much better.)
Do you put any probability on "superintelligence is uninterested in autonomy"? It may find us humans much much more interesting than we do. It might want to observe how far we (Humans + AI) go much more than how far it can go.
Are you in full agreement with Instrumental convergence?
Curated and popular this week