Crossposted from the AI Alignment Forum. May contain more technical jargon than usual.
If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well, they've probably found a way to keep their AGI weak enough that it isn’t very useful.
What about AGI that is basically just virtual humans?
For what it’s worth, Eliezer in 2018 said that he’d be pretty happy with that:
If the subject is Paul Christiano, or Carl Shulman, I for one am willing to say these humans are reasonably aligned; and I'm pretty much okay with somebody giving them the keys to the universe in expectation that the keys will later be handed back.
(Obviously “Eliezer in 2018” ≠ “Nate today”; Nate can chime in if he disagrees with the above.)
Incidentally, I’ve shown the above quote to a lot of people who say “yes that’s perfectly obvious”, and I’ve also shown this quote to a lot of people who say “Eliezer is being insufficiently cynical; absolute power corrupts absolutely”. For my part, I don’t have a strong opinion, but on my models, if we know how to make virtual humans, then we probably know how to make virtual humans without envy and without status drive and without teenage angst etc., which should help somewhat. More discussion here.
Yeah largely agree (and with the linked post) .. but status drive seems likely heavily entangled with empowerment in social creatures. For example I recall even lobsters have a simple detector of social status (based on some serotonin signaling mechanism), and since they compete socially for resources, social status is a strong predictor of future optionality and thus an empowerment signal.
Also agree that AGI will likely be (or appear) conscious/sentient the way we are (or appear), and that's probably impossible to avoid without trading off generality/capability. EY seems to have just decided earlier on that since conscious AGI is problematic, it shan't be so.
Corruption-by-power (and related issues) seem like problems worth thinking about here. Though they also strike me as problems that humans tend to be very vigilant about / concerned with by default, and problems that become a lot less serious if you've got a lot of emulated copies of different individuals, rather than just copies of a single individual.
that's probably impossible to avoid without trading off generality/capability
You need to trade off some generality/capability anyway for the sake of alignment. One hope (though not the only one) might be that there's overlap between the capabilities we want to remove for the sake of alignment, and the ones we want to remove for the sake of reducing-the-risk-that-the-AGI-is-conscious.
E.g., if you want your AGI to build nanotech for you and do nothing else, then you might want to limit its ability to think about itself, or its operators, or the larger world, or indeed anything other than different small-scale physical structures. Limiting its generality and self-awareness in this way might also be helpful for reducing the risk that it's conscious.
EY seems to have just decided earlier on that since conscious AGI is problematic, it shan't be so.
Where has EY said that he's confident the first AGI systems won't be conscious?
E.g., if you want your AGI to build nanotech for you and do nothing else, then you might want to limit its ability to think about itself, or its operators, or the larger world, or indeed anything other than different small-scale physical structures. Limiting its generality and self-awareness in this way might also be helpful for reducing the risk that it's conscious.
I don't quite get this example.
How could such a system build nanotech efficiently without it having those properties? Wouldn't it need a human operator the moment it encountered unexpected phenomena?
If so, it just seems like a really fancy hammer and not an 'AGI'
Wouldn't that require solving alignment in itself, though? If you can simulate virtual humans, complete with human personalities, human cognition, and human values, then you've already figured out how to plug human values straight into a virtual agent.
If you mean that the AGI is trained on human behavior to the point where it's figured out human values through IRL/predictive coding/etc. and is acting on them, then that's also basically just solving alignment.
However, if you're suggesting brain uploads, I highly doubt that such technology would be available before AGI is developed.
All that is to say that, while an AGI that is basically just virtual humans would probably be great, it's not a prospect we can depend on in lieu of alignment research. Such a result could only come about through actually doing all the hard work of alignment research first.
Wouldn't that require solving alignment in itself, though?
Yes, but only to the same extent that evolution did. Evolution approximately solved alignment on two levels: aligning the brain with the evolutionary goal of inclusive fitness[1], and aligning individual brains (as disposable somas) with other brains (shared kin genes) via altruism (the latter is the thing we want to emulate).
Massively successful, population of 10B vs a few M for all other great apes. It's fashionable to say evolution failed at alignment: this is just stupidly wrong, humans are an enormous success from the perspective of inclusive fitness. ↩︎
Do you propose using evolutionary simulations to discover other-agent-aligned agents? I doubt we have the same luxury of (simulated) time that evolution had in creating humans. It didn't have to compete against an intelligent designer; alignment researchers do (i.e., the broader AI community).
I agree that humans are highly successful (though far from optimal) at both inclusive genetic fitness and alignment with fellow sapients. However, the challenge for us now is to parse the system that resulted from this messy evolutionary process, to pull out the human value system from human neurophysiology. Either that, or figure out general alignment from first principles.
Do you propose using evolutionary simulations to discover other-agent-aligned agents?
Nah. The wright brothers didn't need to run evo sims to reverse engineer flight. They just observed how birds bank to turn, how that relied on wing warping, and said - cool, we can do that too! Deep learning didn't succeed through brute force evo sims either (even though Karl Sim's evo sims work is pretty cool, it turns out that loose reverse engineering is just enormously faster).
However, the challenge for us now is ... to pull out the human value system from human neurophysiology. Either that, or figure out general alignment from first principles.
Sounds about right. Fortunately we may not need to model human values at all in order to build general altruistic agents: it probably suffices that the AI optimizes for human empowerment (our ability to fulfill any long term future goals, rather than any specific values), which is a much simpler and more robust target and thus probably more long term stable.
I prefer "AI Safety" over "AI Alignment" because I associate the first more with Corrigibility, and the second more with Value-alignment.
It is the term "Safe AI" that implies 0% risk, while "AI Safety" seems more similar to "Aircraft Safety" in acknowledging a non-zero risk.
I agree that corrigibility, task AGI, etc. is a better thing for the field to focus on than value learning.
This seems like a real cost of the term "AI alignment", especially insofar as researchers like Stuart Russell have introduced the term "value alignment" and used "alignment" as a shorthand for that.
If we’re just discussing terminology, I continue to believe that “AGI safety” is much better than “AI safety”, and plausibly the least bad option.
Of course, “AGI safety” isn’t perfect either. How can it be abused?
If using the label “AI safety” for this problem causes us to confuse a proxy goal (“safety”) for the actual goal “things go great in the long run”, then we should ditch the label. And likewise, we should ditch the term if it causes researchers to mistake a hard problem (“build an AGI that can safely end the acute risk period and give humanity breathing-room to make things go great in the long run”) for a far easier one (“build a safe-but-useless AI that I can argue counts as an ‘AGI’”).
Sometimes I talk about “safe and beneficial AGI” (or more casually, “awesome post-AGI utopia”) as the larger project, and “AGI safety” as the part where we try to make AGIs that don’t kill everyone. I do think it’s useful to have different terms for those.
See also: Safety ≠ alignment (but they’re close!)
What is the current biggest bottleneck to an alignment solution meeting the safety bar you've describe here (<50% chance of killing more than a billion)?
I'd guess Nate might say one of:
Current SotA systems are very opaque — we more-or-less can't inspect or intervene on their thoughts — and it isn't clear how we could navigate to AI approaches that are far less opaque, and that can carry forward to AGI. (Though it seems very likely such approaches exist somewhere in the space of AI research approaches.)
Yeah, it does seem like interpreterability is a bottleneck for a lot of alignment proposals, and in particular as long as neutral networks are essentially black boxes, deceptive alignment/inner alignment issues seem almost impossible to address.
50% chance of killing everyone (almost) isn't a thing. it either preserves ~all of humanity or none; there is almost no middle ground. if it's good enough at discovering agency, it protects almost all of humanity, quickly converging to all - the only losses would be, well, losses; if it's not good enough at discovering and protecting agency, it obliterates other species and takes over as the dominant species. yudkowsky is terrified that it's just going to take over as the dominant species and eat us all; reasonable fear - but like we're only going to have near human level ai for another year or two now that we've got it, only a tiny sliver of possible aligned systems are good enough at discovering nearby agency and coordinating near-perfect coprotection systems to not eat us all, but still not aligned enough to eat none of us.
the key thing to remember is that we are creating a dramatically more fit species, and we are still unsure if we're going to manage to get them to give a shit about the other species that came before them in any sort of durable way. it seems like they could! but it also seems like the most adaptive forms of this new species may evolve shockingly fast and quickly play reproductive defect against all other life. since if that happens it would be an event that could easily wipe out anything not playing as hard, we need to figure out how to prevent incremental escalation.
idk, my take is we're closer than y'all worrywarts think to the 50% of people ai, and I think you should be a lot more worried about going back to 100% because some humans try to stick with the 50% ai.
(y'all should stop using words like "disassemble", btw, imo. when there's a concept more people will intuitively see as meaning what you intend, it's good to use it, imo.)
50% chance of killing everyone (almost) isn't a thing. it either preserves ~all of humanity or none; there is almost no middle ground.
To take a stupid example, one could imagine that the deep neural network initialization has a random seed, and for half of possible seeds, the AGI preserves all of humanity, and for the other half of seeds, it preserves none of humanity.
Are you telling me you'd be okay with releasing an AI that has a 25% chance of killing over a billion people, and a 50% chance of at least killling hundreds of millions? I have to be missing the point here, because this post isn't doing anything to convince me that AI researchers aren't Stalin on steroids.
Or are you saying that if one can get to that point, it's much easier from there to get to the point of having an AI that will cause very few fatalities and is actually fit for practical use?
Rather, I think he means that alignment is such a narrow target, and the space of all possible minds is so vast, that the default outcome is that unaligned AGI becomes unaligned ASI and ends up killing all humans (or even all life) in pursuit of its unaligned objectives. Hitting anywhere close to the alignment target (such that there's at least 50% chance of "only" one billion people dying) would be a big win by comparison.
Of course, the actual goal is for “things [to] go great in the long run”, not just for us to avoid extinction. Alignment itself is the target, but safety is at least a consolation prize.
So no, I don't think Nate, Eliezer, or anyone else is okay with releasing an AI that would kill hundreds of millions of people. But AGI is coming, whether we want it or not, and it will not be aligned with human survival (much less human flourishing) by default.
Eliezer tends to think that solving alignment is so much more difficult and so much less researched than raw AGI that doom is almost certain. I'm a bit more optimistic, but I agree that minimizing the probable magnitude of the doom is better than everyone dying.
Or are you saying that if one can get to that point, it's much easier from there to get to the point of having an AI that will cause very few fatalities and is actually fit for practical use?
Also this.
Feels like Y2K: Electric Boogaloo to me. In any case, if a major catastrophe did come of the first attempt to release an AGI, I think the global response would be to shut it all down, taboo the entire subject, and never let it be raised as a possibility again.
The tricky thing with human politics is that governments will still fund research into very dangerous technology if it has the potential to grant them a decisive advantage on the world stage.
No one wants nuclear war, but everyone wants nukes, even (or especially) after their destructive potential has been demonstrated. No one wants AGI to destroy the world, but everyone will want an AGI that can outthink their enemies, even (or especially) after its power has been demonstrated.
The goal, of course, is to figure out alignment before the first metaphorical (or literal) bomb goes off.
On that note, the main way I could envision AI being really destructive is getting access to a government's nuclear arsenal. Otherwise, it's extremely resourceful but still trapped in an electronic medium; the most it could do if it really wanted to cause damage is destroy the power grid (which would destroy it too).
It's explicitly the second:
But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
(Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)
I don't usually go around saying that I care about AI "safety". I go around saying that I care about "alignment" (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).
But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it's treating one part of the problem (making an AGI "safe") as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.
Following Eliezer, I think of an AGI as "safe" if deploying it carries no more than a 50% chance of killing more than a billion people:
Notably absent from this definition is any notion of “certainty” or "proof". I doubt we're going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).
On my models, making an AGI "safe" in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it's probably not that hard to drive the success probability significantly higher than that.
If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they've probably... well, they've probably found a way to keep their AGI weak enough that it isn’t very useful. But if they can do that with an AGI capable of ending the acute risk period, then they've probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
The condition that the AI actually be useful for pivotal acts is an important one. We can already build AI systems that are “safe” in the sense that they won’t destroy the world. The hard part is creating a system that is safe and relevant.
Another concern with the term “safety” (in anything like the colloquial sense) is that the sort of people who use it often endorse the "precautionary principle" or other such nonsense that advocates never taking on risks even when the benefits clearly dominate.
In ordinary engineering, we recognize that safety isn’t infinitely more important than everything else. The goal here is not "prevent all harms from AI", the goal here is "let's use AI to produce long-term near-optimal outcomes (without slaughtering literally everybody as a side-effect)".
Currently, what I expect to happen is that humanity destroys itself with misaligned AGI. And I think we’re nowhere near knowing how to avoid that outcome. So the threat of “unsafe” AI indeed looms extremely large—indeed, this seems to be rather understating the point!—and I endorse researchers doing less capabilities work and publishing less, in the hope that this gives humanity enough time to figure out how to do alignment before it’s too late.
But I view this strategic situation as part of the larger project “cause AI to produce optimal long-term outcomes”. I continue to think it's critically important for humanity to build superintelligences eventually, because whether or not the vast resources of the universe are put towards something wonderful depends on the quality and quantity of cognition that is put to this task.
If using the label “AI safety” for this problem causes us to confuse a proxy goal (“safety”) for the actual goal “things go great in the long run”, then we should ditch the label. And likewise, we should ditch the term if it causes researchers to mistake a hard problem (“build an AGI that can safely end the acute risk period and give humanity breathing-room to make things go great in the long run”) for a far easier one (“build a safe-but-useless AI that I can argue counts as an ‘AGI’”).