I wish I could believe that.
To start off, I agree with pretty much all of it. It's unlikely that any of the main players actively want the world to end, and inasmuch as they'll bring that outcome about, it'll be by mistake. It's marginally more likely that some of them are risking the world in the pursuit of personal power/status/ideology, and monstrously consider everyone's deaths an acceptable risk, but I'm not certain of that either.
That said, I agree that we could, in principle, all cooperate here. The thing often touted about as a counterargument is "but China". Except China doesn't want to die any more than the US, and there isn't, in principle, any reason the Chinese government can't be convinced of the seriousness of the danger. They would believe in the reality of an asteroid on a collision course with Earth if shown the evidence, and the AGI threat is no less real.
What we all can cooperate towards is to ban AGI development, and seek other, more controllable and less unilateral ways to create superintelligences. Human cognitive enhancement, genetic engineering, uploads, whatever.
What we can all do, together, is avert the omnicide. It's in ~no-one's interests.
What I don't believe is that we could cooperate on building an AGI that would bring about an utopia.
The term "AI Alignment" is a bit obfuscatory. The technical problem of AI Alignment would be better termed the control problem. An aligned AGI isn't necessarily an AGI that does the best for humanity; an aligned AGI is one that does precisely what its designer(s) intended for it to.
And the utopias of the majority of people, or companies, or governments, across history and across the world today, would be death or a hellscape for most other people. We would not enjoy life in a world in which North Korea, or a corporate sociopath, or Trump's US, or some bureaucracy, solved AGI alignment.
Again, I would like to empathize that "that means we need to outrace everyone else" is not the correct takeaway from this.
- Firstly, who are "we"? There is no person or a group of people Humanity can trust with doing it right.
- No-one who is currently racing can be trusted. A US company successfully solving alignment isn't particularly more likely to result in a pro-humanity utopia, or an utopia for you, than the Chinese doing it first.
- No-one who may start racing can be trusted. The US military nationalizing the project isn't going to result in a good outcome, either. And if we set things up such that the AI's value system is being decided as a matter of national policy, via a democratic process? Where people discuss it among themselves, with politicians chiming in, and maybe then vote on it, and then the result is interpreted by some bureaucratic process, and then someone types it in? Don't make me laugh. We'd be lucky if the AI that'd result from this just kills everyone and tiles the universe with paperwork, rather than trapping everyone in some inescapable Kafkaesque nightmare.
- (To be clear, it's not because the people will want bad things. It's because our processes for eliciting and agglomerating their preferences – any and all processes in wide use – are an abomination.)
- To do it right, whatever process builds the AGI would need to be actively, desperately trying not to leave its fingerprints on the future. But I trust pretty much no-one and no thing to actually do that, instead of hijacking the future for their values. ... Except myself and a few specific people, of course. But I rather doubt that you or the people of Indonesia or humanity as a whole would feel particularly enthusiastic about handing off that decision to me, yeah?
- Same for any other candidature. There is nobody to whom Humanity as a whole can entrust its future; nobody to whom it should feel comfortable deferring that decision.
- Secondly, that'd be giving up too early. None of the above invalidates the core argument: even if we can't agree on an utopia, pretty much all of us would prefer to keep things as they are, keep incrementally improving our conditions and painstakingly negotiating compromises, to all of us just dying overnight. And the policy of "so we need to outrace the competitors before they build a hell" doesn't actually lead to your utopia, because you are not going to solve alignment in those conditions either. That policy leads to death. Which, again, ~nobody wants.
So I agree that we can all cooperate on this, that the current state of affairs is a mistake, and that we can negotiate for a better outcome. Ban AGI research internationally. Keep advancing using other technologies.
We will still need to reconcile our differences later on, of course. But it can be done incrementally, a steady pace of negotiation and power balancing and cultural mingling and sanity-raising. There are routes of intelligence enhancement that are more gradual, that let us preserve this sort of incrementalism and stability while still letting Humanity keep empowering itself. Gradual intelligence augmentation, via biotechnological or cyborgism-like or upload-based means.
Humanity-as-a-whole can't entrust its future to any given part of itself. But it can still build a future for itself. There doesn't have to be a singular point in time at which we are deciding the whole shape of the future and are then unable to backtrack.
Bottom line: As things stand, anyone anywhere solving either AGI or AGI alignment is not, on expectation, going to lead to a good outcome for humanity. Our processes are too dysfunctional:
- We can't trust each other to let each other solve alignment in peace, and–
- – we can't survive if we let that distrust get to us and start racing each other, because then none of us solve alignment and we all die.
The best outcome we can all cooperate towards – and that is a good outcome that we can all cooperate towards – is to ban the accursed thing.
2
1
There's one thing history seems to have been trying to teach us: that the contents of the future are determined by power, economics, politics, and other conflict-theoritic matters.
Turns out, nope!
Almost all of what the future contains is determined by which of the two following engineering problems is solved first:
…and almost all of the reasons that the former is currently a lot more likely are mistake theory reasons.
The people currently taking actions that increase the probability that {the former is solved first} are not evil people trying to kill everyone, they're confused people who think that their actions are actually increasing the probability that {the latter is solved first}.
Now, sure, whether you're going to get a chance to talk with OpenAI/Deepmind/Anthropic's leadership enough to inform them that they're in fact making things worse is a function of economics and politics and the like. But ultimately, for the parts that really matter here, this is a matter of explaining, not of defeating.
And, sure, the implementation details of "utopia" do depend on who launches the aligned superintelligent AI, but I expect you'd be very happy with the utopia entailed by any of the possibilities currently on the table. The immense majority of the utility you're missing out on is from getting no utopia at all and everyone dying forever, rather than getting the wrong utopia implementation details.
The reason that the most likely outcome is that everyone dies forever, is that the people who get to impact which of those outcomes is going to happen are mistaken (and probably not thinking hard enough about the problem to realize that they're mistaken).
They're not evil and getting them to update to the correct logical beliefs is a matter of reason (and, if they're the kind of weak agents that are easily influenced by what others around them think, memetics) rather than a matter of conflict.
They're massively disserving everyone's interests, including their own. And the correct actions for them to take would massively serve their own interests as well as everyone else's. If AI kills everyone they'll die too, and if AI creates utopia they'll get utopia along with everyone else — and those are pretty much the only two attractors.
We're all in this together. Some of us are just fairly confused, not agentically pursuing truth, and probably have their beliefs massively biased by effects such as memetics. But I'm pretty sure nobody in charge is on purpose trying to kill everyone; they're just on accident functionally trying to kill everyone.
And if you're not using your power/money to affect which of those two outcomes is more likely to happen than the other, then your power/money is completely useless. They won't be useful if we all die, and they won't be useful if we get utopia. The only use for resources, right now, if you want to impact in any way what almost all of the future contains (except for maybe the next 0 to 5 years, which is about how long we have), is in influencing which of those two engineering problems is solved first.
This applies to the head of the major AI orgs just as much as it applies to everyone else. One's role in an AI org is of no use whatsoever except for influencing which of those two problems are solved first. The head of OpenAI won't particurly get a shinier utopia than everyone else if alignment is solved in time, and they won't particularly die less than everyone else if it isn't.
Power/money/being-the-head-of-OpenAI doesn't do anything post-singularity. The only thing which matters, right now, is which of those two engineering problems is solved first.