[cross-posted from EAF]
Agreed that extreme power concentration is an important problem, and this is a solid writeup.
Regarding ways to reduce risk: My favorite solution (really a stopgap) to extreme power concentration is to ban ASI [until we know how to ensure it's safe], a solution that is notably absent from the article's list. I wrote more about my views here and about how I wish people would stop ignoring this option. It's bad that the 80K article did not consider what is IMO the best idea.
1Thanks for the comment Michael.
A minor quibble is that I think it's not clear you need ASI to end up with dangerous levels of power concentration, so you might need to ban AGI, and to do that you might need to ban AI development pretty soon.
I've been meaning to read your post though, so will do that soon.
I think that to ban ASI you'd need to ban something like AGI because of intelligence explosion dynamics, so not clear it makes a big difference.
Extreme power concentration was supposed to rely on the AIs being used for most cognitive work. In theory, one could develop the AIs and have them used only for things like automated teaching which don't undermine human potential or the bargaining power which the humans have.
There are also galaxy-brained arguments that power concentration is fine/good (because it’s the only way to stop AI takeover, or because any dictator will do moral reflection and end up pursuing the good regardless).
I think the most salient argument for this (which is brought up in the full article) is that monopolization of power solves the proliferation problem. If the first ASI actors perform a pivotal act to preemptively disempower unapproved dual-use AIs, we don’t need to worry much about new WMDs or existing ones falling in price.
If AI enabled offense-dominant tech exists, then you need to do some minimum amount of restriction on the proliferation of general superintelligence, and you need enforcement power to police those restrictions. Therefore, some concentration of power is necessary. What's more, you likely need quite a lot becuase 1. preventing the spread of ASI would be hard and get harder the more training costs fall, and 2. you need lots of strategic power to prevent extractive bargaining and overcome deterrence against your enforcement.
I think the important question to ask at that point is how we can widen political control over a much smaller number of intent-aligned AIs, as opposed to distributing strategic power directly and crossing our fingers that the world isn’t vulnerable.
Thanks, I think these are interesting points.
I agree that some power concentration is likely necessary, and that it could be a lot, though I'm pretty unsure there.
In terms of what to do about that:
- Do you have ideas about what widening political control over a small number of AIs would look like?
- Another alternative to crossing our fingers would be to distribute strategic power but also build AI resilience. This doesn't work if there are big capability gaps I think, but if capabilities are relatively evenly distributed then there might be ways to build defensive enclaves
You're right that there are ways to address proliferation other than to outright restrict the underlying models (such as hardening defensive targets, bargaining with attackers, or restricting the materials used to make asymmetric weapons). These strategies can look attractive either because we inevitably have to use them (if you think restricting proliferation is impossible) or because they require less concentration of power.
Unfortunately, each of these strategies are probably doomed without an accompanying nonproliferation regime.
1. Hardening - The main limitation of defensive resilience is that future weapons will be very high impact, and that you will need to be secure against all of them. Tools like mirror life can plausibly threaten everyone on Earth, and we'd need defense dominance against not just it, but every possible weapon that superintelligences can cheaply design before they can be allowed to be widely proliferated. It strikes me as very unlikely that there will happen to be defense-dominant solutions against every possible superweapon, especially solutions that are decentralized and don't rely on massive central investment anyways.
Although investing in defense against these superweapons is still a broadly good idea because it raises the ceiling on how powerful AIs will have to be before they have to be restricted (ie, if there are defense-dominant solutions against mirror life but not insect-sized drones, you can at least proliferate AIs capable of designing only the first and capture their full benefits), it doesn't do away with the need to restrict the most powerful/general AIs.
And even if universal defense dominance is possible, it's risky to bet on ahead of time, because proliferation is an irreversible choice: once powerful models are out there, there will be no way to remove them. Because it will take time to ensure that proliferation is safe (the absolute minimum being the time it takes to install defensive technologies everywhere) you still inevitably end up with a minumum period where ASIs are monopolized by the government and concentration of power risks exist.
2. Bargaining - MAD deterrence only functions for today's superweapons because the number of powerful actors is very small. If general superintelligence democratizes strategic power through making superweapons easier to build, then you will eventually have actors interested in using them (terrorists, misaligned ASIs) or such a large number of rational self-interested actors that private information, coordination problems, or irreconcilable values that superweapons eventually get deployed regardless.
3. Input controls - You could also try to limit inputs to future weapons, like we do today by limiting gene samples and fissile material. Unfortunately, I think future AI-led weapons R&D will not only increase the destructive impact of future weapons (bioweapons -> mirror life) but also make them much cheaper to build. The price of powerful weapons is probably completely orthogonal to their impact: the fact that nukes costs billions and blow up a single city makes no difference to the fact that an engineered bioweapon could much more cheaply kill hundreds of millions or billions of people.
If asymmetric weapons are cheap enough to make, then the effort required to police their inputs might be much greater than just restricting AI proliferation in the first place (or performing some pivotal act early on). For example, if preventing mirror life from existing requires monitoring every order and wet lab on earth (including detecting hidden facilities) then you might as well have used that enforcement power to limit access to unrestricted superintelligence in the first place.
----
Basically, I think that defensive reslience has a place, but doesn't stand on its own. You'll still need to have some sort of centralized effort (probably by the early ASI states) to restrict proliferation of the most powerful models, because those models are capable of cheaply designing high impact and asymmetric weapons that can't be stopped through other means. This nonproliferation effort has to be actively enforced (such as by detecting and disabling unapproved training runs adversarially) which means that the government needs enforcement power. In particular, it needs enough enforcement power to either a) continually expand its surveillance and policing in response to falling AI training costs, or b) it needs enough to perform an early pivotal act. You can't have this enforcement power without a monopoly/oligopoly over the most powerful AIs, because without it there's no monopoly on violence.
Therefore, the safest path (from a security perspective) is fewer intent-aligned superintelligences. In my view, this ends up being the case pretty much by default: the US and China follow their national-security incentives to prevent terrorism and preserve their hegemony, using their technological lead to box out competitors from ever developing AIs with non-Sino-US alignments.
From there, the key questions for someone interested in gradual disempowerment are:
1. How is control over these ASIs' goals distributed?
2. How bad are the outcomes if they're not distributed?
For (1), I think the answer likely involves something like representative democracy, where control over the ASI is grafted onto our existing institutions. Maybe congress collectively votes on its priorities, or the ASI consults digital voter proxies of all the voters it represents. Most of the risk of a coup comes from early leadership during the development of an ASI project, so any interventions that increase the insight/control the legistlative branch has relative to the executive/company leaders seem likelier to result in an ASI created without secret loyalties. You might also avoid this by training AIs to follow some values deontologically, which ends up persisting through the period where they become superintelligent.
Where I feel more confident is (2), based on my beliefs that future welfare will be incredibly cheap and that s-risks are very unlikely. Even in a worst-case concentration of power scenario where one person controls the lightcone, I expect that the amount of altruism they would need to ensure everyone on earth very high welfare lives would be very small, both because productive capacity is so high and because innovation has reduced the price of welfare to an extremely low level. The main risk of this outcome is that it limits upside (ie, an end to philosophy/moral progress, lock-in of existing views) but it seems likely to cap downside at a very high level (certainly higher than the downsides of unrestricted proliferation, which is mass death through asymmetric weapons).
I'm noticing that almost all of these comments are cross-posted between here and EAF. It would be nice if there were a way to have comments automatically show up on both. (I think this already exists for LW <> Alignment Forum?)
1Given very different norms of LW and EAF I don’t really know how to make this work. I think it’s kind of rare that people want to crosspost comments, and the risks from making it easy outweigh the benefits.
[cross-posted from EAF]
Thanks for writing this!!
This risk seems equal or greater to me than AI takeover risk. Historically the EA & AIS communities focused more on misalignment, but I'm not sure if that choice has held up.
Come 2027, I'd love for it to be the case that an order of magnitude more people are usefully working on this risk. I think it will be rough going for the first 50 people in this area; I expect there's a bunch more clarificatory and scoping work to do; this is uncharted territory. We need some pioneers.
People with plans in this area should feel free to apply for career transition funding from my team at Coefficient (fka Open Phil) if they think that would be helpful to them.
1(Crossposted from EAF)
Nice write-up on the issue.
One thing I will say is that I'm maybe unusually optimistic on power concentration compared to a lot of EAs/LWers, and the main divergence I have is that I basically treat this counter-argument as decisive enough to make me think the risk of power-concentration doesn't go through, even in scenarios where humanity is basically as careless as possible.
This is due to evidence on human utility functions showing that most people have diminishing returns on utility on exclusive goods to use personally that are fast enough that altruism matters much more than their selfish desires on stellar/galaxy wide scales, combined with me being a relatively big believer in quite a few risks like suffering risks being very cheap to solve via moral trade where most humans are apathetic on.
More generally, I've become mostly convinced of the idea that a crucial positive consideration on any post-AGI/ASI future is that it's really, really easy to prevent most of the worst things that can happen in those futures under a broad array of values, even if moral objectivism/moral realism is false and there isn't much convergence on values amongst the broad population.
Edit: I edited in a link.
11Suppose that the humans do have diminishing returns of utility functions. Unfortunately, existing combination of instincts and moral intuitions do not prompt the majority of humans to help the poor, especially those who are far from potential helpers' set of friends[1], with nearly anything. And those who do so are unlikely to stay in power or were unlikely to receive fortunes or occupy relevant positions.
- ^
Friends are also likely to be in the same class as the potential helpers.
I recently wrote 80k’s new problem profile on extreme power concentration (with a lot of help from others - see the acknowledgements at the bottom).
It’s meant to be a systematic introduction to the risk of AI-enabled power concentration, where AI enables a small group of humans to amass huge amounts of unchecked power over everyone else. It’s primarily aimed at people who are new to the topic, but I think it’s also one of the only write-ups there is on this overall risk,[1]so might be interesting to others, too.
Briefly, the piece argues that:
That’s my best shot at summarising the risk of extreme power concentration at the moment. I’ve tried to be balanced and not too opinionated, but I expect many people will have disagreements with the way I’ve done it. Partly this is because people haven’t been thinking seriously about extreme power concentration for very long, and there isn’t yet a consensus way of thinking about it. To give a flavour of some of the different views on power concentration:
So you shouldn’t read the problem profile as an authoritative, consensus view on power concentration - it’s more a waymarker, my best attempt to give an interim overview of a risk which I hope we will develop a much clearer understanding of, hopefully soon.
Some salient things about extreme power concentration that I wish we understood better:
(For more musings on power concentration, you can listen to this podcast, where Nora Ammann and myself discuss our different takes on the topic.)
If you have thoughts on any of those things, please comment with them! And if you want to contribute to this area, consider:
Thanks to Nora Ammann, Adam Bales, Owen Cotton-Barratt, Tom Davidson, David Duvenaud, Holden Karnofsky, Arden Koehler, Daniel Kokotajlo, and Liam Patell for a mixture of comments, discussion, disagreement, and moral support.
I think AI-enabled coups, gradual disempowerment and the intelligence curse are the best pieces of work on power concentration so far, but they are all analysing a subset of the scenario space. I’m sure my problem profile is, too - but it is at least trying to cover all of the ground in those papers, though at a very high level. ↩︎
A few different complaints about the distinction that I’ve heard:
(This is just an opportunistic breakdown based on the papers I like. I’d be surprised if it’s actually the best way to carve up the space, so probably there’s a better version of this question.) ↩︎
This is a form run by Forethought, but we’re in touch with other researchers in the power concentration space and intend to forward people on where relevant. We’re not promising to get back to everyone, but in some cases we might be able to help with funding, mentorship or other kinds of support. ↩︎