This is part of a series of lists of projects. The unifying theme is that the projects are not targeted at solving alignment or engineered pandemics but still targeted at worlds where transformative AI is coming in the next 10 years or so. See here for the introductory post.
Commonly discussed motivations cited for why rapid AI progress might be scary are:
But even aside from these risks, it seems likely that advanced AI will lead to explosive technological and economic growth across the board, which could lead to a large number of problems emerging at a frighteningly fast pace.[1]
The growth speed-ups could be extreme. The basic worry is that AI would let us return to a historical trend of super-exponential growth.[2] If this happens, I don’t know any reassuring upper limit for how fast growth could go.
(Illustratively: Paul Christiano’s suggested definition for slow takeoff is “There will be a complete 4 year interval in which world output doubles, before the first 1 year interval in which world output doubles.” If world GDP doubled in a single year, that would mean that growth was ~30x faster than it is right now.)
If technological growth speeds up by, say, 30x, then that suggests that over just a few years, we might have to deal with all the technologies that would (at a “normal” pace) be discovered over 100 years. That’s an intense and scary situation.
This section is about problems that might arise in this situation and governance solutions that could help mitigate them. It’s also about “meta” solutions that could help us deal with all of these issues at once, e.g. by improving our ability to coordinate to slow down the development and deployment of new technologies.
Note: Many of the projects in this section would also be useful for alignment. But I’m not covering any proposals that are purely focused on addressing alignment concerns.
I think there’s substantial value to be had in vetting and describing the case for explosive growth, as well as describing why it could be terrifying. Explosive growth underlies most of the concerns in this section — so establishing the basic risk is very important.
Note: There’s some possible backfire risk, here. Making a persuasive case for explosive growth could motivate people to try harder to get there even faster. (And in particular, to try to get there before other actors do.) Thereby giving humanity even less time to prepare.
I don’t think that’s a crazy concern. On the other hand, it’s plausible that we’re currently in an unfortunate middle ground, where everyone already believes that frontier AI capabilities will translate into a lot of power, but no one expects the crazy-fast growth that would strike fear into their hearts.
On balance, my current take is that it’s better for the world to see what’s coming than to stumble into it blindly.
Related/previous work:
Examples of how to attack this problem:
Although a fast intelligence explosion would be terrifying, the resulting technology could also be used to create a fantastic world. It would be great to be able to combine (i) appropriate worry about a poorly handled intelligence explosion and (ii) a convincing case for how everyone could get what they want if we just coordinate. That would make for a powerful case for why people should focus on coordinating. (And not risk everything by racing and grabbing for power.)
For a related proposal and some ideas about how to go about it, see Holden Karnofsky’s proposal here.
For some previous work, see the Future of Life Institute’s worldbuilding contest (and follow-up work).
Here are three concrete areas that might need solutions in order for humanity to build an excellent world amidst all the new technology that might soon be available to us. Finding policy solutions for these could:
(“Vulnerable world hypothesis” is in reference to this paper.)
If too many actors had access to incredibly destructive tech, we’d probably see a lot of destruction. Because a small fraction of people would choose to use it.
Unfortunately, a good heuristic for what we would get from rapid technological growth is: cheaper production of more effective products. This suggests that sufficiently large technological growth would enable cheap and convenient production of e.g. even more explosive nukes or deadlier pandemics.
Will technology also give us powerful defenses against these technologies? I can easily imagine technological solutions to some issues, e.g. physical defenses against the spread of deadly pandemics. But for others, it seems more difficult. I don’t know what sort of technology would give you a cheap and convenient defense against a nuclear bomb being detonated nearby.
One solution is to prevent the problem at its source: By preventing access to the prerequisite materials or technologies and/or by monitoring people who have the capacity to cause large amounts of destruction.
But in some scenarios, almost anyone could have the capacity to cause widespread destruction. So the monitoring might have to be highly pervasive, which would come with significant costs and risks of misuse.
Exploring potential solutions to this problem hasn’t really been done in depth. It would be great to find solutions that minimize the harms from both destructive technologies and from pervasive surveillance.[3]
Examples of how to attack this problem:
Historically, we’ve taken on large risks from brinkmanship around nukes. For example, during the Cuban missile crisis, President Kennedy thought that the risk of escalation to war was "between 1 in 3 and even".
I would expect similar risks to go up substantially at a time when humanity is rapidly developing new technology. This technology will almost certainly enable new, powerful weapons with unknown strategic implications and without any pre-existing norms about their use.
In addition, AI might enable new commitment mechanisms.
This could help solve coordination problems. But it could also create entirely new strategies and risks around brinkmanship and threats.
What policies should people adopt in the presence of strong commitment abilities? My impression is that state-of-the-art theory on this isn’t great.
The main recommendation from traditional game theory is “try hard to commit to something crazy before your opponent, and then it will be rational for them to do whatever you want”. That doesn’t seem like the way we want future decisions to be made.
There are some alternative theoretical approaches under development, such as open-source game theory. But they haven’t gotten very far. And I don’t know of any ambitious attempts to answer the question of how people ought to behave around this in the real world. Taking into account real-world constraints (around credibility, computation power, lack of perfect rationality, etc.) as well as real-world advantages (such as a shared history and some shared human intuitions, which might provide a basis for successfully coordinating on good norms).
Here’s one specific story for why this could be tractable and urgent: Multi-agent interactions often have a ton of equilibria, and which one is picked will depend on people’s expectations about what other people will do, which are informed about their expectations of other people, etc. If you can anticipate a strategic situation before it arrives and suggest a particular way of handling it, that could change people’s expectations about each other and thereby change the rational course of action.
Examples of how to attack this problem:
Advanced AI could enable dangerously high concentrations of power. This could happen via at least two different routes.
Examples of how to attack this:
(Thanks to Carl Shulman for discussion.)
There are a lot of governance issues raised by the possibility of digital minds. For example, what sort of reform is needed in one-person-one-vote democracies when creating new persons is as easy as copying software? See also Develop candidate regulation from the “Digital Sentience & Rights” post in this series.
Painting a picture of a great outcome suggested outlining an acceptable “endpoint” to explosive growth.
Separately, there’s a question of what the appropriate norms are for how we get from where we are today to that situation.
Other than risks from misaligned AI along the way, I think the 3 central points here are:
What follows are some candidate norms and proposals. Each of them could be:
(For ideas in this section, I’ve especially benefitted from discussion with Carl Shulman, Will MacAskill, Nick Beckstead, Ajeya Cotra, Daniel Kokotajlo, and Joe Carlsmith.)
No “first strike” intelligence explosion
One candidate norm could be: “It’s a grave violation of international norms for a company or a nation to unilaterally start an intelligence explosion because this has a high likelihood of effectively disempowering all other nations. A nation can only start an intelligence explosion if some other company or nation has already started an intelligence explosion of their own or if they have agreement from a majority of other nations that this is in everyone’s best interest.” (Perhaps because they have adequate preparations for reassuring other countries that they won’t be disempowered. Or perhaps because the alternative would be to wait for an even less responsible intelligence explosion from some other country.)
Never go faster than X?
The above proposal was roughly: Only step through an intelligence explosion if you have agreements from other nations about how to do it.
A more opinionated and concrete proposal is: Collectively, as a world, we should probably never grow GDP or innovate technology faster than a certain maximum rate. Perhaps something like: Technological and economic growth should be no more than 5x faster than 2023 levels.
Concrete decision-making proposals
A proposal that would be more opinionated in some ways, and less opinionated in other ways, would be: At some level of AI capabilities, you’re supposed to call for a grand constitutional convention to decide how to navigate the rest of the intelligence explosion, and what to do afterward.
You could make suggestions about this constitutional convention, e.g.:
The “constitutional convention” proposal resembles something like deliberative democracy insofar as the selected people are randomly chosen, and looks more like typical geopolitical deliberations insofar as nations’ governments decide which representatives to send.
(Thanks to Will MacAskill for discussion.)
Technical proposals for slowing down / coordinating
In order to be especially effective, all 3 above proposals require credible technical proposals for how nations could verify that they were all collectively slowing down. Exactly how such proposals should work is a big open problem. Which, thankfully, some people are working on.
I don’t know any detailed list of open questions here. (And I’d be interested if anyone had such a list!) But some places to look are:
Naming some other candidate directions:
Dubiously enforceable promises
Get nations to make promises early on along the lines of “We won’t use powerful AI to offensively violate other countries’ national sovereignty. Not militarily, and not by weird circumspect means either (e.g. via superhuman AI persuasion).” Maybe promising other nations a seat at the bargaining table that determines the path of the AI-driven future. Maybe deciding on a default way to distribute future resources and promising that all departures from that will require broad agreement.
It is, of course, best if such promises are credible and enforceable.
But I think this would have some value even if it's just something like: The US Congress passes a bill that contains a ton of promises to other nations. And that increases the probability that they'll act according to those promises.
There’s a lot of potential work to do here in drafting suggested promises that would:
(Thanks to Carl Shulman for discussion.)
Technical proposals for aggregating preferences
A different direction would be for people to explore technical proposals for effectively aggregating people’s preferences. So that, during an intelligence explosion, it’s more convenient to get more accurate pictures of what different constitutions would recommend. Thereby making it harder to legitimately dismiss such demands.
Previous work:
Bad actors could make dangerous decisions about what to do with AI technology. Perhaps increasing risks from brinkmanship or other destructive technology — or perhaps seizing and maintaining indefinite power over the future and making poor choices about what to do with it. See also some of the risks described in Reducing long-term risks from malevolent actors.
One line of attack on this problem is to develop policies for avoiding AI-assisted human coups. But here are a few more.
If you have opportunities to affect the policies of important institutions, it could be valuable to reduce the probability that malevolent individuals get hired and/or are selected for key roles.
Examples of how to attack this question: (h/t Stefan Torges)
See also Reducing long-term risks from malevolent actors for more analysis and intervention ideas.
There is (thankfully) some significant effort going on in this space already, so I don’t have a lot to add.
Examples of how to attack this question:
This is a riskier proposition, and it’s extremely easy to accidentally do harm here. But in principle, one way to give bad actors relatively less power is to differentially accelerate good actors. C.f. successful, careful AI lab section of Holden Karnofsky’s playbook for AI risk reduction.
If an intelligence explosion could lead to 100 years of “normal” technological progress within just a few years, then this is a very unusually valuable time to have some foresight into what technologies are on the horizon.
It seems particularly valuable to anticipate technologies that could (i) pose big risks or (ii) enable novel solutions to other risks.
On (i), some plausible candidates are:
Some candidates that could feature on both (i) and (ii):
Good results here could influence:
This could be seen as a project or could alternatively be seen as a framing for how to best address many of these issues. (Not just from this post, but also issues from other posts in this series.)
I think it’s plausible that our biggest “value-add” on these topics will be that we see potentially important issues coming before other people. This suggests that our main priority should be to clearly flag all the thorny issues we expect to appear during an intelligence explosion and ask questions about how labs (and possibly other relevant institutions) plan to deal with them.
This could be favorably combined with offering suggestions for how to address all the issues. But separate from any suggestions, it’s valuable to establish something as a problem that needs some answer so that people can’t easily dismiss any one solution without offering an alternative one.
Some of these questions might be framed as “What should your AI do in situation X?”. Interestingly, even if labs don’t engage with a published list of questions, we can already tell what (stated) position the labs’ current AIs have on those questions. They can be presented with dilemmas, the AIs can answer, and the results can be published.
Having a single big list of questions that must be addressed would also make it easier to notice when certain principles conflict. I.e., situations when you can’t fulfill them both at once and are forced to choose.
Example of how to attack this:
(Thanks to Carl Shulman for this idea and discussion.)
That’s all I have on this topic! As a reminder: it's very incomplete. But if you're interested in working on projects like this, please feel free to get in touch.
Other posts in series: Introduction, governance during explosive growth, epistemics, sentience and rights of digital minds, backup plans & cooperative AI.
Some of these risks are also fairly commonly discussed. In particular, centralization of power and risks from powerful AI falling into the wrong hands are both reasonably common concerns, and are strongly related to some of the projects I list in this section.
Why could this happen? Historically, the pace of innovation may have been tightly coupled to world GDP, because population size (i.e. the number of potential innovators) was constrained by the supply of food. In semi-endogenous growth models, this makes super-exponential growth plausible. But recently, growth has outpaced population growth, leading to a slower pace of innovation than our current amount of resources could theoretically support. But AGI would make it easy to convert resources into automated scientists, which could return us to the historical state of affairs. For more on this, see e.g. the duplicator.
C.f. this comment from Michael Nielsen’s notes on the vulnerable world hypothesis: “How to develop provably beneficial surveillance? It would require extensive work beyond the scope of these notes. It is worth noting that most existing surveillance regimes are developed with little external oversight, either in conception, or operationally. They also rarely delegate work to actors with different motives in a decentralized fashion. And they often operate without effective competition. I take these facts to be extremely encouraging: they mean that there is a lot of low-hanging fruit to work with here, obvious levers by which many of the worst abuses of surveillance may be reduced. Classic surveillance regimes have typically prioritized the regime, not humanity at large, and that means the design space here is surprisingly unexplored.”
Though new technology would likely enable rapid reproduction for biological humans and super rapid reproduction for digital minds. That’s one of the technologies that we’ll need to decide how to handle. If we allow for an explosively fast increase in population size, then population size and/or per-capita resources would again be limited by economic growth.
As explained by Ben Garfinkel here, it’s plausible that democracy has recently become common because industrialization means that it’s unusually valuable to invest in your population, and unusually dangerous to not give people what they want. Whereas with widespread automation, states would rely less on satisfying the demands of their population.
Here, I think there’s an important difference between:
- You can only credibly commit to an action if you have the consent of the person who you want to demonstrate this commitment to.
- You can unilaterally make credible commitments.
I think the former is good. I think the latter is quite scary, for reasons mentioned earlier.