Robin Hanson suggests, partly in response to calls for a pause in development of AGI, liability rules for risks related to AGI rapidly becoming powerful.

My intuitive reaction was to classify foom liability as equivalent to a near total ban on AGI.

Now that I've found time to think more carefully about it, I want to advocate foom liability as a modest improvement over any likely pause or ban on AGI research. In particular, I want the most ambitious AI labs worldwide to be required to have insurance against something like $10 billion to $100 billion worth of damages.

Liability has the obvious advantage that whoever is financially responsible for the liability (AI labs? insurance companies?) will have better incentives to study the risks carefully than would politicians, bureaucrats, or standards committees composed of industry representatives.

A more subtle advantage: limited risk that it will be used to protect incumbents or permanently stifle good technologies.

I see somewhat less risk of liability being hijacked by culture warriors.

Liability wouldn't produce ideal incentives, due to scenarios where AGI accidents cause more damage than any company can compensate for (e.g. human extinction). I'll guess that liability would provide one tenth of the optimal incentive.

But remember that there's not too much reason to worry about improving the financial incentives to avoid extinction. I'm not too concerned with the risk that DeepMind will train a system that they think has a 5% chance of killing us all. I'm more concerned about a less responsible lab training a system that their insurance company would think had a 5% chance of starting a minor war, and which I think has a 5% chance of killing us all. I.e. my main concern is with AI labs that are overconfident about safety, but who are rational enough to respond to projected penalties for medium-sized accidents.

For the risks that worry me most, there would need to be an international agreement on such liability / insurance. That creates difficulties beyond those that would confront a direct ban. Some countries would allow fly-by-night insurance companies to provide "insurance" that would not, in practice, cover much of the liability.

How Much Liability

Key variables are: who needs to buy liability insurance (or provide the equivalent evidence of self-insurance), and how much harm does the insurance policy need to cover?

It feels pretty arbitrary to assign a specific number to many of the AI accidents that I can imagine.

Suppose an AI makes a serious attempt at world conquest: It hacks and blackmails its way into apparent control of several nuclear arsenals. It releases scandalous evidence about influential people who warned about AIs gaining power. Yet it gets shut down without any clear harm to innocent people.

I'm guessing that penalties in the vicinity of $10 billion, give or take a factor or 10, would be in the right ballpark.

Much more than that would mean that only the wealthiest organizations could afford to work on AGI. I was tempted to claim that such organizations will on average be more responsible than smaller organizations. Then I remembered Elon Musk, and decided it was unclear.

It feels harder to adequately describe what entities would need to be insured. E.g. open source clones of leading LLMs are unlikely to be smarter than the original LLMs (compute-intensive training seems likely to remain important), but might be more dangerous because they attract more reckless tinkerers who make them more agenty. That is a problem for any proposal to regulate or slow AI development. I'm focusing in this post on how foom liability compares to other options. I'm not claiming to know whether any of these proposals are feasible.

Scenarios

I'll now imagine several scenarios as to how foom liability would play out. I'm giving probabilities to indicate my vague intuitions about which scenarios are most likely. All probabilities are conditional on some sort of international agreement to require strict liability for foom-like accidents.

Scenario 1: Permanent Ban on AGI

I am reminded of what liability rules have done to innovation in small aircraft, and of what regulation has done to nuclear power. From Where is my Flying Car?:

One of the more ironic regulatory pathologies that has shaped the world of general aviation is that most of the planes we fly are either 40 years old or homemade - and that we were forced into that position in the name of safety.

In this scenario, I imagine an airtight requirement that key types of AI research get an insurance company to write a policy covering that research. I also imagine that insurance companies are reluctant to write those policies. The existence of somewhat special rules for foom liability reinforces widespread concern over the risks, in ways that cause a perpetual upward creep in expected damage awards from arguably dysfunctional courts.

This requires suppression of unauthorized research that is a good deal more effective than what I know how to implement. Yet if most professors at leading universities decided that AGI would make their jobs obsolete (while GPT-4 won't), then I wouldn't want to bet against their ability to devise an airtight ban.

I'll give this scenario a 5% chance.

Scenario 2: Leaky Ban on AGI

More likely, it will be hard to fully enforce any insurance requirement. Most of the problem is likely to come from the difficulty of identifying what kind of research is risky enough to require insurance.

An analogy would be rules that outlaw unlicensed taxis and hotels. Uber and Airbnb created businesses that compete with incumbents in those industries, without meeting the formal definition of taxis and hotels, defeating the goal of protecting those incumbents from competition.

I imagine that it's hard to draft an insurance requirement that reliably distinguishes between safe software and software that might foom. Any attempt to strike a reasonable balance between allowing good software to remain unregulated, and restricting all risky software, will leave opportunities for clever startups to create unlicensed AGIs that will foom to whatever extent foom is possible.

I'm imagining in this scenario that AI labs mostly keep roughly the same mix of focus on capability and safety research that they have now. They postpone risky training of large systems. That slows down capability advances, and slows down some safety research that depended on the availability of more powerful AIs.

I expect this scenario would buy AI safety researchers a year or five. That would come at the cost of an increased risk that AGIs will be designed by more reckless developers. It's unclear what net affects this scenario would have on our safety.

I'll give this scenario a 55% chance.

Scenario 3: Goldilocks

In my hoped for scenario, foom liability is effective at delaying AGI development for a year or two, and at spurring increased safety research.

Insurance companies initially indicate that they would charge more than $10 billion for a policy covering the most reputable AI labs. A year later, they sell one policy for $5 billion. Six months after that, multiple policies are sold for $3 billion.

Many people in the industry end up agreeing that the insurance requirement caused the industry to reduce some important risks, at a fairly acceptable cost in delaying benefits.

I'll give this scenario a 10% chance.

Scenario 4: Oligopoly

The main effect might be to slow development, by prohibiting small organizations from doing any important AI development.

I'm imagining here that a handful of companies are able to buy insurance for maybe $10 billion each. They were already doing most of what they were able to do to minimize risks. The insurance was a nuisance due to the need to articulate safety measures, most of which required expert knowledge to understand. The insurance companies didn't learn enough to provide any useful ideas about safety beyond whatever was the default path. Everyone ends up agreeing that there are important risks, but we find no way to reach any consensus on how to handle the risks.

My best guess is that this makes us slightly safer, via stopping a few reckless companies from competing, and via slowing down competitive races between leading AI labs.

That safety comes at a cost of increased concentration of power in a few big companies.

I'll give this scenario a 10% chance.

Scenario 5: Full Speed Ahead

Foom liability might be ineffective. The benefits of AI could persuade many companies to pursue powerful AGI regardless of the insurance costs.

This is a clear possibility if companies are allowed to self-insure, or if small companies are able to compete.

I'll give this scenario a 20% chance.

Scenario 6: China versus West

I'm unclear whether this scenario is affected by the difference between liability versus a temporary ban on development, so maybe it doesn't belong in this post at all. China seems slower than the US to treat smarter-than-human AI as a near-mode issue. China seems at least as willing in general to ban new technologies, and somewhat more likely to enforce those bans effectively. Most likely Chinese concern over smarter-than-human AI will follow US concern with a delay of a year or so.

I'd be pretty optimistic about an international agreement if China was mainly concerned with a commercial balance of power. But I see a strange interplay between AGI risk and conflict over Taiwan.

My biggest concern is that China will see restrictions on AGI (including foom liability) as partly an attempt to keep China from participating in the AI revolution. That would increase the already important pressure on China to at least blockade Taiwan. The resulting GPU shortage would delay AI progress by a year or so. That would be a high-risk way of buying time for safety research.

There will likely be political pressure in the US for advocates of restrictions on AI to ally with forces that want to cripple China.

I'll treat this as a subset of the Leaky Ban scenario, and not give it a separate probability.

Conclusion

There are still many details that would need to be clarified. Imagine that an AGI manipulates South Korea into liberating North Korea, causing 100k immediate deaths, but the AGI projects doing that will save lives in the longer run. How do we decide whether to penalize the AGI's creators? I suspect we get decent incentives whichever way we decide such questions, as long as we have relatively clear rules for deciding on those penalties.

A foom insurance requirement looks hard to implement well, but only a little bit harder than a more direct pause or ban on AGI development.

I'll guess that a foom insurance has a 5% chance of producing an important safety benefit. Given how precarious our position looks, that seems like a great deal.