MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form.
I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well.
The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal.
The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences.
At it's core, the bill does four things:
Ban AI that has one of six dangerous capabilities
Automate or greatly accelerate AI research and development
Access secured systems without authorization
Keep operating despite attempts to shut it down
Meaningfully help with nuclear, chemical, or biological weapons
Modify its own functions
Scheme, deceive, or avoid human oversight
Only allow certified orgs to train “Advanced AI Systems” - An advanced AI system is defined as any model that has been trained with greater than 10^25 operations. Only certified orgs are allowed to develop such models. As such, they are the only people allowed to posses an advanced AI system that has not yet been certified as safe.
Establish a government department tasked with certifying that an AI system is safe - No Advanced AI System can be distributed unless it has received certification from this department.
Impose strict penalties for people who recklessly distribute dangerous AI - Up to 20 years in prison and the forfeiture of all assets of a business.
This basic framework seems broadly sensible, although there are legitimate questions about the extent to which we want to ban the automation of AI research, whether this gives the government too much control, and whether it limiting distribution of pre-approved models might make AI safety research harder.
The biggest problem with the bill is that, as currently drafted, it would have lots of negative consequences. In particular:
The dangerous capabilities rules are vaguely defined, and they come into force immediately, without waiting for them to be clarified - This makes it hard to know what is actually illegal. A bill would normally be expected to task the department with coming up with a clear testing standard, and require that standard to be agreed before people could be punished for violating it.
Existing models are not grandfathered in - You’d expect all models prior to the enactment of the bill to be declared legal to avoid chaos - maybe with some regime for distributing signatures for permitted models. As drafted, it seems that existing models might be illegal to possess or distribute, and there initially isn’t any way to know what is legal.
The dangerous capability rules appear to apply to any AI, not just advanced AI - I suspect that they had intended the rules to only apply to advanced AI systems (those that take more than 10^25 operations to train and require approval before release), but that is not the way the bill is written. Smaller open source models become a legal grey area because there is no requirement that they be certified as legal, but the law still applies. Indeed, since the bill doesn't define AI, or a threshold for its banned capabilities, a particularly pedantic reading of the bill could argue that a calculator or online textbook were illegal.
It punishes looking for danger - If you look for risks and you find one, you are now in possession of a banned model and face harsh penalties. A law would normally provide safe harbor for safety research. Nobody is required to check for precursors and there would now be a big incentive not to.
All development of advanced AI pauses until the department is fully staffed and has finalized its rules, with no timeline for doing that - The department has no budget and would likely take a while to ramp up. This would mean an indefinite shutdown of US AI labs and thus likely a migration of their research to other countries during that period of uncertainty.
The “Advanced AI Systems” threshold is sufficiently low to include GPT4 and other old models - The bill is unclear, but a strict reading would say that it is illegal to distribute all current US frontier models - because they are "advanced AI systems" and have not yet been certified by the government as safe.
It potentially makes mutual agreement with China much harder - The main leverage we have with China is that the US is currently ahead in AI. If the US was not ahead in AI then that leverage would be lost.
The limit of 10^25 training operations is hard to enforce and likely will be gamed - This wouldn’t matter so much if it was just the point at which sensible regulation come into play (e.g. for some of the other bills), but matters a lot if larger models are essentially banned. Companies will be dishonest about their degree of training, or bypass it by putting more effort into algorithmically generated training data that gives a model a smooth path to training to the desired shape.
My overall sense is that what this bill is trying to do is broadly sensible, and it has the potential to turn into a good bill with a bit of editing, but it is not a good bill in it's current form. This is something of a shame, because I worry that, in a way reminiscent of the Toxoplasma of Rage the bill in its current form becomes the main way people argue about more strictly limiting the development of superintelligent AI, making it harder for a better bill to emerge.
MIRI recently endorsed the Ban Artificial Superintelligence Act, but others like ControlAI have called it overly broad, and a few other respected experts have outright endorsed it in its current form.
I decided to take a look at it myself and see both what the bid is trying to do and whether it actually does it well.
The goal of this act is to ban any AI that shows traits that indicate it could cause great harm to society, and to restrict the development of advanced AI to certified institutions who can be trusted to do so safely. This seems like a reasonable goal.
The problem with this bill, as far as I can see it, isn't that its goal is wrong, but that the way it's been drafted contains lots of flaws, which are likely to have unintended negative consequences.
At it's core, the bill does four things:
This basic framework seems broadly sensible, although there are legitimate questions about the extent to which we want to ban the automation of AI research, whether this gives the government too much control, and whether it limiting distribution of pre-approved models might make AI safety research harder.
The biggest problem with the bill is that, as currently drafted, it would have lots of negative consequences. In particular:
My overall sense is that what this bill is trying to do is broadly sensible, and it has the potential to turn into a good bill with a bit of editing, but it is not a good bill in it's current form. This is something of a shame, because I worry that, in a way reminiscent of the Toxoplasma of Rage the bill in its current form becomes the main way people argue about more strictly limiting the development of superintelligent AI, making it harder for a better bill to emerge.