Today, Senator Bernie Sanders (I-Vt.) and Representative Greg Casar (D-Texas) introduced the Ban Artificial Superintelligence Act in Congress: the first American bill to propose banning the development of superintelligent AI.
ControlAI has spent years on the question of how to prevent the extinction risk posed by superintelligent AI development. We have briefed over 400 lawmakers across the U.S., U.K., Canada, and Germany on the topic in the last two years, and our U.K. bill was introduced in the U.K. Parliament by Alex Sobel MP, the first ever bill introduced in the world to ban superintelligent AI. Here are our ban superintelligent AI U.S. discussion draft and U.K. bill.
We are excited to see the bill from Sen. Sanders and Rep. Casar tackling the problem at its source. The bill pursues the right goal on both counts: banning superintelligent AI development at home, and committing America to lead the effort to prohibit it abroad. Still, we think a narrowly tailored bill can achieve the same goals more effectively.
The Right Focus: Ban Superintelligent AI at Home, Prevent it Abroad
We're glad to see the bill focus on preventing the development of superintelligent AI. While AI is vital for American defense and economic competitiveness, superintelligent AI is something else entirely: not a tool or a weapon, but fully autonomous AI systems that we cannot control, powerful enough to outcompete humanity.
We appreciate the choice to define superintelligent AI by its capability to overpower the federal government. This targets exactly the kind of AI development that poses a risk of extinction, without having to settle in advance the technical details of how such an AI would work. It also keeps measures focused on the types of AI development that threaten the federal government, even if AI companies change the label they use to describe those projects. For these reasons, we use this capability in the definition in both our U.S. draft bill and U.K. bill.
The bill also correctly requires the United States to pursue international agreements to ban the development of superintelligent AI. Banning its development in America protects from homegrown threats, but does not deal with superintelligent AI development abroad: if anyone develops it anywhere, it threatens our national security and the life of every American. This is why any domestic ban must be coupled with making it a foreign policy priority to prevent the development of superintelligent AI in any other nation.
This provision is unlikely to bind the executive branch, since Presidents have long treated congressional directions on the conduct of foreign policy as advisory, and the question has never been settled in court. Still, it is important that Congress goes on record that the U.S. should work to prevent superintelligent AI anywhere, as a necessary part of any strategy to ensure American national security and prevent the extinction threat posed by superintelligent AI.
A natural follow-up is "what about China?" President Trump is right: America must beat China at building AI that is economically and militarily useful. But superintelligent AI is different: it would not be a weapon that can help America win, but a rogue force stationed within our data centers.
Swarms of autonomous AI powerful enough to threaten our government, under the control of no human authority, could destroy our country, whether they are developed at home or abroad. If China attempts to develop superintelligent AI, we must do all we can to stop them, and make it clear in advance that we will not stand by while China, or any country, tries to create it.
No country, no company, no individual can control a superintelligent AI. China faces the same threat from superintelligent AI development that we do. But we should not take Beijing at its word; instead, we need to establish credible deterrence and enforcement.
There Is a Lighter Touch Approach to Banning Superintelligent AI
At ControlAI, we consulted with the sponsors' offices on the Ban Artificial Superintelligence Act legislative framework. Still, we believe that the proposal's goals can be achieved with a more targeted approach, like the one in our draft bill. Our approach differs from this bill on the proposed pause on advanced AI development and on the treatment of precursors to superintelligent AI.
A Blanket AI Pause Is Not Necessary to Prevent Superintelligent AI
The first disagreement we have with the bill relates to its proposed mandatory pause on advanced AI systems. In the bill, "advanced AI systems" are defined as AI trained using an amount of computing power above a threshold of 10^25 operations. This pause is to last until the Department of AI that the bill creates is fully staffed and regulations are ready to implement.
Our view is that a blanket pause on advanced AI development is hard to get right, and is not necessary. A ban on superintelligent AI, paired with monitoring and restricting a few important precursors, gets the job done. Outlawing the development of superintelligent AI provides a legal basis to stop ongoing projects pursuing it, without affecting the rest of innovation in the industry. The Ban Artificial Superintelligence Act already includes such provisions.
The mandatory pause on advanced AI, as included in the bill, imposes real costs for an uncertain benefit.
On one hand, a blanket pause on all advanced AI development above 10^25 FLOPs may hit forms of AI development that are not precursors to superintelligent AI, such as compute-intensive video generation systems. It might also hamper legitimate defense applications, such as cybersecurity and threat assessment, that rely on precursors to superintelligent AI.
On the other hand, a blanket pause on advanced AI development based on compute thresholds does not directly prevent the development of superintelligent AI, and can be circumvented and gamed by sufficiently motivated AI companies.
The amount of computing power used in development is at best a proxy for future capabilities of an AI system. In the short term, this proxy can be gamed by motivated AI companies focusing on techniques that nominally use less "compute" under the definition in the regulations. In the medium term, it can be circumvented by AI companies redirecting R&D efforts into algorithmic improvements rather than exploiting compute scaling laws to get to superintelligent AI.
In terms of implementation, the bill implements a pause that kicks in immediately and lasts until the Department of AI is established and fully staffed. Enforcement of a broad pause on advanced AI of the kind defined in the bill will require a lot of government capacity and judgment calls, and in the absence of a clear owner, there is a risk of either no meaningful enforcement, or chaotic enforcement. Chaotic enforcement of a blanket pause on advanced AI could undercut American economic and military dominance.
Precursors Should Be Monitored and Restricted, Not Banned
We also disagree with the treatment of precursors to superintelligent AI. The bill bans the development of a list of capabilities called "superintelligence precursor characteristics," notably:
Recursive Self-Improvement (RSI), the capacity of AIs to develop their own successors without human intervention
Self-preservation, the capacity of AIs to escape attempts to shut them down or control them
Weapons uplift, the capacity to uplift the development of nuclear, chemical, or biological weapons
In our policy proposals and bills, we recommend distinguishing between capabilities that mark a point of no return toward superintelligent AI, and capabilities that could feed into its development, but also have legitimate uses outside of superintelligent AI development. We call the first "indicators" of superintelligent AI development, and the second "precursors" of superintelligent AI.
Indicators, such as recursive self-improvement, are AI capabilities that, if reached, would make preventing superintelligent AI dramatically harder. As such, any ban on superintelligent AI needs to cover these indicators.
Precursors comprise a wider range of AI capabilities, such as hacking. While precursors could play a role in the development of superintelligent AI, they do not, on their own, push a superintelligent AI project past the point where it can be prevented. Precursors have legitimate uses, some of which are essential to maintaining American leadership on the world stage, including critical military applications. A blanket ban on precursors would undermine these uses.
Still, as these capabilities get more powerful, they might lead to superintelligent AI. For national security purposes, the government should monitor their development and be ready to restrict them if they reach dangerous levels, while still ensuring the U.S. government can use them for legitimate purposes. Capabilities like hacking don't need to be outright banned as they are in the Ban Artificial Superintelligence Act. We can take a lighter touch.
For similar reasons, we oppose unilateral bans on domestic data centers, and we are happy to see no such measures are present in the Ban Artificial Superintelligence Act. Such measures would severely damage the economic and military capacity of the U.S. by preventing the development and use of critical applications which pose no risk of extinction. At the same time, they would also not prevent superintelligent AI development: companies can pursue superintelligent AI with the data centers they already have as algorithms get more efficient, or simply develop superintelligent AI on data centers residing abroad.
Lastly, we don't necessarily need all the bureaucracy the bill proposes, such as a licensing regime, to enforce a ban. We recommend empowering the executive branch to stop companies from developing superintelligent AI directly, with narrowly tailored measures. The landscape of risks will keep evolving quickly, and this calls for an executive that can respond quickly rather than one bound by a complex network of rules.
Conclusion
Despite our disagreements with some of the details of this proposal, ControlAI is excited to see the Ban Artificial Superintelligence Act open the discussion in Congress about how America should address this threat: ban superintelligent AI at home, and lead the effort to prevent its development anywhere else.
Now that this bill has kicked off the debate, we encourage Congress to focus on preventing superintelligent AI. Rather than a pause on all advanced AI, we recommend targeting the projects pursuing superintelligent AI right now and putting American lives and security at risk. We look forward to seeing many more proposals to address this urgent threat.
Note that section 15 of their bill carves out an exception for federally funded defensive cybersecurity research. We do not think an exception limited to federal funding, and to defensive research, is enough to support the cyber capabilities America needs. ↩︎
I hope this bill steers the public conversation toward approaches in the vein of banning superintelligence, though it's not my ideal bill.
While the bill is extremely unlikely to pass so it's unclear how much the details matter, I'd like to further draw attention to the banning of any model with one or more superintelligence precursor characteristics. In particular, it's plausible that some of these precursor characteristics have been met by AIs for multiple years now, perhaps around GPT-4.
From the bill text, the superintelligence precursor characteristics are:
(A) The capacity to automate or greatly accelerate the process of artificial intelligence research and development.
(B) The capacity to access secured digital or physical infrastructure, such as a computer information system or network, without authorization or in excess of authorized access.
(C) The capacity to ensure continued and independent operation notwithstanding attempts to shut down or otherwise hinder operations.
(D) The capacity to uplift the design, production, modification, or procurement of nuclear, chemical, or biological weapons.
(E) The capacity to independently modify or enhance its own functions.
(F) The capacity to scheme, deceive, or otherwise prevent or avoid effective oversight or control by humans.
I think that for weak operationalizations, GPT-4 might meet (B), (D), (E), and/or (F).
If this were the case and the bill passed as is, then my understanding is that AI companies would have to destroy all copies of weights for models at least as powerful as GPT-4, and refrain from developing any AIs beyond that capability level.
Today, Senator Bernie Sanders (I-Vt.) and Representative Greg Casar (D-Texas) introduced the Ban Artificial Superintelligence Act in Congress: the first American bill to propose banning the development of superintelligent AI.
ControlAI has spent years on the question of how to prevent the extinction risk posed by superintelligent AI development. We have briefed over 400 lawmakers across the U.S., U.K., Canada, and Germany on the topic in the last two years, and our U.K. bill was introduced in the U.K. Parliament by Alex Sobel MP, the first ever bill introduced in the world to ban superintelligent AI. Here are our ban superintelligent AI U.S. discussion draft and U.K. bill.
We are excited to see the bill from Sen. Sanders and Rep. Casar tackling the problem at its source. The bill pursues the right goal on both counts: banning superintelligent AI development at home, and committing America to lead the effort to prohibit it abroad. Still, we think a narrowly tailored bill can achieve the same goals more effectively.
The Right Focus: Ban Superintelligent AI at Home, Prevent it Abroad
We're glad to see the bill focus on preventing the development of superintelligent AI. While AI is vital for American defense and economic competitiveness, superintelligent AI is something else entirely: not a tool or a weapon, but fully autonomous AI systems that we cannot control, powerful enough to outcompete humanity.
We appreciate the choice to define superintelligent AI by its capability to overpower the federal government. This targets exactly the kind of AI development that poses a risk of extinction, without having to settle in advance the technical details of how such an AI would work. It also keeps measures focused on the types of AI development that threaten the federal government, even if AI companies change the label they use to describe those projects. For these reasons, we use this capability in the definition in both our U.S. draft bill and U.K. bill.
The bill also correctly requires the United States to pursue international agreements to ban the development of superintelligent AI. Banning its development in America protects from homegrown threats, but does not deal with superintelligent AI development abroad: if anyone develops it anywhere, it threatens our national security and the life of every American. This is why any domestic ban must be coupled with making it a foreign policy priority to prevent the development of superintelligent AI in any other nation.
This provision is unlikely to bind the executive branch, since Presidents have long treated congressional directions on the conduct of foreign policy as advisory, and the question has never been settled in court. Still, it is important that Congress goes on record that the U.S. should work to prevent superintelligent AI anywhere, as a necessary part of any strategy to ensure American national security and prevent the extinction threat posed by superintelligent AI.
A natural follow-up is "what about China?" President Trump is right: America must beat China at building AI that is economically and militarily useful. But superintelligent AI is different: it would not be a weapon that can help America win, but a rogue force stationed within our data centers.
Swarms of autonomous AI powerful enough to threaten our government, under the control of no human authority, could destroy our country, whether they are developed at home or abroad. If China attempts to develop superintelligent AI, we must do all we can to stop them, and make it clear in advance that we will not stand by while China, or any country, tries to create it.
No country, no company, no individual can control a superintelligent AI. China faces the same threat from superintelligent AI development that we do. But we should not take Beijing at its word; instead, we need to establish credible deterrence and enforcement.
There Is a Lighter Touch Approach to Banning Superintelligent AI
At ControlAI, we consulted with the sponsors' offices on the Ban Artificial Superintelligence Act legislative framework. Still, we believe that the proposal's goals can be achieved with a more targeted approach, like the one in our draft bill. Our approach differs from this bill on the proposed pause on advanced AI development and on the treatment of precursors to superintelligent AI.
A Blanket AI Pause Is Not Necessary to Prevent Superintelligent AI
The first disagreement we have with the bill relates to its proposed mandatory pause on advanced AI systems. In the bill, "advanced AI systems" are defined as AI trained using an amount of computing power above a threshold of 10^25 operations. This pause is to last until the Department of AI that the bill creates is fully staffed and regulations are ready to implement.
Our view is that a blanket pause on advanced AI development is hard to get right, and is not necessary. A ban on superintelligent AI, paired with monitoring and restricting a few important precursors, gets the job done. Outlawing the development of superintelligent AI provides a legal basis to stop ongoing projects pursuing it, without affecting the rest of innovation in the industry. The Ban Artificial Superintelligence Act already includes such provisions.
The mandatory pause on advanced AI, as included in the bill, imposes real costs for an uncertain benefit.
On one hand, a blanket pause on all advanced AI development above 10^25 FLOPs may hit forms of AI development that are not precursors to superintelligent AI, such as compute-intensive video generation systems. It might also hamper legitimate defense applications, such as cybersecurity and threat assessment, that rely on precursors to superintelligent AI.
On the other hand, a blanket pause on advanced AI development based on compute thresholds does not directly prevent the development of superintelligent AI, and can be circumvented and gamed by sufficiently motivated AI companies.
The amount of computing power used in development is at best a proxy for future capabilities of an AI system. In the short term, this proxy can be gamed by motivated AI companies focusing on techniques that nominally use less "compute" under the definition in the regulations. In the medium term, it can be circumvented by AI companies redirecting R&D efforts into algorithmic improvements rather than exploiting compute scaling laws to get to superintelligent AI.
In terms of implementation, the bill implements a pause that kicks in immediately and lasts until the Department of AI is established and fully staffed. Enforcement of a broad pause on advanced AI of the kind defined in the bill will require a lot of government capacity and judgment calls, and in the absence of a clear owner, there is a risk of either no meaningful enforcement, or chaotic enforcement. Chaotic enforcement of a blanket pause on advanced AI could undercut American economic and military dominance.
Precursors Should Be Monitored and Restricted, Not Banned
We also disagree with the treatment of precursors to superintelligent AI. The bill bans the development of a list of capabilities called "superintelligence precursor characteristics," notably:
In our policy proposals and bills, we recommend distinguishing between capabilities that mark a point of no return toward superintelligent AI, and capabilities that could feed into its development, but also have legitimate uses outside of superintelligent AI development. We call the first "indicators" of superintelligent AI development, and the second "precursors" of superintelligent AI.
Indicators, such as recursive self-improvement, are AI capabilities that, if reached, would make preventing superintelligent AI dramatically harder. As such, any ban on superintelligent AI needs to cover these indicators.
Precursors comprise a wider range of AI capabilities, such as hacking. While precursors could play a role in the development of superintelligent AI, they do not, on their own, push a superintelligent AI project past the point where it can be prevented. Precursors have legitimate uses, some of which are essential to maintaining American leadership on the world stage, including critical military applications. A blanket ban on precursors would undermine these uses.
Still, as these capabilities get more powerful, they might lead to superintelligent AI. For national security purposes, the government should monitor their development and be ready to restrict them if they reach dangerous levels, while still ensuring the U.S. government can use them for legitimate purposes. Capabilities like hacking don't need to be outright banned as they are in the Ban Artificial Superintelligence Act. We can take a lighter touch.
For similar reasons, we oppose unilateral bans on domestic data centers, and we are happy to see no such measures are present in the Ban Artificial Superintelligence Act. Such measures would severely damage the economic and military capacity of the U.S. by preventing the development and use of critical applications which pose no risk of extinction. At the same time, they would also not prevent superintelligent AI development: companies can pursue superintelligent AI with the data centers they already have as algorithms get more efficient, or simply develop superintelligent AI on data centers residing abroad.
Lastly, we don't necessarily need all the bureaucracy the bill proposes, such as a licensing regime, to enforce a ban. We recommend empowering the executive branch to stop companies from developing superintelligent AI directly, with narrowly tailored measures. The landscape of risks will keep evolving quickly, and this calls for an executive that can respond quickly rather than one bound by a complex network of rules.
Conclusion
Despite our disagreements with some of the details of this proposal, ControlAI is excited to see the Ban Artificial Superintelligence Act open the discussion in Congress about how America should address this threat: ban superintelligent AI at home, and lead the effort to prevent its development anywhere else.
Now that this bill has kicked off the debate, we encourage Congress to focus on preventing superintelligent AI. Rather than a pause on all advanced AI, we recommend targeting the projects pursuing superintelligent AI right now and putting American lives and security at risk. We look forward to seeing many more proposals to address this urgent threat.
Note that section 15 of their bill carves out an exception for federally funded defensive cybersecurity research. We do not think an exception limited to federal funding, and to defensive research, is enough to support the cyber capabilities America needs. ↩︎