The ability to control AI systems is not increasing, but rather becoming more difficult with every new model. Governance is not keeping up with the pace of development, and legislation to protect us from the risks of AI is insufficient or non-existent. In a new statement, employees of frontier AI companies call for governance efforts to pace the frontier AI development. In this article, I explain why the lack of direct feedback loops between AI governance and technical AI research is the key bottleneck to achieve such effective governance.
Frontier AI development is speeding up and the frontier AI models show how they are increasingly harder to control. Both Anthropic and OpenAI’s newer models have been able to access systems they were not supposed to (with OpenAI’s models gaining unauthorized access to the internet and breaking out of its sandbox, and Anthropic proceeding to find a misconfiguration in the evaluation environment for their models). If we cannot control these systems on a technical level, there is a need for effective governance to regulate the development of these systems. After 90+ conversations with professionals I’ve seen the pattern of many people switching from technical AI safety to AI governance, precisely because of the difficulty and intractability of the alignment problem. But in order to achieve effective legislation, we need direct collaboration between legislation and technical research and a channel for direct influence on each other. The lack of these closed feedback loops is a key bottleneck for technically grounded and meaningful regulation.
The new statement signed by over 1,300 employees of frontier AI companies (including Dario Amodei, CEO of Anthropic) shows once again the need for AI governance as the technical solutions are insufficient. That engineers and the developers of these systems are calling for stronger governance is once again iterating this story.
AI could help create a dramatically better future, but that outcome is not guaranteed. The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.
Building on work already underway to monitor frontier model releases:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
This statement also clearly states the need to develop technical and governance tools. It does not imply a need for more knowledge on AI risk or what to govern, but rather the development of tools to act and create effective legislation of frontier AI development.
Below I lay out the argument that the bottleneck for the development of such tools at the moment is the lack of direct feedback loops between legislative bodies and technical AI research. Existence of such direct feedback loops, would incentivise collaboration and enable effective legislation tools that are technically grounded and meaningful.
The problem
One of the biggest problems within the AI Safety field right now is that the actors that have legislative power and the technical researchers developing solutions, don’t have enough incentives or a way to directly influence each other. Due to this problem, policymakers struggle to know what regulations would be technically plausible and meaningful, while the technical solutions or information needed for certain regulations are not being developed by technical researchers. This communication issue stems from unclear feedback loops and the disconnect between technical knowledge and legislative bodies. The AI Safety Atlas by CeSIA explains this problem, framed as the need for creating “epistemic communities”; “The development of nuclear arms control agreements wasn't solely the work of diplomats and politicians. It relied heavily on input from scientists, engineers, and other technical experts who understood the technology and its implications. These experts formed a network of professionals with recognized expertise in a particular domain, or as what political scientists call an "epistemic community". They played important roles in shaping policy debates, providing technical advice, and even serving as back-channel diplomats during tense periods of the Cold War. Unlike nuclear physicists, who were often employed directly by governments, many AI experts work in the private sector, so a challenge to forming such networks for global AI governance will be ensuring that epistemic communities can effectively inform policy decisions.”.
What is being done today, and the limitations of current practice
Some organisations are partly filling this role by informing policymakers or through policy research, but this usually takes the form of a thinktank (i.e. CFG, TFS, IAPS, GovAI, CLTR.) or a lobbyist/advocacy organization (such as ControlAI).
Advocacy organizations and think tanks aim to address part of the problem by informing policymakers on what regulations should or could be implemented for mitigating AI risk. These organizations are mostly emphasizing policy research and informing about risks, with a usually limited technical focus. The other crucial aspect of the problem is not being addressed; to inform technical researchers what policymakers would need in terms of research/knowledge/solutions for certain legislation. The information is currently unidirectional, and bottlenecked by organisations that translate technical findings for policy purposes. This unidirectional communication does not foster collaboration and is not comprehensive. The lack of direct bidirectional information flows in combination with the presence of strong lobbyism (in both directions), erodes the trust from policymakers and creates uncertainty in decisions.
Governance directed technical research is being executed by the EU AI Office but their aim is the implementation of the EU AI Act, not to proactively enable the creation of further legislation.
The existing organisations are effective at what they do, but they are not addressing the problem that I am describing above, simply because that is not their aim.
The feedback loops needed for effective development of solutions in AI Safety are too long, slow and largely one-directional.
The bottleneck for effective AI governance is not a lack of knowledge of the risks or even what kinds of legislation that needs to be put in place. The real bottleneck is the lack of direct feedback loops between legislative power and technical AI research to develop the tools to act upon this knowledge.
The ability to control AI systems is not increasing, but rather becoming more difficult with every new model. Governance is not keeping up with the pace of development, and legislation to protect us from the risks of AI is insufficient or non-existent. In a new statement, employees of frontier AI companies call for governance efforts to pace the frontier AI development. In this article, I explain why the lack of direct feedback loops between AI governance and technical AI research is the key bottleneck to achieve such effective governance.
Frontier AI development is speeding up and the frontier AI models show how they are increasingly harder to control. Both Anthropic and OpenAI’s newer models have been able to access systems they were not supposed to (with OpenAI’s models gaining unauthorized access to the internet and breaking out of its sandbox, and Anthropic proceeding to find a misconfiguration in the evaluation environment for their models). If we cannot control these systems on a technical level, there is a need for effective governance to regulate the development of these systems. After 90+ conversations with professionals I’ve seen the pattern of many people switching from technical AI safety to AI governance, precisely because of the difficulty and intractability of the alignment problem. But in order to achieve effective legislation, we need direct collaboration between legislation and technical research and a channel for direct influence on each other. The lack of these closed feedback loops is a key bottleneck for technically grounded and meaningful regulation.
The new statement signed by over 1,300 employees of frontier AI companies (including Dario Amodei, CEO of Anthropic) shows once again the need for AI governance as the technical solutions are insufficient. That engineers and the developers of these systems are calling for stronger governance is once again iterating this story.
The Pacing the Frontier statement is as follows:
This statement also clearly states the need to develop technical and governance tools. It does not imply a need for more knowledge on AI risk or what to govern, but rather the development of tools to act and create effective legislation of frontier AI development.
Below I lay out the argument that the bottleneck for the development of such tools at the moment is the lack of direct feedback loops between legislative bodies and technical AI research. Existence of such direct feedback loops, would incentivise collaboration and enable effective legislation tools that are technically grounded and meaningful.
The problem
One of the biggest problems within the AI Safety field right now is that the actors that have legislative power and the technical researchers developing solutions, don’t have enough incentives or a way to directly influence each other. Due to this problem, policymakers struggle to know what regulations would be technically plausible and meaningful, while the technical solutions or information needed for certain regulations are not being developed by technical researchers. This communication issue stems from unclear feedback loops and the disconnect between technical knowledge and legislative bodies. The AI Safety Atlas by CeSIA explains this problem, framed as the need for creating “epistemic communities”; “The development of nuclear arms control agreements wasn't solely the work of diplomats and politicians. It relied heavily on input from scientists, engineers, and other technical experts who understood the technology and its implications. These experts formed a network of professionals with recognized expertise in a particular domain, or as what political scientists call an "epistemic community". They played important roles in shaping policy debates, providing technical advice, and even serving as back-channel diplomats during tense periods of the Cold War. Unlike nuclear physicists, who were often employed directly by governments, many AI experts work in the private sector, so a challenge to forming such networks for global AI governance will be ensuring that epistemic communities can effectively inform policy decisions.”.
What is being done today, and the limitations of current practice
Some organisations are partly filling this role by informing policymakers or through policy research, but this usually takes the form of a thinktank (i.e. CFG, TFS, IAPS, GovAI, CLTR.) or a lobbyist/advocacy organization (such as ControlAI).
Advocacy organizations and think tanks aim to address part of the problem by informing policymakers on what regulations should or could be implemented for mitigating AI risk. These organizations are mostly emphasizing policy research and informing about risks, with a usually limited technical focus. The other crucial aspect of the problem is not being addressed; to inform technical researchers what policymakers would need in terms of research/knowledge/solutions for certain legislation. The information is currently unidirectional, and bottlenecked by organisations that translate technical findings for policy purposes. This unidirectional communication does not foster collaboration and is not comprehensive. The lack of direct bidirectional information flows in combination with the presence of strong lobbyism (in both directions), erodes the trust from policymakers and creates uncertainty in decisions.
Governance directed technical research is being executed by the EU AI Office but their aim is the implementation of the EU AI Act, not to proactively enable the creation of further legislation.
The existing organisations are effective at what they do, but they are not addressing the problem that I am describing above, simply because that is not their aim.
The feedback loops needed for effective development of solutions in AI Safety are too long, slow and largely one-directional.
The bottleneck for effective AI governance is not a lack of knowledge of the risks or even what kinds of legislation that needs to be put in place.
The real bottleneck is the lack of direct feedback loops between legislative power and technical AI research to develop the tools to act upon this knowledge.