The key bottleneck for effective AI governance is not political will, nor translation from technical findings to policymakers.
To enable technically grounded legislation with existing will from policymakers, we need mechanisms for policymakers to direct policy-enabling technical research.
Policy-directed research was crucial for enabling nuclear arms control treaties.
A proposed mechanism for an org to address this in AI safety would look something like:
Listen to policymakers and what they want to regulate → internal technical team researches solution → solution gets packaged into policy proposal → return policy proposal with technical solution to policymakers.
The neglected problem for effective AI governance
When discussing the lack of effective AI governance two problems are regularly addressed; political awareness and will, and technical knowledge reaching policymakers. But a key bottleneck is neglected: the lack of direct feedback loops between AI governance and technical AI safety research. Policymakers do not currently have a mechanism to directly influence or direct research toward specific solutions needed for policy. As a result, that risks the specific research for policy never getting produced at all.
Current organisations focus on conducting policy research or informing policymakers, usually taking the form of a think tank or a lobbyist/advocacy organization. Work is being done to translate or communicate existing technical research for policy use, but no existing organisation (to my knowledge) directs technical research towards the specific solutions needed to enable legislation that already has political will behind it.
The feedback loops helped nuclear governance
Past emerging technologies have relied on direct influence and feedback loops between policy and technical research. With nuclear weapons, the technical research was done within the state which created direct access and influence between the two. Policymakers could direct technical research and technical researchers could inform and influence policymakers, creating direct and closed feedback loops between governance and research.
The AI Safety Atlas by CeSIA refers to this problem and the need for creating ‘epistemic communities’: “The development of nuclear arms control agreements wasn't solely the work of diplomats and politicians. It relied heavily on input from scientists, engineers, and other technical experts who understood the technology and its implications. These experts formed a network of professionals with recognized expertise in a particular domain, or as what political scientists call an ‘epistemic community’. They played important roles in shaping policy debates, providing technical advice, and even serving as back-channel diplomats during tense periods of the Cold War. Unlike nuclear physicists, who were often employed directly by governments, many AI experts work in the private sector, so a challenge to forming such networks for global AI governance will be ensuring that epistemic communities can effectively inform policy decisions.” [1]
An example of how direct feedback loops enabled nuclear arms legislation is the Comprehensive Nuclear-Test-Ban Treaty (CTBT) in 1996. Negotiations between the USSR, United Kingdom, and the United States had already begun in 1958 to create a comprehensive testing ban. But due to the lack of technical solutions of verifying compliance the treaty could not be passed. In May, 1960, the President of the United States therefore announced the expansion of the R&D program VELA UNIFORM with the goal to provide a markedly improved capability in detecting and identifying underground nuclear explosions during the coming two years. [2] This led to sustained, well-funded research on a narrowly scoped research question, and the development of Seismic Verification technology (that can detect underground nuclear testing) that enabled the CTBT. [3]
Without the explicit policy-directed research, this technology would not have been developed since academic and other research labs are driven by other incentives. Additionally, this technology was not developed by either capability or safety researchers, but by specialists in the specific field that was necessary for enabling that legislation. Although seismic verification technology was not sufficient on its own for a ban such as CTBT, it was necessary and without it, the CTBT would not have been possible.
Similarly, the Intermediate-Range Nuclear Forces (INF) Treaty in 1987 was enabled by perimeter-portal monitoring, a technological result of policy-directed research. In this instance, the research was directed before negotiations of the treaty. Henrietta Toivanen concludes that “Had the anticipatory research and development efforts not been made, the capabilities would not have been available when they were needed by diplomats.” [4] This shows that policy-directed research can be undertaken before negotiations begin, by anticipating the need instead of responding to it.
A proposed mechanism for feedback loops in AI safety
A potential solution to the gap between AI governance and technical research within AI safety would be an organisation that directly creates feedback loops between them. The following steps of how a feedback loop would function, should be the workflow of the organisation.
The first step would be to talk to policymakers, gather information on political will and what legislation would be feasible if certain technological solutions existed. This information would have to be fed into an internal or external technical research team where the direction of research is set solely by the question of how a certain regulation could be enabled technically. Even though the technical solution is often a necessity, it is as previously mentioned not sufficient for legislation. There also needs to be a political will, along with awareness of technical feasibility. To communicate and make policymakers aware of the technical solutions developed, they need to be translated into a policy context where it is clear how they can be used for legislation purposes. Therefore the organisation should also aim to package the technical results into a policy ready proposal and present it to policymakers.
This would create a closed feedback loop where policymakers can steer technological research towards solutions that enable effective AI governance.
If this bottleneck is correctly observed, this would allow for AI legislation to be adopted more quickly as well as being more technically grounded when they are decided upon. This would not only be beneficial on a national level but could also enable international treaties on AI governance where technical enforceability is crucial (as seen above with nuclear arms governance).
Possible failure modes
One plausible reason for this mechanism to fail could be that it would be premature to start this work before there is a clear need to enable specific legislation. The research leading to seismic verification technology was directed after international negotiations had begun, and there was a clear need for such a technology. Anticipating the need can be successful as with perimeter-portal monitoring enabling the INF treaty, but whether this would be true for AI governance remains an open question.
Additionally, AI governance differs from nuclear arms control in a significant way. Nuclear arms were developed by states and directly for military use, while AI is a general-purpose technology developed by private companies, which means that policymakers don’t have direct access to relevant expertise on AI as they did with nuclear arms. Due to this different context, policymakers might lack the understanding needed to regulate AI effectively.
Prematurity in work on policy-directed research could risk targeting the wrong research questions that do not directly enable legislation. Other problems, such as lack of insight into policymakers’ will, as well as poor tractability of the technical research questions, remain. One benefit of this approach rather than broad AI safety research is that the research questions would be more targeted and narrower in scope, which feasibly makes them more tractable to work on.
Considering the rapid AI development and increasing need for AI governance, we should aim to act sooner rather than later, and build these feedback loops now. This is especially true if development of technical solutions for legislation takes longer than building the political will for the legislation they would enable. The technical feasibility should be ready if and when legislation is considered, and should not be the bottleneck that prevents legislation from coming into force.
Even if these failure modes hold, work on this problem now would still be valuable, since it would enable more rapid action once the direction of political will becomes clear. It would also build the infrastructure needed for when this becomes the key bottleneck for effective AI governance.
TL;DR
Listen to policymakers and what they want to regulate → internal technical team researches solution → solution gets packaged into policy proposal → return policy proposal with technical solution to policymakers.
The neglected problem for effective AI governance
When discussing the lack of effective AI governance two problems are regularly addressed; political awareness and will, and technical knowledge reaching policymakers. But a key bottleneck is neglected: the lack of direct feedback loops between AI governance and technical AI safety research. Policymakers do not currently have a mechanism to directly influence or direct research toward specific solutions needed for policy. As a result, that risks the specific research for policy never getting produced at all.
Current organisations focus on conducting policy research or informing policymakers, usually taking the form of a think tank or a lobbyist/advocacy organization. Work is being done to translate or communicate existing technical research for policy use, but no existing organisation (to my knowledge) directs technical research towards the specific solutions needed to enable legislation that already has political will behind it.
The feedback loops helped nuclear governance
Past emerging technologies have relied on direct influence and feedback loops between policy and technical research. With nuclear weapons, the technical research was done within the state which created direct access and influence between the two. Policymakers could direct technical research and technical researchers could inform and influence policymakers, creating direct and closed feedback loops between governance and research.
The AI Safety Atlas by CeSIA refers to this problem and the need for creating ‘epistemic communities’: “The development of nuclear arms control agreements wasn't solely the work of diplomats and politicians. It relied heavily on input from scientists, engineers, and other technical experts who understood the technology and its implications. These experts formed a network of professionals with recognized expertise in a particular domain, or as what political scientists call an ‘epistemic community’. They played important roles in shaping policy debates, providing technical advice, and even serving as back-channel diplomats during tense periods of the Cold War. Unlike nuclear physicists, who were often employed directly by governments, many AI experts work in the private sector, so a challenge to forming such networks for global AI governance will be ensuring that epistemic communities can effectively inform policy decisions.” [1]
An example of how direct feedback loops enabled nuclear arms legislation is the Comprehensive Nuclear-Test-Ban Treaty (CTBT) in 1996. Negotiations between the USSR, United Kingdom, and the United States had already begun in 1958 to create a comprehensive testing ban. But due to the lack of technical solutions of verifying compliance the treaty could not be passed. In May, 1960, the President of the United States therefore announced the expansion of the R&D program VELA UNIFORM with the goal to provide a markedly improved capability in detecting and identifying underground nuclear explosions during the coming two years. [2] This led to sustained, well-funded research on a narrowly scoped research question, and the development of Seismic Verification technology (that can detect underground nuclear testing) that enabled the CTBT. [3]
Without the explicit policy-directed research, this technology would not have been developed since academic and other research labs are driven by other incentives. Additionally, this technology was not developed by either capability or safety researchers, but by specialists in the specific field that was necessary for enabling that legislation. Although seismic verification technology was not sufficient on its own for a ban such as CTBT, it was necessary and without it, the CTBT would not have been possible.
Similarly, the Intermediate-Range Nuclear Forces (INF) Treaty in 1987 was enabled by perimeter-portal monitoring, a technological result of policy-directed research. In this instance, the research was directed before negotiations of the treaty. Henrietta Toivanen concludes that “Had the anticipatory research and development efforts not been made, the capabilities would not have been available when they were needed by diplomats.” [4] This shows that policy-directed research can be undertaken before negotiations begin, by anticipating the need instead of responding to it.
A proposed mechanism for feedback loops in AI safety
A potential solution to the gap between AI governance and technical research within AI safety would be an organisation that directly creates feedback loops between them. The following steps of how a feedback loop would function, should be the workflow of the organisation.
The first step would be to talk to policymakers, gather information on political will and what legislation would be feasible if certain technological solutions existed. This information would have to be fed into an internal or external technical research team where the direction of research is set solely by the question of how a certain regulation could be enabled technically. Even though the technical solution is often a necessity, it is as previously mentioned not sufficient for legislation. There also needs to be a political will, along with awareness of technical feasibility. To communicate and make policymakers aware of the technical solutions developed, they need to be translated into a policy context where it is clear how they can be used for legislation purposes. Therefore the organisation should also aim to package the technical results into a policy ready proposal and present it to policymakers.
This would create a closed feedback loop where policymakers can steer technological research towards solutions that enable effective AI governance.
If this bottleneck is correctly observed, this would allow for AI legislation to be adopted more quickly as well as being more technically grounded when they are decided upon. This would not only be beneficial on a national level but could also enable international treaties on AI governance where technical enforceability is crucial (as seen above with nuclear arms governance).
Possible failure modes
One plausible reason for this mechanism to fail could be that it would be premature to start this work before there is a clear need to enable specific legislation. The research leading to seismic verification technology was directed after international negotiations had begun, and there was a clear need for such a technology. Anticipating the need can be successful as with perimeter-portal monitoring enabling the INF treaty, but whether this would be true for AI governance remains an open question.
Additionally, AI governance differs from nuclear arms control in a significant way. Nuclear arms were developed by states and directly for military use, while AI is a general-purpose technology developed by private companies, which means that policymakers don’t have direct access to relevant expertise on AI as they did with nuclear arms. Due to this different context, policymakers might lack the understanding needed to regulate AI effectively.
Prematurity in work on policy-directed research could risk targeting the wrong research questions that do not directly enable legislation. Other problems, such as lack of insight into policymakers’ will, as well as poor tractability of the technical research questions, remain. One benefit of this approach rather than broad AI safety research is that the research questions would be more targeted and narrower in scope, which feasibly makes them more tractable to work on.
Considering the rapid AI development and increasing need for AI governance, we should aim to act sooner rather than later, and build these feedback loops now. This is especially true if development of technical solutions for legislation takes longer than building the political will for the legislation they would enable. The technical feasibility should be ready if and when legislation is considered, and should not be the bottleneck that prevents legislation from coming into force.
Even if these failure modes hold, work on this problem now would still be valuable, since it would enable more rapid action once the direction of political will becomes clear. It would also build the infrastructure needed for when this becomes the key bottleneck for effective AI governance.
AI Safety Atlas, "Governance Architectures", Edition 1, Chapter 4.
Charles C. Bates, "Vela Uniform—Nation's Quest for Better Detection of Underground Nuclear Explosions". AAPG Bulletin 45, no. 1 (1961): 131.
Paul G. Richards, "Building the Global Seismographic Network for Nuclear Test Ban Monitoring", Lamont-Doherty Earth Observatory (1999).
Henrietta Toivanen, "The Significance of Strategic Foresight in Verification Technologies: A Case Study of the INF Treaty", Center for Global Security Research, Lawrence Livermore National Laboratory (2020).