ControlAI's mission is to avert the extinction risks posed by superintelligent AI. We believe that in order to do this, we must secure an international prohibition on its development.

We're working to make this happen through what we believe is the most natural and promising approach: helping decision-makers in governments and the public understand the risks and take action.

We believe that ControlAI can achieve an international prohibition on ASI development if scaled sufficiently. We estimate that it would take approximately a $50 million yearly budget in funding to give us a concrete chance at achieving this in the next few years.

In this post, we lay out some of the reasoning behind this estimate, and explain how additional funding past that threshold, including and beyond $500 million, would continue to significantly improve our chances of preventing extinction risk from ASI.

Preventing ASI 101

Negotiating, implementing and enforcing an international prohibition on ASI is, in and of itself, not the work of a single non-profit. You need to have the weight of nations behind you to achieve this kind of goal. If humanity manages to achieve an international ban on ASI, it'll be through the efforts of a sufficiently motivated, sufficiently powerful initial coalition of countries.

Assuming that we work in multiple countries in parallel, we could say the problem statement is: get each country to be motivated to achieve an international prohibition on ASI. It’s not obvious what it means for a country to be “motivated” to do something, so it’s worth taking a second to unpack.

Our full theory of change chart, which backtracks from the desired outcome to our currently running workstreams.

Normally, parts of a country's executive branch are responsible for international negotiations around urgent issues concerning national and global security. In practice, these are the groups who need to be sufficiently motivated to achieve the ban to throw their weight behind it.

Branches of the government are generally not in the business of independently taking bold positions and then pursuing those positions to their logical ends. Instead, their stances and actions are mostly shaped by prevailing social currents.

Some of these currents are informal. This includes things like the conversations they have with their colleagues, advisors, confidants and family members. It also includes an...

bkhgvk

bkhgvk