Thanks for reading our thing & for this thoughtful critique. You've said a lot of things, and I'm writing quickly late at night so I'll be brief I'm afraid.
Re 1: We debated amongst ourselves how much structure to impose on the regulations. I think the way we did it is best. (a) If you have a better idea for a specific international governance structure that would be better than realpolitik, you are welcome to propose it, and we're glad to accept it as an improvement on our current Plan A if it works. We might put more effort into thinking of one later. (b) Even the relatively structureless thing we proposed -- research transparency + MACD but otherwise the nations of the world just have to muddle through and handle things on a case by case basis -- is a significant improvement over the status quo, for reasons we've articulated in the piece. You seem to disagree with this but I don't see why.
Re 2: Yeah I mean if China ends up catching up, or not realizing that a gap of a few months could be fatal, then that significantly undermines our argument for why they'd want a deal. OTOH it would make it more plausible that the US would want a deal. In general the party falling behind will have more reason to want a deal than the party in the lead. It seems like we have a disagreement about whether China will catch up, but it's probably not a massive one (neither of us are confident) right?
Re 3: Do you think the surveillance possibilities are less bad in Plan D, C, or B? I would argue that the surveillance possibilities are even worse in those worlds than in Plan A. For reasons we've discussed (greater number of frontier AI companies, greater number of countries with frontier AI, vastly more visibility of the public into how AIs are trained and regulated, making it harder for corporations and governments to abuse their power over AI) Only Plan S can seriously claim to be less scary viz-a-viz surveillance. If there's some plan different from all of the above that you prefer, can you sketch what it is?
Fwiw Plan A would still work OK without the open-weights ban and with a much larger fraction of inference ZDR; would those modifications make you happy? (i.e. it would still be better than the so-far-proposed alternatives IMO) I totally take the point that the US and China could start to do Plan A but then switch to a worse, more authoritarian version that e.g. doesn't have ZDR, but isn't that also an argument against the other plans on offer too? Do you have a proposal for how to stop governments from being authoritarian? Plan A does something at least -- it creates conditions so that more companies, across more countries, have frontier AI, and so that the public has more visibility into it all.
We really tried to make Plan A reduce concentration of power risks. One of the things that saddens me about the reaction by various people online (including some you approvingly link to) is that they seem to have come in with this prejudice that we AI safety people only care about AI takeover and are planning to concentrate power hugely in the service of our perceived greater good of stopping the misaligned AIs. This stereotype has some basis in reality -- various past proposals by various AI safety people were basically like that, and I'd argue that the current plans of Anthropic and OpenAI are basically that or worse -- but we really truly do care about concentration of power and we thought a lot about how to prevent it, and I hope it came across in the final product.
1Without the open-weights ban I would expect terrorists engineering pandemics. As for the ZDR policy I would guess that data is not to be retained if it comes from a weak AI or if a classifier decides that the data is not related to hazardous topics in a manner similar to deciding not to reroute queries from Fable 5 to Opus 4.8.
On the other hand, I wonder if concentration of power can't actually be prevented even by perfect epistemics and coordination of the humans.
Until @Daniel Kokotajlo or another author provides a response, I think that I see the following objections:
- The AI-2040 authors presented other plans competitive with their variant of a systematic oversight: plan S, the domestic-first Plan A, compute restrictions aka Plan A*, GPU arms control, CERN for AI. The AI-2040 Plan A as-written does mean that the deal requires systematic renegotiations. Most likely you would approve the idea of a UN-accountable monopoly on AI.
- Suppose that China does see the deal as unnecessary and the USA implements its Plan A. Then China would instantly learn anything about American TAI's capabilities. Once the world sees, say, robots having a doubling time of less than three months, China instantly learns about them and tries to replicate the success. Alas, without the deal, nothing stops China from implementing potentially hazardous architectures which the USA rejected. Once that happens, either we or China are screwed.
- Why can't one, say, implement a classifier and warn the user that the data WILL be retained because the request was related to something potentially hazardous, like bioweapons, then de-retent the data if it is found to be safe, then have the lab transparently verify that the protocols don't cause private information to be retained unless it's actually necessary?
I very much agree with you first point (see my comment here), I'm also somewhat sympathetic to your second point, but I'm a bit confused about the third.
Isn't it already true that the US government can in theory surveil all the conversations you have with closed-source models and with open-source models running on AWS and other large servers?
So is the only difference in Plan A that you lose locally-run open-weight models? I agree that's a real loss, but my understanding is that currently only very few tech-savvy and unusually privacy-loving people use those. Plus it seems pretty likely that competeitive open-weight development will stop anyway due to economic reasons and normal government regulations.
At least in Plan A there are some official efforts towards privacy-preserving technologies (though you are right that there is very little guarantee that nations don't secretly pull out of the privacy-preserving rules). And more importantly, multiple countries have their own AI and datacenters, so they can compete for customers by showing stronger verification mechanisms that they are not actually spying on you unless a reliable classifier flags that you are building superintelligence or WMDs. In particular, I can imagine that many small European governments will want to use the European AI for processing state secrets, so the different European governments will set up a pretty reliable multi-party auditing mechanism to verify that none of the others are spying on the conversations of the European AI. If that happens, you can also use the European AI if you care a lot about privacy.
Maybe I'm missing something, but Plan A doesn't seem worse to me than the default trajectory from a privacy perspective.
The "AI Futures Project" has released their AI 2040: Plan A scenario.
While their previous scenario AI 2027 was a forecast of what they thought a future with many powerful AIs would look like, AI 2040 is intended to be normative -- it's a description of what one ought to do, granted the likelihood of a future with many powerful AIs.
I'm going to review some objections I have to their proposal as a normative plan. Some are within-frame objections -- reasons that I expect trying for the AI 2040 plan that would fail to accomplish the goals of the authors. Others are my own objections -- reasons that I expect trying for the AI 2040 plan would destroy things that I, personally, care about.
Before I start: two elements of the plan that I like.
First, in their "incremental AI policy wishlist" -- the ideal policy that one should execute soon -- AI 2040 recommends trying to limit the gap between the intelligence of internal and external model deployments, i.e., the gap between the "intelligence" accessible to Anthropic / OpenAI employees and to everyone else. I'm a fan of efforts in this direction; equality of intelligence between the insiders and outsiders, the government and the public, seems likely to help people understand AI more, and to help spread the benefits of AI to everyone.
Second, this plan includes measures to ensure that people outside AI companies can understand how AIs are trained. I'm uncertain about their implementation of this -- "radical transparency." But in general; I think broad, public knowledge of how AIs work and how their training works is good. Without open, reproducible AI science it's going to be impossible for people to orient around what's going on with AI. Significant parts of AI safety have previously advocated for knowledge of how AI works to be kept mostly secret, and I'd be happy for such advocacy to end.
Alright; to some objections.
1. The Deal Has No Actual Decision Procedures
AI 2040 is subtitled "Plan A — The Deal." But the plan really contains only half of a deal.
That is, Plan A has a lot of detail about (1) making sure all compute use is visible to governments and (2) making sure that both the US and China are locked into a mutually-assured-compute-destruction stalemate, where one can destroy the compute of the other so long as they're willing to be destroyed in turn.
But it has basically no detail about the procedures that would bind how governments permit and prohibit any specific use of that compute, short of that destruction. That is, it has lots of details about how individual companies might be bound by regulators internal to nations; but it has no details about procedures for international negotiation between nations about what these regulators should or should not prohibit.
That's a huge problem; if one's proposal is for two nations to put themselves in a mutually-vulnerable situation, where each one could cripple the economy of the other more or less at will, then before accepting the proposal I expect these nations would reasonably want to know what rules or procedures would be used to settle their disagreements short of such complete destruction. But there are no such procedures proposed by AI 2040.
Consider how AI 2040 describes one such conflict between governments over what is prohibited:
There are a lot of problems with this scenario. The biggest, though, is that it depicts the "deal" as solely a transparency mechanism, a kind of channel that permits the well-informed brute exercise of force. After "radical transparency" surfaces some particular training practice, what determines the prohibition or acceptance of this training practice is if it is "generally agreed to be unsafe by a majority of nations (weighted by bargaining clout)".
And well, perhaps the authors of AI 2040 would respond that indeed, they are merely proposing a channel that lets the US and China exercise brute force in a well-informed way. But this kind of realpolitik would be fake wisdom; actual agreements between peers are usually meant to be something other than avenues for such an exercise of brute force, and nations would be reluctant to sign them if this were not so.
That is, in general, actual agreements, contracts or Constitutions are meant to constrain the space in which bargaining takes place to something smaller and more determinate than the space in which bargaining took place before the agreement: for instance, the World Trade Organization dispute settlement system is supposed to function by offering procedures that allow agreements different than the agreements that would dictated by a naked balance of power. So if you propose an international agreement, and your proposed decision procedure is "Xi and the President yell at each other and bargain and threaten (!!)," then you've failed to offer the chief thing that international agreements are supposed to supply. We need the game theory about decision as well as the game theory about destruction.
This is an obstacle to the acceptance of the proposal, as well as an obstacle to its execution, because the shadow of the future determines the present. Nations would be reasonably extremely hesitant to sign a deal, where the result of a bargaining failure is "the obliteration of an increasingly-large segment of their entire economy" without some procedures about how they would settle disagreements before so obliterating that segment. Consider the chain of thought: "Yeah, if we think the other nation is doing something unsafe, we flip the switch that obliterates their most valuable investments, then they obliterate ours." --- "What do we do before then?" -- "Idk, we yell at each other and threaten each other?" You will note how incomplete this feels. Is the plan to have a regularly scheduled Cuban Missile Crisis?
A further problem with this proposal is that there's very little reason to expect a weighed-by-bargaining-clout decision procedure to result in wise decisions. This gets into how others have critiqued the scenario for "selective optimism."
That is, they call this "agreement" between-nations the Consortium. But it's unclear whether this scenario is a (1) forecast that a Consortium dominated by the powerful would make wise decisions or (2) a hope that a Consortium so established would make wise decisions. It's clear in the scenario that the Consortium does make wise decisions, from the perspective of the authors. It steers AI development directions: "AIs that are released publicly by the Consortium should be bad at AI research." It applies verification methods to the robot workforce. It pauses AI capability development at one point. These are hugely consequential decisions. But again, in the absence of any determinate decision procedure other than a balance power, the authors should be unsure whether the Consortium will actually do what they think it should do in such moments.
Alternately, the authors might respond that there would be some such determinate decision procedures, they just haven't figured them out yet. But I don't think the absence is an accident; any such decision procedure that would be acceptable to the US would tend to be unacceptable to China, and vice-versa. If both China and the US are the only partners in this, how do they settle their disagreements? If other nations get a vote, will either China or the US be happy to cede the tiebreaker vote to such nations?
And of course, any specific mechanism design might also result in unhappy equilibria.
This whole proposal is taking place because the authors are unhappy with the dynamics resulting from competition between rival AI companies; but they have no guarantee that the incentives governing some actual intergovernmental institution would be better. Refraining from detailing the mechanisms of such an institution merely means that the authors will be unable to identify such perverse incentives ahead of time; not that there wouldn't be bad ones.
Additionally, I think the lack of modeling such incentives is the kind of thing -- generally -- that lies behind the optimism that AI 2040 has for top-down solutions. It's easy to think that a particular dynamic, multipolar system, would be better replaced with a system that you model as a point mass. But sadly, neither AI systems nor human systems are well modeled as masses.
2. China Will Likely See Such a Deal as Unnecessary
One reason that the scenario predicts it will be pretty easy to get China to join a deal is because if they do not join a deal, they will be disempowered.
In general, I'm just much less confident than many in AI safety that China will fall behind the West.
Right now the US has perhaps an 8-month lead over China in the quality of its AI models; it also has a lead over China in tons-to-orbit and the production of commercial jets. By contrast, China leads the US in the production of electric cars, batteries, solar panels, rare earths, most metals, quadcopters, mid-range drones, transformers, power plants, electricity, humanoid robots, industrial robots, CNC machines, high-speed rail, mature-node semiconductors, and an increasing number of other kinds of technology.
I remain somewhat unsure whether this 8-month lead of the US over China is going to grow or narrow. I also remain somewhat unsure whether the AI-takeoff is going to be so fast that an 8-month lead would result in across-the-board US dominance. And even if the 8-month lead remains, and even if an 8-month lead would result in across-the-board dominance, I finally also remain somewhat unsure whether China would perceive such an actual imminent dominance as being so. All this leads me to be uncertain whether China would have interest in a world-historically invasive deal to prevent its own obsolescence.
China is a rising and confident power; it would be quite a turn for a mere few years to move them to think they require one of the most invasive deals in world history to prevent their downfall, even in the uncertain world in which this is actually true.
This appears to me a pretty big obstacle and I'm not sure how they plan to overcome it. I think that the overall belief of the authors is that -- because the authors of AI 2040 believe themselves to have true beliefs about the world -- China's beliefs will converge on what they believe in the future. I'd like to note that AI 2027 made predictions on the basis of some such similar convergence, and as far as I can tell they were worse than my own predictions.
3. The Surveillance Possibilities are Actually Quite Bad
The standard AI 2040 proposal involves locating all (or mostly all) use of compute in inference, and making it physically accessible for government monitoring as plaintext through optical taps. This means that it would be technically trivial for government to surveil basically anything that runs through AI inference -- which, given that the authors expect the entire economy & all society to run on AI inference, is basically everything. Like others, I find this alarming.
It's easy to be a bit confused here, because the plan also mentions zero-data retention policies for consumers. In general "zero-data retention" is a design choice wherein AI companies do not store the prompts and queries sent to AI. Actually implementing zero-data retention is thus a kind of guarantee of privacy for consumers. But the real guarantees AI 2040 has around this seem to be quite thin.
First, they are thin because only a small number of comparatively stupid AIs are permitted to be run in this way. Their proposal is for there to be a cap of a hundred thousand H100-equivalents devoted worldwide to zero-data retention inference for consumers; for there to be a cap of a hundred million H100-equivalents with probabilistic ZDR; and for there to be a cap of a hundred billion H100-equivalents used with no-ZDR inference at all. So, the vast majority of FLOPs of compute are those that involve no ZDR. And remember, the plan also involves banning any further advanced open-weight models, so you're only ever going to run inference on those monitored computers!
Second, they actually have few mechanisms in place to ensure that ZDR is actually enforced where they would like it to be enforced, so far as I can tell. That is, there are elaborate game-theoretic proposals to prevent nations from pulling out of the mutually-assured compute destruction agreement once they enter it, but there are no such proposals to ensure that either the US or China actually sticks to ZDR. But it's a completely detachable part of the plan; and if something like this were to happen, I expect it would be detached. The authors do not -- as they might say of other stories of how AI goes well -- actually have a plan for making sure genuine privacy happens. They have a hope, which is not a plan.
Again, I really want to emphasize that in this scenario everything that matters on Earth flows through inference. If you want to run a business, you'll get advice from an AI. If you want to engineer a product, you'll do it with AI. If you want to run for politics, you'll strategize with an AI. If you don't think "physical access to all AI activity" is a big deal then you aren't taking AI seriously.
Consider a proposal that would give the government physical access to every file on your computer. Would you be alarmed by this? Then I think you should be alarmed by AI 2040. Is there any similar proposal of absolutely universal oversight in world history that you believe to have been justified? If not, why is this an exception?
Conclusion
I'm not really happy with any of the above as a summary of my objections. Overall I probably feel worse about the scenario than my views above reflect, and haven't summarized my reasoning here in a way I find totally satisfactory.
(crosspost from my blog)