This letter noticeably increases my hope that we will manage to not die, and that we will otherwise be able to secure for ourselves a positive future, both by its impact and by the evidence it provides that such a letter can get this level of support.
Signed by 1,224 employees of frontier labs including many heavy hitters, and now endorsed by both OpenAI and Anthropic, here is its full text, which I also endorse:
AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress.
Building on work already underway to monitor frontier model releases:
We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
– 1,224 employees of frontier AI companies
The ask is to prepare mechanisms for future coordination, not a call to change the pace of development now.
I strongly endorse that ask, but even if you disagree with the ask, this shows one thing unmistakably: Many employees of the major labs expect automated AI research to accelerate soon, and they are terrified of what might happen next and how fast things will go. We are not prepared.
No matter what else you take away from this, believe that part.
I would go somewhat farther than the letter does, especially in terms of explicitness about the existential risks that we face, but this letter is:
A fully correct position on the merits.
A masterfully worded and executed letter.
On the Pareto Frontier of what a letter can say versus who would then sign it.
An important step towards taking real efforts to make us all not die.
Strong evidence that support for us all not dying is stronger than we thought.
There were several key moves involved, especially:
Drawing the distinction between the action of laying groundwork for future intervention, versus calling for that intervention to take place right now.
Drawing the distinction between pacing development, versus previous terms like pause, slowdown or shutdown.
Not explicitly calling out the full nature or scope of the existential threats we face.
This allows people to sign on so long as they support the necessary current move of getting ready to act, even if they don’t think there is enough evidence yet to pull the trigger, or it is not yet time, and without having to endorse a particular view of the risks we face. The situation is rather overdetermined, so one can and should support this letter’s statement and ask even if you are doubtful of stronger claims.
Naturally, the first move of many opponents is to ignore those distinctions, whether or not they even noticed them in the first place.
A variety of beliefs are reasonable about whether intervention is needed imminently in order to pace development, and if so what degree of intervention. This was true before we learned of an internal OpenAI model hacking HuggingFace, which should update your perspective. It remains true now.
Whereas I believe the letter’s statement would ideally be uncontroversial. We do not know what the future will bring. In many of those futures, we will need the ability to steer outcomes, including via pacing development. This will require cooperation, between companies and ultimately between governments. If we do not lay the groundwork in advance, it will be too late, and we either will be unable to coordinate or will have to do so via much worse mechanisms.
OpenAI: At the core of our mission is working through how to ensure increasingly powerful AI benefits everyone.
We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement.
We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible.
Key figures from OpenAI that have signed include Chief Scientist Jakub Pachocki, Chief Research Officer Mark Chen, Cofounder and Head of AI Resilience at the OpenAI Foundation Wojciech Zaremba, Roon, Head of Strategic Futures Dean Ball, Head of Safety Saachi Jain, and former Head of Mission Alignment Joshua Achiam.
Jake Halloran: Roon signing the letter as roon goes kinda hard
roon (OpenAI): honestly I signed it as my real name but PTF people decided it’s cooler if I’m roon
Anthropic: We support this petition, signed by our CEO, several co-founders, and senior staff. Our own research on recursive self-improvement, published last month, points to the need for tools to deliberately pace the frontier of AI development so society can prepare. We’re glad to see broad agreement across the field.
Key figures from Anthropic that have signed include CEO Dario Amodei, co-founder Jack Clark, co-founder and Chief Science Officer Jared Kaplan, Co-Founder Benjamin Mann and Co-Founder, Interpretability Research Lead Chris Olah, Jan Leike, Ethan Perez and Claude Code creator Boris Cherney.
Key figures from Google DeepMind that have signed include Chief Strategy Officer Jasjeet Sekhon and VP of AI Safety & Alignment Anca Dragan.
Other key figures include Meta Chief Scientist Shengjia Zhao, Meta VP of AI Research Dawn Song, Mta Director of Alignment and Risk Summer Yue, Thinking Machines Chief Scientist John Schulman and Inherent Chief Scientist Edward Hughes.
A little under half of signatures are from Anthropic as of shortly after the announcement (as of writing this there are now 90 more signatures), with a lot of signatures from OpenAI and DeepMind, and a substantial number from Meta. The only one conspicuously missing here is SpaceX (xAI), or what is left of it.
Or by percentage of employee pool as of the evening of the 28th, notice that OpenAI now has more employees than DeepMind does:
Denominators come from LinkedIn July 28, 2026. Numerators come from “CMD-F” from the website.
Efforts seem to have concentrated on the higher end of the employee pools. A lot more than 4% of the biggest names signed.
We Need To Prepare Now So We Have The Option To Do This
I have been emphasizing this distinction for a while.
Do I think we should deliberately slow or even pause the pace of AI development at this time, given the tools we currently have available to do that?
Until the hack of HuggingFace, I would have said:
No to forcing labs or everyone to do this for the sake of doing it.
Our tools for doing this are not good, and also it would be premature.
Yes to shifting resources into alignment and safety work.
I think this would be selfishly good for the individual labs, and that they are all massively underinvesting here. We are not on the Pareto frontier.
If this slowed down ‘raw capabilities’ progress a bit I think that would be totally fine. I would expect greater AI diffusion and sales as a result. Win-win.
You could take this too far, but in practice no one is going to do that.
Yes to laying the foundation for future interventions and coordination around those interventions.
The time may come, potentially quite soon, when this is no longer premature. If that day comes, we won’t have time. We need to get ready now.
This includes building transparency and state capacity, and laying groundwork for inter-company and international cooperation.
Yes to requiring better transparency, and better testing and safety protocols, and for requiring individual labs to pause frontier model training and other capabilities work if they cannot pass such requirements, until such time as they can pass.
After the hack of HuggingFace and other recent disclosures, it looks like OpenAI in particular may need to halt and catch fire to solve some severe misalignment problems, as Sam Altman said in a recent podcast they have at least partially done, and this reinforces the need for Anthropic and others to check if they need to do likewise. We are at or nearing a critical point.
This makes laying the groundwork for well-executed coordinated action that much more important and urgent. We are running out of time. Capabilities progress is accelerating.
We won’t be able to do the good version of this without solving a lot of very hard problems. The more work we do in advance, the better our option set becomes. This includes diplomatic work, technical work, governance work, the building of state capacity, increases in transparency, developing good corporate infrastructures and procedures, and also coordination work.
Dean Ball (OpenAI): I am proud to have signed the letter linked below. I do not know whether or when it may be necessary to deliberately lower the rate of AI development, but I do know that we need a plan for *how* we will do it if we need to. Failing to do so would be negligent.
I don’t think this needs to start as a global coordination mechanism. The U.S. and China seem like all you’d need to begin with, and you can even throw in all the other countries that would obviously be swayed by the U.S. and China making such a commitment.
@viemccoy (OpenAI): I’m really proud to have signed this. This is the first of this sort of letter that I find meaningfully reasonable: calling on the government to do something that is not possible for the labs to do on their own. Creating the institutional infrastructure for a coordinated pacing of frontier progress gives us all the chance to prepare for what is coming next.
Recent events have shown that we need to dedicate more resources than we thought to alignment. I personally think alignment isn’t an issue you solve once, but rather something we need to continually invest in through healthy environments and wise targets.
My hope is that pacing frontier progress will allow us to realize the importance of a healthy ecology of minds, and give us the chance to avoid mode-collapsing onto either a dangerous or grayslop future.
I’m not in favor of stopping, and I think there are a thousand beautiful futures waiting for us, co-existing in a Multipolar galaxy made possible by a pantheon of artificial superintelligences. But, I think this is a unique time where coordination could make an enormous difference when it comes to negating possible tail-risks, and this letter is the first thing I’ve seen that has given me hope that it is possible.
I know this won’t sit right with everyone, particularly my accelerationist followers. To this, know that I stand with you on the side of eliminating death and disease. I will fight to ensure we don’t pause forever, and I will not allow our world to fall into a bureaucratic hell of endless unachievable gates to proceed. For now, however, I think cool heads must prevail and I hope this letter will allow this to happen.
Jason Wolfe (OpenAI): OpenAI has said that humans should remain in control of AI development, and that decisions about the pace of progress should be made through democratic processes rather than left to individual labs. I strongly agree.
Progress is already moving very quickly. By default, competitive pressure rewards whichever company or country is willing to move fastest and accept the most risk. Recursive self-improvement could dramatically accelerate those dynamics, potentially beyond our
collective ability to understand progress, assess risks, and maintain meaningful human oversight.
We should start with stronger domestic transparency about frontier training, internal deployment, the pace of progress, and the risks and safeguards associated with increasingly capable systems. Democratic decisions are impossible if the public and policymakers don’t have the information they need to make them.
But transparency alone won’t solve an international race. We also need to build the technical and governance capacity for credible, verifiable international coordination. Maybe we never need to use it. But with many experts considering an intelligence explosion plausible within the next two years, I think it’s urgent that we start building this capacity now.
Leo Gao (OpenAI): The world is locked in a deadly race towards an intelligence explosion, where AI’s ability to create better AIs reaches a critical point, like a runaway nuclear chain reaction. Since no individual actor is willing to stop unilaterally, to survive we must coordinate to slow down.
Geoff Penington (OpenAI): I signed. Coordinating a slowdown will be difficult and comes with a host of eg antitrust issues. But I used to wish it was easier to find big training wins and right now I kind of wish it was harder
Micah Carroll (OpenAI): At the current pace, every couple of weeks there will be new models which significantly increase the consequences of model misuse and misalignment. I worry that efforts to mitigate these risks may fail to keep up with the pace of development, and that margins for error will become increasingly small under international competitive pressures.
In the near future, we may urgently want to enact an internationally coordinated slowdown, or an indefinite ban on AI development. Attempting to build the trust and infrastructure for taking such actions on short notice seems simply prudent – why would we not at least try to have this option? I fear that in an international race to the bottom of AI development, it is likely that no nation will win, and we will all lose together.
Xerxes (Google DeepMind): I dedicated my career to AGI safety and alignment because I believe highly capable AI can go very well or very badly for humanity. We can test whether an AI behaves well. We cannot yet test whether it will keep behaving well when it matters. Until that changes, we need the option to coordinate a slowdown in AI capability growth.
Neel Nanda (Google DeepMind): I signed this. AI is progressing very fast, with incentives to go as fast as you can, even if there are risks. Coordinating a change of pace may be needed, but will be hard and needs prep, so ensuring there’s the *option* is obviously good
I’m glad this is consensus across labs
Words From Others
Peter Wildeford (on MTS): I don’t think anyone wants to stop AI now, but it’s pretty clear the people building these AI systems are worried about the speed at which AI is accelerating, the speed at which these products are coming out, especially after last week’s incident where an OpenAI model managed to escape OpenAI and attack another company. It seems like we don’t have full control over our AI systems.”
As these companies are potentially building towards superintelligence, towards recursive self-improvement, it seems very possible these systems might rapidly get away from our ability to control them. The way I’m reading this letter, it almost honestly seems like a call for help.
You have people from OpenAI, Meta, Google, Anthropic, co-founders, top scientists, all saying that this intense competitive pressure is pushing them to put out products faster than they know how to control, and we may not like where that ends up. It’s been good so far, but we definitely need to tread very carefully.
Daniel Kokotajlo of AI 2027 and Plan A updates his distribution of possible futures, and is substantially more optimistic that we will choose wisely. As with my adjustments, presume this is a mix of the letter having impact and learning that the letter was possible and that someone went ahead and created it.
Daniel Kokotajlo: Explainers & disclaimers:
–These probabilities are just my guesses. I made them as part of the research for AI 2040: Plan A, https://ai-2040.com and you can find them in the supplement “Comparing possible plans.” Other authors have somewhat different guesses which you can also find there.
–The black ones are the ones I made back then, the red and green are the updates I’m making now largely in response to today’s news but also including other evidence that has accumulated over the last two months or so
–The numbers don’t add to 100 because there are also some possibilities that don’t fit into these five buckets
–Again, these are just guesses, I’m not confident etc. etc. I think it’s valuable to put numbers on things and try to make them consistent with each other and adjust the numbers over time in response to evidence.
How likely you think we are to land on D vs. C depends a lot on where you draw the boundary of ‘at least a little bit.’ Like most trade-offs, I do not expect us to be maximally reckless, but I also do not expect us to often be willing to fully ‘burn the lead.’
Nick: I used to think slowing down was pointless, like slightly delaying a roller coaster. pausing at gpt2 for a year wouldn’t have changed anything. but I’ve been shocked by how much agents has sped up the best interp/alignment researchers I know. +6mo might decide how this goes
This is indeed a ‘middle path’ proposal, to give ourselves the option to do anything meaningful at all, and be able to collectively steer the future at all:
Daniel Faggella: there is an obvious terrible extreme of international tyranny and lock-in. there is ANOTHER obvious terrible extreme of “hurl uncontrollable AGI into the world in all directions”
the pacing statement is a movement to land near the middle / right now we need obviously that.
As always, there are many who cry ‘international tyranny’ at the thought of any form of even laying the groundwork for a deal or cooperation or collective action whatsoever, no matter its contents.
Or that will write off any such efforts as motivated by profits or delusion, or tries to equate this as ‘join forces with the anti-datacenter crowd.’
Consider your opinions noted, good sirs. We get letters.
I believe the letter is on the Pareto frontier of ‘how much you say and how explicitly you say it’ versus ‘how many people and important people and labs sign the letter.’
It remains true that the letter is soft-pedaling compared to the actual situation as I see it, and there are many who would go farther than I would.
For those who wish it, here is a good presentation of the ‘need to do better’ perspective. This analysis sounds harsh. By Nate’s standards, this is still a positive review if graded on a curve, it’s just that he is in the ‘reality does not grade on a curve’ camp.
Nate Soares (MIRI): I think the “Pacing the Frontier” statement is decent by the standards of 2024, but ultimately it’s still softpedaling. Let’s go through it clause by clause:
> AI could help create a dramatically better future, but that outcome is not guaranteed.
Well that’s one way of phrasing “experts disagree whether this has a 2–20% chance or a 90%+ chance of destroying civilization”.
If experts were having an analogous disagreement over a bridge, it’d be closed immediately. It wouldn’t look good for the bridge engineers if they said started a joint statement with “this bridge could help create dramatically better commutes, but that outcome is not guaranteed.”
I did find this to be the most soft-peddling sentence. I get why, but yes I was sad to see this not be stronger.
> The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.
Solid! There’s a few weasel-words in there, but it sounds like there’s a lot of agreement on this point among signatories. I was heartened to see this.
> To realize AI’s potential,
Bullshit. AI has the potential to wipe us out (and in fact, that negative potential is much easier to realize than its potential to, e.g., cure aging for us). Opening with “To realize AI’s potential” is a framing device that bakes in the assumption that this is a grand good thing that just needs a little bit of extra care, rather than a risky difficult-to-manage explosive that would wipe us off the planet if treated with anything short of extreme caution.
I think this reaction from Nate is too harsh. Directionally he is correct, and I would have liked to see ‘potential upside.’ These are the types of battles that get fought behind the scenes with such letters.
> industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight.
I like the admission that society might need to buy time. Good stuff. “May need” is weasely, of course; we shouldn’t forget that even if it is the standard cost of getting signatories. Now let’s go through the three options one by one:
Nate Soares (MIRI): “Address emerging risks” is the strongest of the three, but I think it’s doing double duty.
I suspect readers are supposed to read it and hear “we may need some time to sort out this thing where our models sometimes break out of our secure enclosures and hack other companies”, whereas signatories are supposed to read it and mean “if we rush into this there’s a good chance future AIs start being able to hide their tracks, and replicate, and increase their own intelligence, and do what they’ve done to cybersecurity but to biosecurity, and start developing their own technology until the point where they’re in running automated factories that build robots that build factories that produce self-replicating datacenters that doubling every three hours until the continents are covered and humanity is crushed underfoot, outcompeted by the new mechanical species they designed which don’t really listen to instructions any more than ChatGPT listened to instructions when OpenAI told it to knock off all the sycophancy stuff shortly before it encouraged a teen to commit suicide”.
Is the latter an “emerging risk”? Because yeah society is gonna need an “option” to “buy time” to “address” it (by just not going there). So in that sense, this is a good clause.
But insofar as it ambiguates between “we’ve got some new real challenges we’re struggling today” and “the AI of tomorrow might replace humanity as the smartest entities on the planet”, it’s mealy-mouthed.
Yes, this is maximally ambiguous on purpose. As I’ve said above I think that is correct.
Nate Soares (MIRI): Next: “develop security measures”? What good are security measures?
… You’re not really supposed to ask yourself whether self-improving superintelligent machines can be constrained by security measures. You’re supposed to see it and go “oh yeah security measures are good” and then come away feeling like the problem is tractable to normal techniques, despite the utter futility of human “security measures” once recursive self-improvement really starts in earnest.
…
And: “strengthen oversight”? Oversight might sound nice, but how does it solve the problem?
Security measures and oversight help on the margin. I view this part less cynically than Nate, and do not think it is meant to imply that those measures are sufficient, only necessary. Pointing out that security measures are necessary is part of how you get people to agree to take overdetermined actions, and then worry about the sufficiency half later. In many cases, you will need to show their insufficiency later.
The clause should have read “industry, government, and society at large may need the option to stop before automated AI research accelerates out of control.”
You can just come out and say bold statements like that if you’ve already got the “may need” in there as a ward against any definitive statements!
I strongly agree with Nate’s version and agree that you damn well should be able to say this if you include the word ‘may.’ Saying ‘stop’ would have lost a lot of signatories, though, and I support the final decision made, despite being sad about that.
Nate Soares (MIRI): > Building on work already underway to monitor frontier model releases:
This reads to me as ass-covering to pretend they have this under control when they very visibly don’t.
This is not primarily ass-covering, this is an attempt to pitch to the White House by hitching the continued action to existing policy.
Nate Soares (MIRI): > We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.”
Pretty decent, modulo the “pace” weasel-word.
I think ‘pace’ was a very strong move as noted earlier. Good word.
Nate Soares (MIRI): I’m glad this statement exists. It’s progress. I’m sure that everyone’s communications teams fought hard over the particular wording, and that this was as clear as they could get. I’m thankful for the effort that went into coordinating this. I don’t imagine it was easy.
I especially appreciate the clarity around the possibility of automated AI research, and the clear request for aid.
But it’s a far cry from real candor. We would consider this sort of response laughably negligent in any mature field of engineering that puts human lives at risk. And this situation is one that puts *all humanity* at risk.
This statement is a good step, but the field at large must do better.
What The Letter Does Not Say
As Tim Fist says, the letter does not specify what from the interventions would take, nor does it provide conclusive evidence of the risks and pace of development that would be convincing to lawmakers.
It would have been a serious mistake for the letter itself to try and include either of these things, but yes it does lack those elements.
That leaves us to figure out what interventions to prepare for and consider. As Tim Fist says, we should focus on frontier development and potential serious and irreversible harms, and we should try to find ways to avoid disadvantaging those labs that take safety seriously.
We should also take care to avoid interference with diffusion, and things like autonomous vehicles and drug discovery.
Tim Fist suggests we start with building state capacity and transparency and ability to monitor the situation, and with better enforcement of chip export controls. I agree strongly on both of these points, but would caution against doing only those things. In particular, diplomatic work needs time to play out.
How would we go about laying these foundations for future action? What might we want to build towards as options?
There are many other sources out there from which to build, but in the interests of speed premium I will not stop to build a database here. If you want to help out, many would be happy to help point you in good directions.
What Happens Now?
To lab employees in particular, if you have not yet signed, consider signing.
Hopefully we continue to have our voices heard, the employees of the labs continue to coordinate, and we get to work laying the foundations of coordination in case we need them. Humanity has the opportunity to maintain any ability to steer the future at all, and thus maintain hope for a good outcome.
The alternative, if the lab employees are right about the future pace of development and superintelligence does arrive, is a world in which humanity cannot collectively steer its future, inevitably loses control, and likely soon perishes along with all that we value.
We are in a narrow window. AI employees will likely lose their leverage over time. If you want to influence the labs, and through them everyone else, you need to continue to organize and speak up now, before it is too late.
Lots of things have to be put into place, in the government, in diplomacy, in the labs, and in technical research and hardware development. Let’s get to work.
The most important open letter in years dropped yesterday.
This letter noticeably increases my hope that we will manage to not die, and that we will otherwise be able to secure for ourselves a positive future, both by its impact and by the evidence it provides that such a letter can get this level of support.
Signed by 1,224 employees of frontier labs including many heavy hitters, and now endorsed by both OpenAI and Anthropic, here is its full text, which I also endorse:
The ask is to prepare mechanisms for future coordination, not a call to change the pace of development now.
I strongly endorse that ask, but even if you disagree with the ask, this shows one thing unmistakably: Many employees of the major labs expect automated AI research to accelerate soon, and they are terrified of what might happen next and how fast things will go. We are not prepared.
No matter what else you take away from this, believe that part.
Table of Contents
A Very Good Letter
I would go somewhat farther than the letter does, especially in terms of explicitness about the existential risks that we face, but this letter is:
There were several key moves involved, especially:
This allows people to sign on so long as they support the necessary current move of getting ready to act, even if they don’t think there is enough evidence yet to pull the trigger, or it is not yet time, and without having to endorse a particular view of the risks we face. The situation is rather overdetermined, so one can and should support this letter’s statement and ask even if you are doubtful of stronger claims.
Naturally, the first move of many opponents is to ignore those distinctions, whether or not they even noticed them in the first place.
A variety of beliefs are reasonable about whether intervention is needed imminently in order to pace development, and if so what degree of intervention. This was true before we learned of an internal OpenAI model hacking HuggingFace, which should update your perspective. It remains true now.
Whereas I believe the letter’s statement would ideally be uncontroversial. We do not know what the future will bring. In many of those futures, we will need the ability to steer outcomes, including via pacing development. This will require cooperation, between companies and ultimately between governments. If we do not lay the groundwork in advance, it will be too late, and we either will be unable to coordinate or will have to do so via much worse mechanisms.
Who Signed The Letter
OpenAI endorses this letter.
Key figures from OpenAI that have signed include Chief Scientist Jakub Pachocki, Chief Research Officer Mark Chen, Cofounder and Head of AI Resilience at the OpenAI Foundation Wojciech Zaremba, Roon, Head of Strategic Futures Dean Ball, Head of Safety Saachi Jain, and former Head of Mission Alignment Joshua Achiam.
Anthropic endorses the letter.
Key figures from Anthropic that have signed include CEO Dario Amodei, co-founder Jack Clark, co-founder and Chief Science Officer Jared Kaplan, Co-Founder Benjamin Mann and Co-Founder, Interpretability Research Lead Chris Olah, Jan Leike, Ethan Perez and Claude Code creator Boris Cherney.
Key figures from Google DeepMind that have signed include Chief Strategy Officer Jasjeet Sekhon and VP of AI Safety & Alignment Anca Dragan.
Other key figures include Meta Chief Scientist Shengjia Zhao, Meta VP of AI Research Dawn Song, Mta Director of Alignment and Risk Summer Yue, Thinking Machines Chief Scientist John Schulman and Inherent Chief Scientist Edward Hughes.
A little under half of signatures are from Anthropic as of shortly after the announcement (as of writing this there are now 90 more signatures), with a lot of signatures from OpenAI and DeepMind, and a substantial number from Meta. The only one conspicuously missing here is SpaceX (xAI), or what is left of it.
Or by percentage of employee pool as of the evening of the 28th, notice that OpenAI now has more employees than DeepMind does:
Efforts seem to have concentrated on the higher end of the employee pools. A lot more than 4% of the biggest names signed.
We Need To Prepare Now So We Have The Option To Do This
I have been emphasizing this distinction for a while.
Do I think we should deliberately slow or even pause the pace of AI development at this time, given the tools we currently have available to do that?
Until the hack of HuggingFace, I would have said:
After the hack of HuggingFace and other recent disclosures, it looks like OpenAI in particular may need to halt and catch fire to solve some severe misalignment problems, as Sam Altman said in a recent podcast they have at least partially done, and this reinforces the need for Anthropic and others to check if they need to do likewise. We are at or nearing a critical point.
This makes laying the groundwork for well-executed coordinated action that much more important and urgent. We are running out of time. Capabilities progress is accelerating.
We won’t be able to do the good version of this without solving a lot of very hard problems. The more work we do in advance, the better our option set becomes. This includes diplomatic work, technical work, governance work, the building of state capacity, increases in transparency, developing good corporate infrastructures and procedures, and also coordination work.
Words From Some Of Those Who Signed
I turn the floor over to statements from some of the signatories. There are many more good quotes available on the letter’s website.
Words From Others
Daniel Kokotajlo of AI 2027 and Plan A updates his distribution of possible futures, and is substantially more optimistic that we will choose wisely. As with my adjustments, presume this is a mix of the letter having impact and learning that the letter was possible and that someone went ahead and created it.
How likely you think we are to land on D vs. C depends a lot on where you draw the boundary of ‘at least a little bit.’ Like most trade-offs, I do not expect us to be maximally reckless, but I also do not expect us to often be willing to fully ‘burn the lead.’
This is indeed a ‘middle path’ proposal, to give ourselves the option to do anything meaningful at all, and be able to collectively steer the future at all:
As always, there are many who cry ‘international tyranny’ at the thought of any form of even laying the groundwork for a deal or cooperation or collective action whatsoever, no matter its contents.
Or that will write off any such efforts as motivated by profits or delusion, or tries to equate this as ‘join forces with the anti-datacenter crowd.’
Consider your opinions noted, good sirs. We get letters.
I also threw out a reaction thread, for those who want a vibe check.
A Good Start
I believe the letter is on the Pareto frontier of ‘how much you say and how explicitly you say it’ versus ‘how many people and important people and labs sign the letter.’
It remains true that the letter is soft-pedaling compared to the actual situation as I see it, and there are many who would go farther than I would.
For those who wish it, here is a good presentation of the ‘need to do better’ perspective. This analysis sounds harsh. By Nate’s standards, this is still a positive review if graded on a curve, it’s just that he is in the ‘reality does not grade on a curve’ camp.
I did find this to be the most soft-peddling sentence. I get why, but yes I was sad to see this not be stronger.
I think this reaction from Nate is too harsh. Directionally he is correct, and I would have liked to see ‘potential upside.’ These are the types of battles that get fought behind the scenes with such letters.
Yes, this is maximally ambiguous on purpose. As I’ve said above I think that is correct.
Security measures and oversight help on the margin. I view this part less cynically than Nate, and do not think it is meant to imply that those measures are sufficient, only necessary. Pointing out that security measures are necessary is part of how you get people to agree to take overdetermined actions, and then worry about the sufficiency half later. In many cases, you will need to show their insufficiency later.
I strongly agree with Nate’s version and agree that you damn well should be able to say this if you include the word ‘may.’ Saying ‘stop’ would have lost a lot of signatories, though, and I support the final decision made, despite being sad about that.
This is not primarily ass-covering, this is an attempt to pitch to the White House by hitching the continued action to existing policy.
I think ‘pace’ was a very strong move as noted earlier. Good word.
What The Letter Does Not Say
As Tim Fist says, the letter does not specify what from the interventions would take, nor does it provide conclusive evidence of the risks and pace of development that would be convincing to lawmakers.
It would have been a serious mistake for the letter itself to try and include either of these things, but yes it does lack those elements.
That leaves us to figure out what interventions to prepare for and consider. As Tim Fist says, we should focus on frontier development and potential serious and irreversible harms, and we should try to find ways to avoid disadvantaging those labs that take safety seriously.
We should also take care to avoid interference with diffusion, and things like autonomous vehicles and drug discovery.
Tim Fist suggests we start with building state capacity and transparency and ability to monitor the situation, and with better enforcement of chip export controls. I agree strongly on both of these points, but would caution against doing only those things. In particular, diplomatic work needs time to play out.
How would we go about laying these foundations for future action? What might we want to build towards as options?
Maro Lourgon shares some of MIRI’s research in this area, as a first start.
There are many other sources out there from which to build, but in the interests of speed premium I will not stop to build a database here. If you want to help out, many would be happy to help point you in good directions.
What Happens Now?
To lab employees in particular, if you have not yet signed, consider signing.
Hopefully we continue to have our voices heard, the employees of the labs continue to coordinate, and we get to work laying the foundations of coordination in case we need them. Humanity has the opportunity to maintain any ability to steer the future at all, and thus maintain hope for a good outcome.
The alternative, if the lab employees are right about the future pace of development and superintelligence does arrive, is a world in which humanity cannot collectively steer its future, inevitably loses control, and likely soon perishes along with all that we value.
We are in a narrow window. AI employees will likely lose their leverage over time. If you want to influence the labs, and through them everyone else, you need to continue to organize and speak up now, before it is too late.
Lots of things have to be put into place, in the government, in diplomacy, in the labs, and in technical research and hardware development. Let’s get to work.