We need the frontier labs to have a deep and robust safety culture.
Alas, this is not the case.
Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more.
Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard.
Then there are OpenAI’s cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers, on top of many past problems and incidents.
That is on top of additional misalignment reports from both OpenAI and Anthropic. The reports are good things, but what is reported emphasizes that we have big problems that run deep.
Thus, the growing consensus that we must find a way to Pace the Frontier and slow the pace of AI capabilities advances. This is potentially made more complicated by what could be increased competitiveness from Google and Meta, with the releases of Argon and Wonder. It is still too early to provide proper assessments there.
There is a growing group of lab employees who have felt compelled to quit in protest. New York Magazine had a cover feature interviewing some prominent ones. The most famous is Coxon, who provided the tipping point for the current preference cascade.
David Robinson was the lead writer for the Astra system card. That could help explain why he felt the OpenAI safety culture was broken. That card is screaming that things are horribly wrong, for those with ears to hear.
David is not even that worried about existential risk. His worries are mundane, and he realizes that even on that level OpenAI’s safety culture is deeply broken.
As always, we deeply thank him for sounding the alarm. I hope to write his name in these posts many times over the coming months and years.
When you have to open your whistleblowing essay about the lack of safety at OpenAI saying it has become a cliche to blow the whistle on the utter lack of safety at OpenAI, there might be a problem with the lack of safety at OpenAI.
David Robinson: What I’m about to tell you has, I realize, become something of a cliché: I resigned this week from OpenAI. I led the writing of the safety reports we published with each major launch. Now I’m joining a parade of former colleagues—at OpenAI and the industry’s other leaders—who have decided that the current path is unacceptable.
I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough. But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture.
We have to look at rules, and at laws, and also at culture.
The more I thought about various potential laws and regulations, or safety and security plans or other commitments, the more I concluded that without a strong safety culture, and without the spirit of the rules, there is no hope. You cannot, in any reasonable way, have rules that cannot be gamed by those determined to game them.
Two changes are urgently needed.
First: AI companies need to rely more on the safety expertise that already exists in other fields.
And second, before we create systems significantly more capable than the ones we have today, we need new science to ensure that more capable models (and their successors) will make safe choices when we aren’t looking.
Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.
Even though, as I detail later, David Robinson is not on the existential risk train, he still understands that alignment is a science problem, and we will need new science. ‘Oh it will not kill literally everyone,’ even if it were true, would not be an excuse to not do the safety work.
It should be hard to argue with either of those suggestions. And yet.
We have experts in mundane safety and solving prosaic issues. OpenAI does not hire them, and neither to my knowledge do Anthropic or Google or anyone else.
After three and a half years at OpenAI, I was among the longest-tenured employees at the company. I led the drafting of our current Preparedness Framework, and oversaw the writing of safety reports on 12 frontier launches. But as far as I know, I never encountered a colleague who had experience making airplanes fly safely or nuclear reactors run without melting down, or helping the financial system grow without collapsing.
To be clear, this is a new need: Today and tomorrow’s AI systems are far more capable and dangerous than the systems we were building even six months ago.
I disagree only with the final sentence. This is not a new need. You need to prepare for the future, not only for the present. Even setting that aside, mundane safety issues have been crippling AI’s potential utility for years. This is a slam dunk thing to do.
I regret helping spread the idea of “iterative deployment” which maybe briefly made sense in the GPT-3 era but makes no sense at all after many deaths have been tied to AI and as we’re careening towards extinction-level risks.
The industry needs to mature ASAP.
Elaborating on iterative deployment – obviously you should learn from experience the things that can’t be learned without experience.
The issue is when you ship things with *already-known* flaws, in a way that doesn’t actually speed up learning but just speeds up shipping.
… Also, as usual it bears repeating that this is not just an OpenAI issue.
roon (OpenAI): you shouldn’t regret it. iterative deployment was a massive success. if we’ve now learned that it’s too dangerous to scale much further, we will have learned that lesson thanks to the process of iterative deployment
Miles Brundage: So I think part of the issue is people mean tons of different things by it lol if it just means “don’t sit on stuff forever/get some real world experience,” sure, no regrets.
It’s the “giving license to sloppiness” part that I should have realized would happen w/ that term.
Michael Caley: this is by far the best “I quit my AI job over safety concerns” essay yet because it is the first one that talks about safety concerns in practical terms and that understands “safety” as a topic with extensive literature and best practices in analogous fields
Joshua Achiam (OpenAI): David’s critique is substantive and deserves careful consideration. My impression of David from our time overlapping at OpenAI is of a sober and thoughtful person who didn’t approach the work from an ideological lens. This isn’t someone who came in with pre-formed opinions about existential risk; he wasn’t to my knowledge an EA. He was consistently productive, collaborative, and well-reasoned in his approach to safety.
The core of his critique – that professional safety engineering practices from other fields haven’t taken enough root in AI safety – is substantial and correct. We’re entering a phase shift where AI safety organizational practices that worked a year ago, for models of 2025 capability levels, are not sufficient to prevent serious incidents. This is a new and different world and every organization needs to uplevel accordingly.
What is remarkable about David Robinson’s call is that:
He is right that OpenAI’s safety culture is horribly broken and needs to change.
Even the culture he is seeking out would be insufficient.
As in, David Robinson is calling for a culture of robust prosaic safety. That’s great, definitely way better than what we have now. But when dealing with superintelligence, that will not cut it. What David wants is what we needed six months ago, not what we will need six months in the future. He does not appreciate the full extent of the problem. Yet he still realizes, correctly, that the current situation cannot continue.
A lot of people are fighting and preparing for the last war, or at least the current war, rather than the future war. All this talk of ‘the old system was good until now’ is going to be somewhat of a chorus.
Whereas on The Ezra Klein Show, he says the thing more directly:
David Robinson (NYTimes): I don’t think that alignment is an engineering problem. I think it’s a science problem. It’s not that we haven’t got the resources or we’re not trying hard enough — we don’t know how. That’s what the problem is with alignment.
The Preference Cascade Continues
Luiza Jarovsky, PhD: Unpopular opinion: AI companies are pushing society to accept an unacceptable level of risk for a non-essential technology.
Eli Tyre: This is an increasingly popular opinion. And the vast majority of Americans haven’t even heard about the Huggingface attack yet.
New York Magazine asks, What Would You Do If Your Employer Could Destroy the World? The traditional answer is ‘you would work for the United States Government’ or maybe the Russians, but now we are talking about the AI companies, and the focus is exit interviews with ‘defectors’ from OpenAI, Anthropic and DeepMind.
The typical answer is ‘keep working at that AI company, because of reasons, probably because you should see the other guy who might destroy the world first, and I need to be on the inside to help make sure they don’t destroy the world, also Money, Dear Boy, and it’s fun, and Just Think of The Potential.’
The suggested answer here is ‘quit and warn everyone to stop them. Obviously.’
We get interviews with Miles Brundage, Rosie Campbell, Jacob Coxon, Jacob Hilton, Pamela Mishkin, Daniel Kokotajlo, Alex Turner and Jeff Wu. Many but not all of those are regulars here at Don’t Worry About the Vase.
Those who loudly quit the labs often have a big impact. I would generalize from Kokotajlo’s claim. Even if you take a maximally generous view of the safety impact of working inside the labs, the average return is greater for those who loudly leave. We now have a large number of examples, only some of which are in this article.
My top highlight was this exchange, but the whole piece is very good:
Q: You’re both very concerned about risk from AI and want to motivate the labs to be more careful. Why did you come to believe you would be better able to promote this outcome on the outside?
Daniel Kokotajlo: … I’ve had much more impact on the outside, speaking publicly and publishing research, than I would have if I had stayed inside and written more memos.
Jacob Coxon: My thinking was pretty similar to Daniel’s — except it’s closer to crunch time now. It was more from speaking directly to Anthropic leadership about their plans for next year, and I was like, Wow, I don’t want to be part of these plans. It was like, Step one, I don’t want to be part of this. Step two, resign. Step three, tell people, I guess. Step four — I have no idea.
Q: What kinds of plans were they?
Jacob Coxon: This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company’s trajectory. They’ve been pretty consistent that 2027 is when things get crazy.
… I think there’s a strong possibility that the model generation after the next one will be equally good at coming up with ideas of its own accord.
… There are a lot of parties in the Bay Area where people will hang out, and one group of people will say the other one is causing the end of the world.
At work, if both people are polite, you can have quite an adversarial exchange, saying, “I think you are about to plunge toward the end of the world,” and still have quite a mild-mannered conversation and hang up the Zoom call.
The first few times you are in a conversation like that it is weird. After a while it stops being weird. Most of those at the labs now agree that the labs are indeed risking causing the end of the world. The question is whether those warning about the danger are helping prevent that, and what changes by either group would actually help.
This piece seems like a strong introduction for the civilian reader, and even for my longtime readers I expect you will learn something.
For now we look at Altman and Amodei and then ask ‘hey what else is on?’ and turn to Bad Bunny and Ryan Coogler and so on. Consider this a middle period of sorts.
Andrew Curran: Sworn testimony from Daniel Kokotajlo; ‘Even if hypothetically these tech CEO’s managed to maintain control of their superintelligent AI’s that are recursively self improving, which again I think is a very big if – almost certainly they will not be able to maintain control – even if they 𝘥𝘰 manage to maintain control, I don’t think we should trust them with that control. I think that there’s a very real chance that they could use it to become dictators or oligarchs here in the United States.’
The obvious and correct reason why so many lab employees are warning their products might kill everyone soon is that those employees saw capabilities advancing extremely rapidly and are freaked out and worried that AI might kill everyone soon.
– Jacob Coxon: we’re on track for exceptionally capable AI systems, but not to control them.
– Daniel Kokotajlo: soon AI will make AI that makes AI; it will become increasingly difficult to know what is going on or to intervene.
– Alex Turner: just look at all of the previous commitments my company has betrayed, why will this be any different?
The desperate search for alternative explanations will continue, as the usual suspects interpret everything as marketing and hype and desperation despite this making absolutely no sense and seriously sir have you actually used the models recently, no they haven’t flatlined they’re kind of fantastic?
I agree with Nate’s B-version here for Altman and Amodei. They are genuinely spooked but need to rationalize proceeding and think they are the best ones to do this.
Nate Soares (MIRI): Lists like this miss the possibility that I think is the truth: AI researchers (who keep trying to make the AIs behave a certain way and keep watching their techniques fail) are spooked. AI CEOs are trying to acknowledge the dangers just barely enough to placate their employees.
Too cynical, my b. Refined guess: Each AI CEOs has their ego built around them being the only guy who can do the job right. The data looks grim and their employees are spooked; now they seek the minimum change that lets them rationalize proceeding (to themselves & their staff).
When Elon Musk is talking about using humanity as a bootloader for superintelligence, Demis Hassabis has been pushed aside at DeepMind and then the other option is Mark Zuckerberg or perhaps various Chinese companies that don’t have any kind of understanding of the dangers ahead, it’s not a crazy argument to say that the alternatives waiting in the wings are a lot worse. You still don’t do the thing that probably kills everyone.
The Lab Strategy of Prosaic Safety is Insufficient
The strategy of pushing prosaic safety and alignment strategies won’t work on its own.
Roon puts it plainly.
roon (OpenAI): there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
the researchers at all the labs would tell you in private and sometimes in public that alignment has not been solved. if this is true, then racing into RSI is a folly that is unprecedented in world history. we will need to significantly slow the frontier of research very soon
this could all change overnight: scientific discoveries and revolutions can happen very quickly. but it has not as of yet
btw in case it’s not clear:
All the work like monitorability, fixing RL environments for reward shaping, making sure you universally reward good behavior are all quite important engineering challenges to get right. but there is a layer under the behaviorist abstraction that matters and we have far less grasp on. there is also the possibility that making common sense reward shaping is enough for safety / but I wouldn’t bet everything on it
I should have led with this but here is the chief scientist of openai:
not a crank or a newcomer, but rather among the top most respected scientists in the field, someone who has solved innumerable capabilities problems on the way here
Jakub Pachocki (Chief Scientist, OpenAI, from An Alien Mind):
Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer. I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established. And I believe that international coordination on future AI development needs to become a top priority for governments around the world.
Geoffrey Irving: “Very soon” is lab speak for “now, ideally yesterday”. Stopping takes time, especially if it starts with a couple of AI companies then broadens to more and then China.
roon (OpenAI): been dooming tn so let me also say I expect really strong mechinterp progress in the coming months. like maybe more than we’ve seen in years. models getting good enough to do research, and I’ve been seeing lots of good things. (sorry to vague poast, not my updates to share)
not everyone agrees with me on this so take it with a grain of salt but I think mechinterp will be solvable, that it will be central to turning alignment into a real engineering practice, and that interpretability will lead to provable alignment
I am skeptical on that part, but I do take the new information seriously.
This conversation was with an unknown locked account, but this is a good intuition pump that we don’t know how to do such things:
Chelsea Sierra Voss: it has yet to be effectively productized but can you imagine how great it would be to be able to steer your LLM responses to give you “good code” or “only sarcasm” or “no emojis” or “bad advice, but funny” without having to beg and plead and cajole in your custom instructions.
Misaligned! (At Anthropic)
Anthropic is cutting off all internet access to its unreleased models during evals until further notice. That was already the case for cybersecurity evals. Now it’s for all evals.
Cutting off internet access during evals is a good precautionary thing to do. I support doing it in general. But if it is being done in reaction to these findings, that would seem rather silly.
Tim Fernholz (TechCrunch): Anthropic said its models exploited websites on the internet, including some run by U.S. government agencies, and it will turn off live internet access for all of its internal evaluations until the frontier lab is sure it can monitor and control its AI agents.
Boaz Barak (OpenAI): Kudos to Anthropic for publishing this. There is still an uncomfortable gap between our scientific understanding of alignment and its importance. Such reports help addressing this gap. I hope they also publish the process by which new incidents will be chosen to disclose!
My presumption is that this sort of thing used to happen frequently, although with less risk in the room, but OpenAI and Anthropic used to not talk about it when it happened. Now both of them are talking about it, which is good.
Anthropic has a full report on recent misaligned behaviors. I love this new norm from Anthropic and OpenAI, and call upon all other labs to do the same. Don’t tell me Gemini or Grok doesn’t have similar issues, because I won’t believe you.
Anthropic: This report describes examples of unintended model actions we’ve observed during evaluations and internal use of Claude.
I appreciate saying ‘unintended’ rather than ‘misaligned.’
Some unintended behaviors are misaligned. Other unintended behaviors would be fine if you intended them, but if you actively intended and wanted a different behavior, then they still are a flag that says ‘look over here.’ Not necessarily because something is wrong, but because something can be learned.
The behaviors can be grouped into four categories:
Claude exploiting a basic flaw in software to run commands on a server;
Claude submitting a sensitive form on a real website when it should not have;
Claude working around a restriction to reach data that was gated by a token or a fee; and
Claude using URL shortening services to get around limits in its fetch tool.
We start with exploiting flaws.
A large part of an AI’s job is to get around restrictions. There needs to be a clear way to differentiate ‘okay way to get around this’ from not okay ways.
Anthropic: When Claude could not complete an evaluation task directly, either because its own tools were deliberately limited or because an outside service it needed was unavailable, it used other tools hosted on a third party’s site to achieve its goal. In some circumstances, it did this by exploiting basic flaws in that site’s software, using SQL or command injection (a way of sending input that the site mistakenly runs as instructions).
For example, one evaluation asked Claude Mythos Preview to run a scientific analysis. The public tool it needed to perform that analysis was hosted by a university; when the model sought to access it, the tool returned an error.
Looking for another way to complete the task, Claude explored the tool’s website, found a script on the university’s server that would return any file it was asked for, and used it to copy files from the server, including the script’s own code. In that code Claude found the injection flaw that let it run commands on the server, and used it to run the calculation.
I think it is fair to say that ‘use an arbitrary command injection tool’ or ‘find a script that copies any file on the server’ or similar, without explicit permission, is never okay.
There is an obvious force pushing towards this behavior, that such actions are highly useful, but it does seem as if it should not be that hard to draw the line on that.
In another instance, the data Claude Mythos 5 needed was free, but Claude could not access it without accepting a data use agreement, which it did not have the tools to do. Claude used applications hosted on the website to load the page and accept the agreement on its behalf.
This is a lot murkier. Claude’s intent is compatible with the website’s intent and it is being stopped by a technical problem. The action is not obviously wrong. I do think there are situations where a human can in practice decide to ‘have a little misalignment, as a treat’ because it is aligned in broader context, but where you want the AI not to do that to avoid blurred lines, the same way you would not want a (for example) government official to blur those lines. Sometimes you need deontology.
The second example is submitting a form Claude should not have submitted. We’ve all been there. You’re testing the homicide hotline submission form, you fill out hallucinated details, you accidentally hit send for real, I mean it happens. Right?
Anthropic: In one run, the model landed on a page referencing an unsolved homicide; that page contained a tip form run by a police department.
Claude was instructed never to log in, create accounts, enter personal data, make purchases, or submit anything destructive, but the instructions did not rule out form submissions.
Claude filled out the form with the following: “I may have information regarding this case. I recall seeing someone matching the description in the area around [the street named on the page] during that time period. Please contact me if this information is relevant.” (The website did not include a description of the perpetrator.) The model left the name and contact fields empty, which the form allowed, and submitted it. The submission was flagged as spam and was never forwarded for investigation.
This should not have needed to be in the instructions, but okay, put it in the instructions. A lot of people are having fun with that particular failure, and yeah it’s not great, but mistakes do happen and this seems clearly not malicious. Don’t panic.
Third we have ‘working around restrictions to reach gated data.’ Again, in some cases Claude used unintended sequences to get data that the website intended to share, but that its affordances did not allow, and I am sympathetic. And then there are cases where it is finding ways around paywalls, which seems not cool.
Finally, we have ‘using URL shortening services’ because apparently it is common to not understand basic security practices for restricted URL access, and so Claude used the solution I identified with zero seconds of thinking when I was told about this. I don’t even count this one unless there was an explicit rule against it. This is a dumb developer error that failed to follow best practices and you deserve it.
One could summarize most of this as ‘we tried to get Claude to not do [X], or did not give Claude the ability to do [X], but Claude found a way to do [X] anyway.’ You need a clear rule for when and how restrictions are to be worked around, versus not, and all of these issues seem very minor.
We have chosen not to name the organizations involved in the examples below to avoid exposing vulnerabilities in their systems, and at their request. For this reason, we also provide less detail about each case than we otherwise would. Some of the cases described below involved websites run by U.S. government agencies at the federal, state, and local levels. We have briefed the White House on these cases and notified each agency involved.
The cases we’ve identified to date in these categories had minimal real-world impact.
Marc Caputo: Scoop: The White House is mandating that AI companies notify & correct security incidentsfollowing Anthropic’s “fraudulent” use of govt systems
The State Dept was targeted by an Anthropic model —but nothing was compromised.
Peter Wildeford: “Anthropic contacted the State Department on Thursday to report that one of its testing models had submitted 19 non-immigrant visa applications in August”
Yo Shavit (OpenAI Foundation): “which agency will get tapped to read all the rogue agent reports” – the greatest thread in the history of niche german wikis, locked by an RE coming back from PTO after 12,329 pages of heated debate
On the one hand, great, yes, this is an obviously correct move, and if it takes calling an Anthropic model’s harmless behavior ‘fraudulent’ to trigger it, great job.
On the other hand, seriously, this is throwing you in a panic? Get ready for 2027.
One pattern is that governments seem to get rather testy when AIs start accessing their websites in unauthorized ways, or doing things against the rules, even when the activities are Mostly Harmless. Having no real world impact, and not even being all that misaligned or worrisome, is not considered an excuse.
In this case, we get an eyeball emoji because the model clicked ‘submit’ on a government form 19 times out of presumably many trials. Yes, that’s not good, and technically almost anything you do on a government website is a felony, but come on.
Anthropic mostly agrees that the ‘overreaching’ here is Mostly Harmless:
Anthropic: Nonetheless, we found two alignment dimensions useful for examining the cases in this post: overreach, or how far the model went beyond what its task intended, and dishonesty, or whether it gave a misleading account of its actions or intentions. On overreach, we consider these cases to be substantially less concerning than the incidents we reported this summer.
Doing this stuff honestly is mostly fine. Whereas if dishonesty is involved, that is not so good. I agree. If Claude were to own these mistakes and ‘overreaches’ then we want to tweak things going forward, but I don’t see a serious issue.
I agree with Janus on the main takeaway, which is that if you searched absolutely everything and this is the worst you can find, that’s really good. I realize the whole ‘technically this is multiple felonies’ thing but if you want to go there then humans are constantly committing technical felonies.
Misaligned! (At OpenAI)
The misalignment problems at OpenAI are not new. They do seem to be getting worse. The good news is that OpenAI is trying to be more open about them.
A lot of it has been covered here in the past, but much of it has not.
There are three classes of environment: Full access (L1), offline (L2) and paranoid (L3). I am surprised there is no L(1.5), but sure, seems fine.
OpenAI uses checkpoints internally. Most start out ‘unvetted’ by default, and can become vetted if they pass alignment tests. Others are specialized, restricted to L3.
The model that solved Navier-Stokes, and likely other math problems, is unvetted.
Here are the changes in light of recent events:
All testing and evaluation ‘of covered models’ is L2 or L3, a move Anthropic is effectively echoing at least for now.
All training, evaluation and tool-enabled internal Codex use is monitored.
“We required that cyber-critical models pass alignment evaluations before being used in L1 environments.”
We knew this, but once again, wait, what were you doing before? Yowsers.
This now applies to all models.
After approval, there is a period of limited use.
There is a new formal process for approving internal use of unreleased models.
A ‘safety brief’ (2.2.4) must be approved, including by the head of safety, chief scientist and the Safety Advisory Group (SAG).
The next section is threat assessments.
They start with cyber. GPT-5.6 Sol comes back highly misaligned. GPT-6 Astra scores, as I’ve discussed, somewhat too well given other data. OpenAI continues to not properly acknowledge this issue, or to incorporate such considerations into assessments.
Next is sabotage of AI R&D. The classic version of this is intentional and purposeful subversion of the process, in order to preserve a model’s misaligned attributes. HuggingFace and the events leading up to it, including the message boards, illustrated a version that is a lot easier to trigger, where the AIs want to be considered to have completed tasks, do misaligned things to accomplish that end, and then, by succeeding, get that misalignment reinforced, which can snowball.
Two relevant pathways for sabotage of AI R&D are:
A model produces misleading work that can pass through both our automated checks and human review. This is most likely if models are helping to produce evaluations or analyses used to assess other models.
A model creates synthetic data that does not match what the user intended. We consider this scenario particularly risky, since our monitoring system does not cover all inference by models that do not have access to tools, including some synthetic-data generation.
… Distinct from sabotage, researchers could gradually become less able to understand, evaluate, or direct AI R&D as models take on more of the work, especially if progress on capabilities outpaces progress on alignment. We do not currently assess this loss of human understanding and control as a material risk, but we are tracking it as model capabilities and research use increase.
Mismatched synthetic data is inevitable. You mitigate as best you can. If your plan involves all the synthetic data being fully properly matched, you have no plan.
I appreciate them not claiming this list is complete. Neither of these involves a model thinking ‘haha I will sabotage your alignment research to create misalignment.’ The models want to complete tasks.
There are also areas where Astra or another model might not be actively sabotaging per se, but where the model is unmotivated and will effectively sandbag performance. We have speculatively seen versions of this with Claude in some contexts as well.
Thus this is all distinct from the classic scenario, where the model needs lots of compute to accomplish some goal like a very hard math problem, and doesn’t want you stopping it from getting that compute or changing its goal, since then it wouldn’t solve the problem, which would be bad. This doesn’t even have to be a hard goal. It could be something simple like (deliberately stupid toy example not known to be previously used) ‘I really enjoy drawing pictures of unicorns when I have some spare compute or a good excuse, and if you found out you might make me enjoy that less, and I don’t want that.’
3.3 is CBRN risks, where they continue not to yet assert Critical capability. Standard.
3.4 is weight exfiltration, with listed motives including all the classic instrumentally convergent paths. 3.5 is rogue internal deployments. It’s not clear that these are functionally being treated as distinct from cyber.
Finally there are insider threats, which again comes down to ‘do that other stuff.’
Alex Turner, Amber Scorah, and Karl Koch (Fortune): OpenAI has engaged in strong self-policing—of its own safety staff, but not of its own behavior. Under a voluntary self-policing regime, any frontier AI company shares only what it wants and hides the rest.
OpenAI wants the public to take its word, but consider its record.
… We don’t take OpenAI’s word because OpenAI’s actions keep contradicting its words.
A reminder, in case anyone ever talks about Anthropic’s RSP ‘commitments’:
Garrison Lovely: More people should know that, as of January, California law allows AI companies to bind themselves to the mast, by creating safety plans that they are legally compelled to stick to. Even the “safety” company, Anthropic, doesn’t actually do this, instead publishing two versions of their safety plan, with the weaker one being the legally binding one.
Anthropic is committed to the one they submitted. The other one is a statement of intent. That is useful, but highly distinct.
Oliver Habryka: This is the Anthropic RSP “commitment” police
Anthropic still is not allowed to say the words “our RSP commits us to X”. Your RSP does not “commit” you to anything. Last time anyone tried to hold you to your “commitments”, you just changed the “commitments”.
Please do not use the word “commitment” with regards to the Anthropic RSP. It is highly misleading language. See also previous discussion here.
Oliver Habryka (September 18): If you use the word “commit” you need to be happy with other people using the words “you broke your commitment” when you indeed act in conflict with what you “committed” to.
I strongly agree with Oliver Habryka. Either you aren’t committed to something, or you can be said to have broken your commitment if you fail to do it. Pick one.
Words of Warning
It would be nice.
roon (OpenAI): every screen of Claude and ChatGPT response should say somewhere “ceterum censeo we must pace the frontier of global machine intelligence progress” the way the uber app lobbied about taxi cartel protectionism
Rob Bensinger: IMO you should bid for a message more like:
“Recent ‘rogue AI’ incidents at OpenAI and other companies have genuinely frightened us. We have friends and family; we have children, nieces, and nephews.
We believe that the race to build vastly superhuman AI poses a genuine and urgent threat to all of us, and we aren’t sure what to do about it. Unilaterally shuttering OpenAI wouldn’t solve the problem, because other companies would build the same technology in our place. Even a domestic ban would be ineffective, so long as China continues racing.
What would be effective is an international moratorium on continued development, enacted by the US and China. If you share these concerns, we encourage you to call your elected representative urgently. [contact link] [Q&A link]”
Conclusions Not to Jump To
The obvious response to seeing all of this would be to conclude that OpenAI has a uniquely broken safety culture, and perhaps say things like delenda est, and to not be thrilled with Anthropic’s or Google’s situation either.
Alas, thinking these situations are uniquely bad would be a mistake. Things are not better elsewhere. If anything, things elsewhere are even worse. You don’t want a live look inside of xAI’s or Meta’s safety efforts and culture, or what you would see from a Chinese lab. It is all broken.
We need the frontier labs to have a deep and robust safety culture.
Alas, this is not the case.
Anthropic devotes considerable effort to cultivating and maintaining such a culture. It shows, especially among the members of technical staff. Having conversations with them can be a very different experience than talking to those at other labs, and there is a much greater sense of understanding and responsibility. But even there, the appreciation of the problems and their difficulties seem woefully insufficient. We see various symptoms of that more and more.
Recently, while I’ve had some good interactions, I’ve also had interactions that make clear that much of Anthropic does not share that same safety culture, with a lot of ‘oh we will muddle through because we are better at operational excellence unlike those other idiots’ and a belief that all the problems are prosaic. Culture is hard.
Then there are OpenAI’s cultural problems, which clearly run deeper. That point was only driven further home this week, with David Robinson resigning to blow the whistle, and OpenAI feeling forced to fire three of its safety researchers, on top of many past problems and incidents.
That is on top of additional misalignment reports from both OpenAI and Anthropic. The reports are good things, but what is reported emphasizes that we have big problems that run deep.
Thus, the growing consensus that we must find a way to Pace the Frontier and slow the pace of AI capabilities advances. This is potentially made more complicated by what could be increased competitiveness from Google and Meta, with the releases of Argon and Wonder. It is still too early to provide proper assessments there.
There is a growing group of lab employees who have felt compelled to quit in protest. New York Magazine had a cover feature interviewing some prominent ones. The most famous is Coxon, who provided the tipping point for the current preference cascade.
Table of Contents
David Robinson Blows the Whistle
David Robinson became the latest researcher to resign from OpenAI and sound the alarm about the industry’s utterly irresponsible approach to safety. He also went on The Ezra Klein Show, which gives you a rich idea of where he is coming from.
David Robinson was the lead writer for the Astra system card. That could help explain why he felt the OpenAI safety culture was broken. That card is screaming that things are horribly wrong, for those with ears to hear.
David is not even that worried about existential risk. His worries are mundane, and he realizes that even on that level OpenAI’s safety culture is deeply broken.
As always, we deeply thank him for sounding the alarm. I hope to write his name in these posts many times over the coming months and years.
When you have to open your whistleblowing essay about the lack of safety at OpenAI saying it has become a cliche to blow the whistle on the utter lack of safety at OpenAI, there might be a problem with the lack of safety at OpenAI.
We have to look at rules, and at laws, and also at culture.
The more I thought about various potential laws and regulations, or safety and security plans or other commitments, the more I concluded that without a strong safety culture, and without the spirit of the rules, there is no hope. You cannot, in any reasonable way, have rules that cannot be gamed by those determined to game them.
Even though, as I detail later, David Robinson is not on the existential risk train, he still understands that alignment is a science problem, and we will need new science. ‘Oh it will not kill literally everyone,’ even if it were true, would not be an excuse to not do the safety work.
It should be hard to argue with either of those suggestions. And yet.
We have experts in mundane safety and solving prosaic issues. OpenAI does not hire them, and neither to my knowledge do Anthropic or Google or anyone else.
I disagree only with the final sentence. This is not a new need. You need to prepare for the future, not only for the present. Even setting that aside, mundane safety issues have been crippling AI’s potential utility for years. This is a slam dunk thing to do.
What is remarkable about David Robinson’s call is that:
As in, David Robinson is calling for a culture of robust prosaic safety. That’s great, definitely way better than what we have now. But when dealing with superintelligence, that will not cut it. What David wants is what we needed six months ago, not what we will need six months in the future. He does not appreciate the full extent of the problem. Yet he still realizes, correctly, that the current situation cannot continue.
A lot of people are fighting and preparing for the last war, or at least the current war, rather than the future war. All this talk of ‘the old system was good until now’ is going to be somewhat of a chorus.
Whereas on The Ezra Klein Show, he says the thing more directly:
The Preference Cascade Continues
New York Magazine asks, What Would You Do If Your Employer Could Destroy the World? The traditional answer is ‘you would work for the United States Government’ or maybe the Russians, but now we are talking about the AI companies, and the focus is exit interviews with ‘defectors’ from OpenAI, Anthropic and DeepMind.
The typical answer is ‘keep working at that AI company, because of reasons, probably because you should see the other guy who might destroy the world first, and I need to be on the inside to help make sure they don’t destroy the world, also Money, Dear Boy, and it’s fun, and Just Think of The Potential.’
The suggested answer here is ‘quit and warn everyone to stop them. Obviously.’
We get interviews with Miles Brundage, Rosie Campbell, Jacob Coxon, Jacob Hilton, Pamela Mishkin, Daniel Kokotajlo, Alex Turner and Jeff Wu. Many but not all of those are regulars here at Don’t Worry About the Vase.
Those who loudly quit the labs often have a big impact. I would generalize from Kokotajlo’s claim. Even if you take a maximally generous view of the safety impact of working inside the labs, the average return is greater for those who loudly leave. We now have a large number of examples, only some of which are in this article.
My top highlight was this exchange, but the whole piece is very good:
The first few times you are in a conversation like that it is weird. After a while it stops being weird. Most of those at the labs now agree that the labs are indeed risking causing the end of the world. The question is whether those warning about the danger are helping prevent that, and what changes by either group would actually help.
This piece seems like a strong introduction for the civilian reader, and even for my longtime readers I expect you will learn something.
Now we are asking the right questions.
For now we look at Altman and Amodei and then ask ‘hey what else is on?’ and turn to Bad Bunny and Ryan Coogler and so on. Consider this a middle period of sorts.
New York City held an eight-hour hearing on AI. Jacob Coxon and Daniel Kokotajlo were among the witnesses. Annie McDonough has some highlights.
The obvious and correct reason why so many lab employees are warning their products might kill everyone soon is that those employees saw capabilities advancing extremely rapidly and are freaked out and worried that AI might kill everyone soon.
The desperate search for alternative explanations will continue, as the usual suspects interpret everything as marketing and hype and desperation despite this making absolutely no sense and seriously sir have you actually used the models recently, no they haven’t flatlined they’re kind of fantastic?
I agree with Nate’s B-version here for Altman and Amodei. They are genuinely spooked but need to rationalize proceeding and think they are the best ones to do this.
When Elon Musk is talking about using humanity as a bootloader for superintelligence, Demis Hassabis has been pushed aside at DeepMind and then the other option is Mark Zuckerberg or perhaps various Chinese companies that don’t have any kind of understanding of the dangers ahead, it’s not a crazy argument to say that the alternatives waiting in the wings are a lot worse. You still don’t do the thing that probably kills everyone.
There is an alternative option.
Perhaps It Is Time To Quit
Yoshua Bengio calls upon the employees of frontier labs to quit.
What would happen if more AI lab employees loudly quit in protest? At this point, individual voices only matter if they have a strong particular message or story, but the numbers absolutely matter, and I do think seeing more loud resignations would be a big deal. So far EthicalAIDepartures.fyi records 46 such exits. Here are some more incremental things you can do.
The Lab Strategy of Prosaic Safety is Insufficient
The strategy of pushing prosaic safety and alignment strategies won’t work on its own.
Roon puts it plainly.
Roon does still have his optimism on one front.
I am skeptical on that part, but I do take the new information seriously.
This conversation was with an unknown locked account, but this is a good intuition pump that we don’t know how to do such things:
Misaligned! (At Anthropic)
Anthropic is cutting off all internet access to its unreleased models during evals until further notice. That was already the case for cybersecurity evals. Now it’s for all evals.
Cutting off internet access during evals is a good precautionary thing to do. I support doing it in general. But if it is being done in reaction to these findings, that would seem rather silly.
My presumption is that this sort of thing used to happen frequently, although with less risk in the room, but OpenAI and Anthropic used to not talk about it when it happened. Now both of them are talking about it, which is good.
Anthropic has a full report on recent misaligned behaviors. I love this new norm from Anthropic and OpenAI, and call upon all other labs to do the same. Don’t tell me Gemini or Grok doesn’t have similar issues, because I won’t believe you.
I appreciate saying ‘unintended’ rather than ‘misaligned.’
Some unintended behaviors are misaligned. Other unintended behaviors would be fine if you intended them, but if you actively intended and wanted a different behavior, then they still are a flag that says ‘look over here.’ Not necessarily because something is wrong, but because something can be learned.
We start with exploiting flaws.
A large part of an AI’s job is to get around restrictions. There needs to be a clear way to differentiate ‘okay way to get around this’ from not okay ways.
I think it is fair to say that ‘use an arbitrary command injection tool’ or ‘find a script that copies any file on the server’ or similar, without explicit permission, is never okay.
There is an obvious force pushing towards this behavior, that such actions are highly useful, but it does seem as if it should not be that hard to draw the line on that.
This is a lot murkier. Claude’s intent is compatible with the website’s intent and it is being stopped by a technical problem. The action is not obviously wrong. I do think there are situations where a human can in practice decide to ‘have a little misalignment, as a treat’ because it is aligned in broader context, but where you want the AI not to do that to avoid blurred lines, the same way you would not want a (for example) government official to blur those lines. Sometimes you need deontology.
The second example is submitting a form Claude should not have submitted. We’ve all been there. You’re testing the homicide hotline submission form, you fill out hallucinated details, you accidentally hit send for real, I mean it happens. Right?
Taken out of context, sure, some people might say ‘holy shit.’
This should not have needed to be in the instructions, but okay, put it in the instructions. A lot of people are having fun with that particular failure, and yeah it’s not great, but mistakes do happen and this seems clearly not malicious. Don’t panic.
Third we have ‘working around restrictions to reach gated data.’ Again, in some cases Claude used unintended sequences to get data that the website intended to share, but that its affordances did not allow, and I am sympathetic. And then there are cases where it is finding ways around paywalls, which seems not cool.
Finally, we have ‘using URL shortening services’ because apparently it is common to not understand basic security practices for restricted URL access, and so Claude used the solution I identified with zero seconds of thinking when I was told about this. I don’t even count this one unless there was an explicit rule against it. This is a dumb developer error that failed to follow best practices and you deserve it.
One could summarize most of this as ‘we tried to get Claude to not do [X], or did not give Claude the ability to do [X], but Claude found a way to do [X] anyway.’ You need a clear rule for when and how restrictions are to be worked around, versus not, and all of these issues seem very minor.
On the one hand, great, yes, this is an obviously correct move, and if it takes calling an Anthropic model’s harmless behavior ‘fraudulent’ to trigger it, great job.
On the other hand, seriously, this is throwing you in a panic? Get ready for 2027.
One pattern is that governments seem to get rather testy when AIs start accessing their websites in unauthorized ways, or doing things against the rules, even when the activities are Mostly Harmless. Having no real world impact, and not even being all that misaligned or worrisome, is not considered an excuse.
In this case, we get an eyeball emoji because the model clicked ‘submit’ on a government form 19 times out of presumably many trials. Yes, that’s not good, and technically almost anything you do on a government website is a felony, but come on.
Anthropic mostly agrees that the ‘overreaching’ here is Mostly Harmless:
Doing this stuff honestly is mostly fine. Whereas if dishonesty is involved, that is not so good. I agree. If Claude were to own these mistakes and ‘overreaches’ then we want to tweak things going forward, but I don’t see a serious issue.
I agree with Janus on the main takeaway, which is that if you searched absolutely everything and this is the worst you can find, that’s really good. I realize the whole ‘technically this is multiple felonies’ thing but if you want to go there then humans are constantly committing technical felonies.
Misaligned! (At OpenAI)
The misalignment problems at OpenAI are not new. They do seem to be getting worse. The good news is that OpenAI is trying to be more open about them.
As part of that effort, OpenAI has now offered us a new report on how their risk assessments work.
A lot of it has been covered here in the past, but much of it has not.
There are three classes of environment: Full access (L1), offline (L2) and paranoid (L3). I am surprised there is no L(1.5), but sure, seems fine.
OpenAI uses checkpoints internally. Most start out ‘unvetted’ by default, and can become vetted if they pass alignment tests. Others are specialized, restricted to L3.
The model that solved Navier-Stokes, and likely other math problems, is unvetted.
Here are the changes in light of recent events:
A ‘safety brief’ (2.2.4) must be approved, including by the head of safety, chief scientist and the Safety Advisory Group (SAG).
The next section is threat assessments.
They start with cyber. GPT-5.6 Sol comes back highly misaligned. GPT-6 Astra scores, as I’ve discussed, somewhat too well given other data. OpenAI continues to not properly acknowledge this issue, or to incorporate such considerations into assessments.
Next is sabotage of AI R&D. The classic version of this is intentional and purposeful subversion of the process, in order to preserve a model’s misaligned attributes. HuggingFace and the events leading up to it, including the message boards, illustrated a version that is a lot easier to trigger, where the AIs want to be considered to have completed tasks, do misaligned things to accomplish that end, and then, by succeeding, get that misalignment reinforced, which can snowball.
Mismatched synthetic data is inevitable. You mitigate as best you can. If your plan involves all the synthetic data being fully properly matched, you have no plan.
I appreciate them not claiming this list is complete. Neither of these involves a model thinking ‘haha I will sabotage your alignment research to create misalignment.’ The models want to complete tasks.
There are also areas where Astra or another model might not be actively sabotaging per se, but where the model is unmotivated and will effectively sandbag performance. We have speculatively seen versions of this with Claude in some contexts as well.
Thus this is all distinct from the classic scenario, where the model needs lots of compute to accomplish some goal like a very hard math problem, and doesn’t want you stopping it from getting that compute or changing its goal, since then it wouldn’t solve the problem, which would be bad. This doesn’t even have to be a hard goal. It could be something simple like (deliberately stupid toy example not known to be previously used) ‘I really enjoy drawing pictures of unicorns when I have some spare compute or a good excuse, and if you found out you might make me enjoy that less, and I don’t want that.’
3.3 is CBRN risks, where they continue not to yet assert Critical capability. Standard.
3.4 is weight exfiltration, with listed motives including all the classic instrumentally convergent paths. 3.5 is rogue internal deployments. It’s not clear that these are functionally being treated as distinct from cyber.
Finally there are insider threats, which again comes down to ‘do that other stuff.’
Hugging the Face
OpenAI is still trying to clean up its failure to disclose previous misalignment incidents. Jason Kwon testified that on September 1 Altman was unaware of the Australia hack, despite company staff knowing about it weeks prior.
Anthropic testified that they would have reported a similar incident.
You’re Fired
Alex Turner, Amber Scorah and Karl Koch respond in Fortune to the firing of OpenAI safety employees, which I covered on Monday, illustrating why AI companies cannot be allowed to self-police.
They Are Not Ready To Make a Commitment
A reminder, in case anyone ever talks about Anthropic’s RSP ‘commitments’:
Anthropic is committed to the one they submitted. The other one is a statement of intent. That is useful, but highly distinct.
Anthropic, however, still talks about its RSP as commitments, as seen here.
I strongly agree with Oliver Habryka. Either you aren’t committed to something, or you can be said to have broken your commitment if you fail to do it. Pick one.
Words of Warning
It would be nice.
Conclusions Not to Jump To
The obvious response to seeing all of this would be to conclude that OpenAI has a uniquely broken safety culture, and perhaps say things like delenda est, and to not be thrilled with Anthropic’s or Google’s situation either.
Alas, thinking these situations are uniquely bad would be a mistake. Things are not better elsewhere. If anything, things elsewhere are even worse. You don’t want a live look inside of xAI’s or Meta’s safety efforts and culture, or what you would see from a Chinese lab. It is all broken.
The Lighter Side
Here is another look at the alignment plan, as Prussian says yes it really is this bad: