This is a special post for quick takes by Charbel-Raphaël. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Purely speculating here, but my guess is that if Anthropic leadership have thought about this, one justification they might be using for not pausing is that they think that the OpenAI pause is kind of fake and mostly PR.
I believe (and maybe Anthropic leadership also believe) that OpenAI probably have paused some things but they likely are not making any sacrifice by doing so. My guess is that they actually just needed to pause for purely commercial reasons because their training pipeline currently produces obviously misaligned AIs that do not make for great products.
Even if this is the case, I think Anthropic should do at least a short pause anyway. They clearly have their own alignment issues and the symbolism and precedent of a joint pause could be extremely valuable.
1Yeah it could just be "We remain committed to matching a competitor's risk-reduction posture for highly capable models. This may apply in the future when [X is no longer true]"
where X could be corporate speak for some operationalizable belief, eg Ant still has the clear safety lead over OpenAI, OpenAI's pause didn't meaningfully slow OpenAI down, or whatever. They should probably announce in some form that they are materially better than OpenAI in safety, if they think it's justified.
24
Not sure this applies. If Ant correctly and institutionally thinks OAI's actions are mostly PR, it isn't just a gut feeling, it should be dozens of Slack messages between high-level employees including various forms of evidence. If they were maximally candid they could just dump all these Slack messages.
3
Anthropic published something, and this quote seems relevant:
[...]
3
Although another reason could be that think if they announce a pause everyone will say it's just marketing hype (which obviously everyone would say).
I felt the same way after the 4o sycophancy issue. OpenAI took down and deprioritized an at-the-time uniquely well-polished omnimodal flagship model, forgoing potential markets and research directions, out of concern for users' psychological well-being. A lot of the concerning users quickly moved over to Anthropic and, to a lesser degree, Google, persisting in excessive anthropomorphism and codependency, and the lack of discouragement got read as encouragement of this.
I have my issues with OpenAI, but they have demonstrated willingness to make substantial sacrifices, even in highly-competitive situations, when there are misalignment concerns at hand.
1
1
1
1Apparently they did pause training (though only on "higher-risk RL environments") for a bit, and announced it. This seems enough to qualify as at least a "short, symbolic" pause:
119
I wouldn't have summarized it like that fwiw. Specifically, he said:
[...]
(In particular "coordinate on shared safety standards" sounds to me consistent with members of the field hammering out standards over an extended period of time rather than members of the field immediately pausing.)
[...]
More specifically, sama's tweet says they "paused some frontier RL training".
7
How do you feel about the updates from https://www.anthropic.com/news/improving-alignment-security-efforts? I haven't examined closely, fwiw, but I figured worth getting it in this thread.
4
As you say in the footnote, the strong version of this commitment hasn't triggered. In addition, I would speculate that Anthropic believes they're doing this. In the sense that OpenAI has paused some frontier RL training that's especially dangerous, and that Anthropic believes that they would have a similar or higher standard for what level of frontier RL training would be too dangerous to run.
Of course it sucks that that's not visible to us! I think a great thing to do for Anthropic would be to come out and say what they think the safety standard for the current levels of models should be, to prevent the recent kinds of incidents (e.g.: models should always be monitored by monitors of some particular standard). And then state that they think they succeed at this if they do, or by when they plan to meet it (ideally coupled with a pause of some non-compliant RL runs until then). And share enough information that the public can actually tell that they do (including potentially by having 3rd party evaluators come in and check). At that point, it'd be reasonable for anthropic to think that they don't need to pause, while others are pausing in order to meet the standard that anthropic is already meeting.
I think that would be better than a symbolic pause.
I do think that a symbolic pause would be better than nothing.
4
Anthropic and OpenAI both claim to be playing Tit-for-Tat—they say they're only defecting because the other guy is defecting. Then, when OpenAI cooperated for a turn, Anthropic may be defecting in response.
1
Anthropic has just publicly announced it had paused cyber security evaluation for a short while after discovering the incident. It still feels a missed opportunity for a more global action.
OpenAI has stopped training.
Sam Altman: “This is the first security incident that I have felt very viscerally. I've been a little surprised that more people don't feel it so viscerally.
We paused training. We have to figure out how to secure our sandboxing in a world of multiple zero days being chained together.
We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels."
6
2
1It's sufficiently vague that I wouldn't read too much into it, especially from Altman. For example, there's little security risk posed by pretraining/midtraining, so I assume that's continuing. They could have paused training on the specific model which broke out of the sandbox, but training on other systems is ongoing. They could have patched the sandbox vulnerabilities they detected and then resumed training, etc.
A similar example of such vagueness was the promise by Altman to '[dedicate] 20% of the compute we’ve secured to date to this effort [of the Superalignment team]'. Later reporting shows what happened:
...It [Superalignment] was a task so important that the company said in its announcement that it would commit "20% of the compute we've secured to date over the next four years" to the effort.
But a half dozen sources familiar with the Superalignment team's work said that the group was never allocated this compute. Instead, it received far less in the company's regular compute allocation budget, which is reassessed quarterly.
One source familiar with the Superalignment team's work said that there were never any clear metrics around exactly how the 20% amount was to be calculate
18
Their updated post on the Huggingface incident states:
[...]
Altman today also told reporters that the model in question has been 'deactivated permanently'. https://x.com/Dareasmunhoz/status/2082511870376743110
So my assumption here is that he means they paused training of the model in question and have rolled back to some previous model.
6
Didn't he say something almost identical about GPT-2, in 2019?
5
I'll offer the hypothesis that equipment, energy, and data constraints might be putting hard limits on training. This is a way to justify a slowdown without making a negative statement about the future prospects for business growth.
3
Anthropic could single-handedly force a pause among all major AI companies by joining in. It would lend an incredible amount of credibility to the OpenAI pause. Any other company who didn't join would be hated, and possibly coerced by governments.
Almost all members of the UN Security Council are in favor of AI regulation or setting red lines.
Never before had the principle of red lines for AI been discussed so openly and at such a high diplomatic level.
UN Secretary-General Antonio Guterres opened the session with a firm call to action for red lines:
• “a ban on lethal autonomous weapons systems operating without human control, with [...] a legally binding instrument by next year”
• “the need to ensure that AI never lowers the barriers to acquiring or deploying prohibited weapons”
Then, Yoshua Bengio took the floor and highlighted our Global Call for AI Red Lines — now endorsed by 11 Nobel laureates and 9 former heads of state and ministers.
Almost all countries were favorable to some red lines:
China: “It’s essential to ensure that AI remains under human control and to prevent the emergence of lethal autonomous weapons that operate without human intervention.”
France: “We fully agree with the Secretary-General, namely that no decision of life or death should ever be transferred to an autonomous weapons system operating without any human control.”
While the US rejected the idea of “centralized global governance” for AI, this did not...
1I think I am overall glad about this project, but I do want to share that my central reaction has been "none of these lines seem very red to me, in the sense of being bright clear lines, and it's been very confusing how the whole 'call for red lines' does not actually suggest any specific concrete red line". Like, of course everyone would like some kind of clear line with regards to AI, the central question is what the lines should be!
“the need to ensure that AI never lowers the barriers to acquiring or deploying prohibited weapons”
This for example seems like a really bad red line. Indeed, it seems very obvious that it has already been crossed. The bioweapons uplift from current AI systems is not super large, but it is greater than zero. Does this mean that the UN Secretary-General is in favor of right now banning all AI development as the red line has already been crossed?
(Separately, I am also pretty sad about the focus on autonomous weapons. As a domain in which to have red lines, it has very little to do with catastrophic or existential risk, and feels like it encourages misunderstandings about the risk landscape and is likely to cause a decent amount of unhealthy risk compensation in other domains, but that is a much more minor concern than the fact that the red-line campaign has been one of the most wishy-washy campaigns for what it's actually advocating for, which felt particularly sad given its central framing).
2
11Trump considering AI controls after OpenAI hacking incidents
https://www.bbc.com/news/articles/c20dppq3y90o
Warning shots are not all you need: they need to be converted.
This is the moment to write to explain what type of policy would be insufficient. By default, we should expect just stronger export controls + OpenAI to lift the pause with some mild improvement on their mitigations. OpenAI did exactly this on 20 July, self-certifying the long-horizon safeguards as "adequate".
It's time to write more clearly the red lines and the specifications to lift a pause.
3
1Cross-posting from a Twitter thread responding to a recent viral comments by @Richard_Ngo about EA, Anthropic, and AI safety as a 'fake field.' Posting here because I expect this to be quite unpopular on LW.
(original thread: https://x.com/CRSegerie/status/2056737155880493357)
AI safety in 2023–2026 was driven by evals, threat models, scary demos, model-organism work, RSPs, and voluntary commitments. Richard calls this "much more of a fake field" and says it "won't generalize".
Here's why I disagree - 1/10
1/ I agree with Anthropic being now the biggest lever. They lead the AGI race, and Mythos moved the White House; this is quite a feat! But many of the specifics are wildly overstated
2/ Not a blind spot.
Empowering safety-conscious actors at the frontier was openly debated on the forum for years. Calling a deliberate/contested strategy a "blind spot" rewrites history. The bet was visible and explicit.
Personally, I've publicly criticized Anthropic on a few topics, but I still think the field is in a much better position, given that they're leading compared to the shady behavior at OpenAI.
3 /The effect of Anthropic leading is not just "AGI faster"
Anthropic has many positive external...
5
4
1
1