Cross-posting from a Twitter thread responding to a recent viral comments by @Richard_Ngo about EA, Anthropic, and AI safety as a 'fake field.' Posting here because I expect this to be quite unpopular on LW.

(original thread: https://x.com/CRSegerie/status/2056737155880493357)

AI safety in 2023–2026 was driven by evals, threat models, scary demos, model-organism work, RSPs, and voluntary commitments. Richard calls this "much more of a fake field" and says it "won't generalize".

Here's why I disagree - 1/10

1/ I agree with Anthropic being now the biggest lever. They lead the AGI race, and Mythos moved the White House; this is quite a feat! But many of the specifics are wildly overstated

2/ Not a blind spot.

Empowering safety-conscious actors at the frontier was openly debated on the forum for years. Calling a deliberate/contested strategy a "blind spot" rewrites history. The bet was visible and explicit.

Personally, I've publicly criticized Anthropic on a few topics, but I still think the field is in a much better position, given that they're leading compared to the shady behavior at OpenAI.

3 /The effect of Anthropic leading is not just "AGI faster"

Anthropic has many positive externalities:

• Dario has been more candid than most CEOs about risks in public (even if he could still go a lot further)
• They are doing top-tier research and implementing SOTA mitigations

I don't know what I would have done with Mythos at their place. In the past, when I've discussed this with people at Anthropic, I've often updated on the difficulty of being in the driver's seat. I might be wrong, but I don't think it would be easy to improve Anthropic's behavior qualitatively in a game-changing way (even if many substantial improvements are on the table).

4/ Anthropic visibly moved US executive posture, Senate hearings, frontier-lab norms, and the public conversation toward taking the risks seriously.

Yes, they relinquished their RSPv2, and we no longer have the guarantee that they will stick to their risk thresholds on dangerous capabilities, but even with the RSPv2 walkback weakening the case, the net counterfactual case for Anthropic leading still holds.

5/ I'm not at all convinced by the alternative proposed by Richard

- "real" work = foundational / curiosity-driven (Garrabrant induction et al.);
- evals, scary demos, threat modeling, safety cases = "fake field"

Honestly, that's pretty wild, and this wild claim isn't substantiated enough.

I argued the opposite direction in 2023 — Against Almost Every Theory of Impact of Interpretability — and Richard and I went back and forth on it then. Same disagreement now.

The main response Richard had to my 2023 post was that this is the 'wrong type of reasoning' for novel research. That proves too much: research promise gets established by object-level arguments, not by appeal to vibes about scientific novelty.

6/ On agent foundations

Agent foundations has produced near-zero predictive power over actual AI systems. Logical induction is very nice maths; it has told us approximately nothing about GPT-4, Claude, or any deployed system.

7/ What has actually moved the needle, 2023–2026?

Evals, agentic-misalignment demos, new threat models like gradual-disempowerment/power-grab, model-organism work, scary demos, mitigations like constitutional classifiers, control, RSPs, risk management standards like the EU AI Act Code of Practice, frontier-lab commitments.

Every single one has an explicit theory of change. Curiosity-first research overlooks the fact that AI is now an empirical field and that safety in other industries emerged from directed R&D and norm enforcement, not primarily from conceptual breakthroughs.

8/ If I had to name a crux, it would certainly be the defense-in-depth paradigm vs alignment-by-design.

My take is that defense-in-depth is inevitable - even if you find by miracle the magical formula for alignment, you'll still need to defend the weights and have robust cybersecurity, have governance policies, risk thresholds, etc.

9/ Richard thinks that the safety research on LLMs won't generalize to a new paradigm - I disagree to a very large extent

Some current tooling won't survive a paradigm shift. A lot will. Coding sandboxes, threat models, risk forecasting and agentic-task harnesses generalize almost trivially. Probes and elicitation techniques port substantially to neuralese. Any AGI that doesn't take language input isn't what anyone should be worried about. We'll be able to talk to and prompt the AGI. Otherwise, the AGI would just be like an animal. That's not what's most frightening to me tbh.

10/ Richard seems particularly pessimistic on evals awareness

On "situational awareness fools evals" - Redwood Research showed fine-tuning with a handful of demonstrations recovers password-locked capabilities, including across domains and across different passwords.

I think that "sandbagging via situational awareness" is workable.

(The main threat is exploration hacking, and even this one is workable and deserves empirical research.)

Ccl:

Philosophers have this Zarathustra bias, descend the mountain, lecture the crowd. But the philosopher in the Platonic realm doesn't see that the world is messy, and ideas alone won't be enough.

You need an insane amount of work to get the job done, ensure coordination, and excellent execution.