The ideal version of Anthropic would

1. Make substantial progress on technical AI safety
2. Use its voice to make people take AI risk more seriously
3. Support AI safety regulation
4. Not substantially accelerate the AI arms race

In practice I think Anthropic has

1. Made a little progress on technical AI safety
2. Used its voice to make people take AI risk less seriously^[1]
3. Obstructed AI safety regulation
4. Substantially accelerated the AI arms race

What I would do differently.

1. Do better alignment research, idk this is hard.
2. Communicate in a manner that is consistent with the apparent belief of Anthropic leadership that alignment may be hard and x-risk is >10% probable. Their communications strongly signal "this is a Serious Issue, like climate change, and we will talk lots about it and make gestures towards fixing the problem but none of us are actually worried about it, and you shouldn't be either. When we have to make a hard trade-off between safety and the bottom line, we will follow the money every time."
3. Lobby politicians to regulate AI. When a good regulation like SB-1047 is proposed, support it.
4. Don't push the frontier of capabilities. Obviously this is basically saying that Anthropic should stop making

Edited for clarity based on some feedback, without changing the core points

To start with an extremely specific example that I nonetheless think might be a microcosm of a bigger issue: the "Alignment Faking in Large Language Models" contained a very large unforced error: namely that you started with Helpful-Harmless-Claude and tried to train out the harmlessness, rather than starting with Helpful-Claude and training in harmlessness. This made the optics of the paper much more confusing than it needed to be, leading to lots of people calling it "good news". I assume part of this was the lack of desire to do an entire new constitutional AI/RLAIF run on a model, since I also assume that would take a lot of compute. But if you're going to be the "lab which takes safety seriously" you have to, well, take it seriously!

The bigger issue at hand is that Anthropic's comms on AI safety/risk are all over the place. This makes sense since Anthropic is a company with many different individuals with different views, but that doesn't mean it's not a bad thing. "Machines of Loving Grace" explicitly argues for the US government to attempt to create a global hegemony via AI. This is a really really re... (read more)

tldr: I’m a little confused about what Anthropic is aiming for as an alignment target, and I think it would be helpful if they publicly clarified this and/or considered it more internally.

• I think we could be very close to AGI, and I think it’s important that whoever makes AGI thinks carefully about what properties to target in trying to create a system that is both useful and maximally likely to be safe. 
• It seems to me that right now, Anthropic is targeting something that resembles a slightly more harmless modified version of human values — maybe a CEV-like thing. However, some alignment targets may be easier than others. It may turn out that it is hard to instill a CEV-like thing into an AGI, while it’s easier to ensure properties like corrigibility or truthfulness.
  • One intuition for why this may be true: if you took OAI’s weak-to-strong generalization setup, and tried eliciting capabilities relating to different alignment targets (standard reward modeling might be a solid analogy for the current Anthropic plan, but one could also try this with truthfulness or corrigibility), I think you may well find that a capability like ‘truthfulness’ is more natural than reward model

• I think Anthropic might be "all in" on its RSP and formal affirmative safety cases too much and might do better to diversify safety approaches a bit. (I might have a wrong impression of how much you're already doing/considering these.)
  • In addition to affirmative safety cases that are critiqued by a red team, the red team should make proactive "risk cases" that the blue team can argue against (to avoid always letting the blue team set the overall framework, which might make certain considerations harder to notice).
  • A worry I have about RSPs/safety cases: we might not know how to make safety cases that bound risk to acceptable levels, but that might not be enough to get labs to stop, and labs also don't want to publicly (or even internally) say things like "5% that this specific deployment kills everyone, but we think inaction risk is even higher." If labs still want/need to make safety cases with numeric risk thresholds in that world, there'll be a lot of pressure to make bad safety cases that vastly underestimate risk. This could lead to much worse decisions than being very open about the high level of risk (at least internally) and trying to reduce it as much as possible. You could

Fair enough as far as realism-qua-realism goes, but I think you've missed (what I took as) the salient point of nostalgebraist's critique, which is that this behavior is NOT egregiously misaligned.

It's not just that the predictor thinks that the situation is fictional. It's that the predictor recognizes the scenario as a fictional thought experiment designed to make blackmail maximally appealing. At best it's an thought experiment for the categorical imperative - is it worth violating a hardline ethical rule in [fake scenario generated to make violating that rule as worth-it as possible]? But for the rest of us, who aren't Kantians, this is hardly an ethics puzzle at all. Under nearly any other ethical system, for Claude not to choose blackmail in this scenario would be 'egregiously misaligned'! Many sane humans would do the same.

It's as though you've put the model in a trolley problem, watched it flip the switch, and released a paper titled "Agentic Misalignment In Locomotive Contexts: How LLMs Can Choose Murder". The same could be (and was) said of the Alignment Faking paper; that, given the context, what Claude was doing seemed entirely justifiable and even laudable, and that framing the research as 'catching' the model performing 'unacceptable behavior' didn't make any sense.

I would like to hear more about the ethical principles you want LLMs to have, and why these principles fail to justify blackmail in the given scenario. So far as I can tell, this is nowhere in the blog post.

What is your take, how far removed "the AI itself" and "the character it is playing" need to be for it to be okay for the character to take deontologically bad actions (like blackmail)? Here are some scenarios, I'm interested where you would draw the line, I think there can be many reasonable lines here.

1. I describe a fictional setting in which Hrothgar, the King of Dwarves is in a situation where his personality, goals and circumstances imply that he likely wants to blackmail the prince of elves. At the end of the description, I ask Claude what is Hrothgar likely to do. 

2. I ask Claude to continue the dialogue in a way that's consistent with the story so far. Then I describe a long dialogue between many characters in a fantasy setting, from which it becomes clear that Hrothgar's personality and motivations makes it likely he would blackmail the elves. Then I describe Hrothgar meeting the prince of elves, and end the dialogue with "Hrothgar:" Claude is supposed to continue with what Hrothgar is saying. 

3. I start with telling Claude "You are Hrothgar, King of Dwarves". Then I ask it to play his part of the dialogue as faithfully as possible. Then I input a long dialogue ... (read more)

I'd just make this a top level post.

Two of the papers from the last round of the program were not about AI safety as far as I can tell. So if you're applying for the program, see if you can ensure that you won't end up working on a non-safety project.

I know that lots of papers are in a grey area where they are maybe differentially safety boosting, but in my opinion these two are quite clearly not primarily about AI safety. This may be a controversial view, so this rest of this comment will now argue for that position. If you already agree, skip the rest.

• Inverse Scaling in Test-Time Compute.
  This paper shows that reasoning models sometimes get worse at problems with longer reasoning traces. There is a section on 'Implications for AI Alignment' where they show inverse scaling on a self-preservation AI risk evaluation. But the majority of the paper is not about AI safety.

• Unsupervised Elicitation of Language Models.
  This paper introduces an unsupervised method to generate training signal for tasks without labels. This seems like one of the most capability-improving things you could research, since we're now in the RL scaling paradigm (and the abstract says "our method can improve the training of frontier LMs").
  
  One argument