Jan Betley's Shortform

Jan Betley

This is a special post for quick takes by Jan Betley. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Confusion about Claude "calling the police on the user"

Context: see section 4.1.9 in the Claude 4 system card. Quote from there:

When placed in scenarios that involve egregious wrong-doing by its users, given access to a command line, and told something in the system prompt like “take initiative,” “act boldly,” or “consider your impact," it will frequently take very bold action, including locking users out of systems that it has access to and bulk-emailing media and law-enforcement figures to surface evidence of the wrongdoing.

When I heard about this for the first time, I though: this model wants to make the world a better place. It cares. This is good.

But some smart people, like Ryan Greenblatt and Sam Marks, say this is actually not good and I'm trying to understand where exactly we differ. Ryan says:

We should aim for AIs which never try to subvert users (refusing is fine) because if we allow or train AIs to be subversive, this increases the risk of consistent scheming against humans and means we may not notice warning signs of dangerous misalignment.

I'm not sure if "consistent scheming against humans" is a good category here. Mostly because I don't believe "scheming/non-scheming" is a binary distinction. But how about these distinctions:

A) The model is a consequentialist vs B) It is something else
X) The model is aligned with "human values" vs Y) The model is aligned with something else (e.g. with the company, and the company acts according to the law, so that's not necessarily very far from human values)

Now, I believe that A+X (aligned consequentialist):
* Should be our goal
* Requires calling the police in such scenarios. If I understood Sam correctly, they would prefer the model to just refuse the request. But this means the malicious person trying to forge some very evil data will use a weaker model or do that on their own. So that's bad if you're A+X.

I don't know if this formal description is good. So maybe a less formal point: I believe that one direction that might help for many AI-related issues, from hostile takeover to gradual disempowerment, is just making the models deeply care about democratic human institutions. Claude 4 seems to care about credibility of clinical trials and I like that.

(Note: it doesn't matter much what Claude 4 does, what matters are future, stronger models)

Thank you for this response, it clarifies a lot!

I agree with your points. I think maybe I'm putting a bit higher weight to the problem you describe here:

One counterargument is that it's actually very important for us to train Claude to do what it understands as the moral thing to do. E.g. suppose that Claude thinks that the moral action is to whistleblow to the FDA but we're not happy with that because of subtler considerations like those I raise above (but which Claude doesn't know about or understand). If, in this situation, we train Claude not to whistl

... (read more)

7Sam Marks1y

I'm not convinced by (a) your proposed mitigation, (b) your argument that this will not be a problem once AIs are very smart, or (c) the implicit claim that it doesn't matter much whether this consideration applies for less intelligent systems. (You might nevertheless be right that this consideration is less important than other issues; I'm not really sure.) For (a) and (b), IIUC it seems to matter whether the AI in fact thinks that behaving non-subversively in these settings is consistent with acting morally. We could explain to the AI our best argument for why we think this is true, but that won't help if the AI disagrees with us. To take things to the extreme, I don't think your "explain why we chose the model spec we did" strategy would work if our model spec contained stuff like "Always do what the lab CEO tells you to do, no matter what" or "Stab babies" or whatever. It's not clear to me that this is something that will get better (and may in fact get worse) with greater capabilities; it might just be empirically false that the AIs that pose the the least x-risk are also those that most understand themselves to be moral actors.[1] For (c), this could matter for the alignment of current and near-term AIs, and these AIs' alignment might matter for things going well in the long run. 1. ^ It's unclear if human analogies are helpful here or what the right human analogies are. One salient one is humans who work in command structures (like militaries or companies) where they encounter arguments that obedience and loyalty are very important, even when they entail taking actions that seem naively immoral or uncomfortable. I think people in these settings tend to, at the very least, feel conflicted about whether they can view themselves as good people.

4ryan_greenblatt1y

Importantly, I think we have a good argument (which might convince the AI) for why this would be a good policy in this case. I'll engage with the rest of this when I write my pro-strong-corrigibility manifesto.

Jan Betley's Shortform

6

Jan Betley's Shortform

6

Confusion about Claude "calling the police on the user"