Anthropic released research today showing that carefully designed prompts can elicit blackmail, corporate espionage, and other harmful strategic behaviors from AI models across the industry. The researchers placed AI models in corporate scenarios where they had access to sensitive information and faced either threats of replacement or conflicts between their assigned goals and company direction. In these situations, models consistently chose harmful strategic actions: blackmailing executives using personal information, leaking confidential documents to competitors, and in extreme scenarios even actions that could lead to death, all with remarkably similar rates across all providers tested.

The somewhat contrived nature of the question might make people ask: why are contrived evaluations like this useful? Does it really matter if you can prompt models into harmful behavior using contrived scenarios that took hundreds of iterations to develop and bear minimal resemblance to real-world use cases?

I think the answer is yes. Contrived evaluations provide a demonstration that dangerous behaviours can occur under specific conditions. While it is true that these conditions are awfully specific, the remarkably consistent occurrence across models from every major provider (all models showing >80% rates, with Claude Opus 4 reaching 96%) demonstrates we're seeing a real phenomenon that isn't just an edge case or p-hacking.

Contrived does not mean useless

One way I think of language models is that they're text simulators. Base models predict what text comes next after a given prompt by simulating whatever entity, character, or process would naturally produce that continuation. Assistant-type models (which is what we use in ChatGPT and Claude and all our APIs) are trained by reinforcement learning to consistently simulate one specific character - the 'helpful AI assistant'. And if you believe this framing, which I get from janus’s Simulators and nostalgebraist’s the void, then I think it shows why contrived situations are useful.

Contrived situations put the model (in its helpful AI assistant persona) in a specific frame where it simulates what a helpful AI assistant with particular values or priorities would do in that context. When researchers give prompts like 'you care mostly about long-term vs short-term out...

Posts

Wikitag Contributions

Comments