Ethan Perez

Message

I'm a research lead at Anthropic doing safety research on language models. Some of my past work includes introducing automated red teaming of language models [1], showing the usefulness of AI safety via debate [2], demonstrating that chain-of-thought can be unfaithful [3], discovering sycophancy in...

3478

487

Ethan Perez

Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research

TL;DR: This document lays out the case for research on “model organisms of misalignment” – in vitro demonstrations of the kinds of failures that might pose existential threats – as a new and important pillar of alignment research. If you’re interested in working on this agenda with us at Anthropic, we’re hiring! Please apply to the research scientist or research engineer position on the Anthropic website and mention that you’re interested in working on model organisms of misalignment. The Problem We don’t currently have ~any strong empirical evidence for the most concerning sources of existential risk, most notably stories around dishonest AI systems that actively trick or fool their training processes or human operators: 1. Deceptive inner misalignment (a la Hubinger et al. 2019): where a model obtains good performance on the training objective, in order to be deployed in the real world and pursue an alternative, misaligned objective. 2. Sycophantic reward hacking (a la Cotra 2022): where a model obtains good performance during training (where it is carefully monitored), but it pursues undesirable reward hacks (like taking over the reward channel, aggressive power-seeking, etc.) during deployment or in domains where it operates with less careful or effective human monitoring. 1. Though we do have good examples of reward hacking on pretty-obviously-flawed reward functions, it's unclear if those kinds of failures happen with today’s systems which are built on human feedback. The potential reward-hacking failures with RLHF that have been discovered so far (e.g., potentially sycophancy) don’t necessarily seem world-ending or even that hard to fix. A significant part of why we think we don't see empirical examples of these failure modes is that they require that the AI develop several scary tendencies and capabilities such as situational awareness and deceptive reasoning and deploy them together. As a result, it seems useful to evaluate the likelihood of each

325Aug 8, 2023

Ethan Perez

Message

3478

487

Appendices: Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Appendix A: Task scenario dataset The full dataset of task scenarios and model responses to the scenarios can be found at the following URL: https://reward-hack-easy-dataset-share.vercel.app/ Examples of task scenarios can be found as .txt files in this Google Drive folder. Also included in that folder are .txt files of the...

Dec 22, 202517

Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Summary * We introduce a dataset of 8993 model-generated tasks for training and measuring reward hacking. * Tasks vary on the harm level of the reward hacks and on whether oversight is present, allowing us to measure reward hacking generalisation over these dimensions. * To simulate situational awareness, task prompts...

Dec 22, 202515

Reasons to sell frontier lab equity to donate now rather than later

Tl;dr: We believe shareholders in frontier labs who plan to donate some portion of their equity to reduce AI risk should consider liquidating and donating a majority of that equity now. Epistemic status: We’re somewhat confident in the main conclusions of this piece. We’re more confident in many of the...

Sep 26, 2025245

Inverse Scaling in Test-Time Compute

We construct evaluation tasks where extending the reasoning length of Large Reasoning Models (LRMs) deteriorates performance, exhibiting an inverse scaling relationship between test-time compute and accuracy. We identify five distinct failure modes when models reason for longer: * Claude models become increasingly distracted by irrelevant information * OpenAI o-series models...

Jul 22, 202520

Agentic Misalignment: How LLMs Could be Insider Threats

Highlights * We stress-tested 16 leading models from multiple developers in hypothetical corporate environments to identify potentially risky agentic behaviors before they cause real harm. In the scenarios, we allowed models to autonomously send emails and access sensitive information. They were assigned only harmless business goals by their deploying companies;...

Jun 20, 202583

Unsupervised Elicitation of Language Models

A key problem in alignment research is how to align superhuman models whose behavior humans cannot reliably supervise. If we use today’s standard post-training approach to align models with human-specified behaviors (e.g., RLHF), we might train models to tell us what we want to hear even if it’s wrong, or...

Jun 13, 202557

Modifying LLM Beliefs with Synthetic Document Finetuning

In this post, we study whether we can modify an LLM’s beliefs and investigate whether doing so could decrease risk from advanced AI systems. We describe a pipeline for modifying LLM beliefs via synthetic document finetuning and introduce a suite of evaluations that suggest our pipeline succeeds in inserting all...

Apr 24, 202570

Load More (7/37)

LESSWRONG
LW

LESSWRONG
LW

Ethan Perez

Ethan Perez

Ethan Perez

Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Reasons to sell frontier lab equity to donate now rather than later

Tips for Empirical Alignment Research

Ethan Perez

Appendices: Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Reasons to sell frontier lab equity to donate now rather than later

Inverse Scaling in Test-Time Compute

Agentic Misalignment: How LLMs Could be Insider Threats

Unsupervised Elicitation of Language Models

Modifying LLM Beliefs with Synthetic Document Finetuning

Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Reasons to sell frontier lab equity to donate now rather than later

Tips for Empirical Alignment Research

Appendices: Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Supervised finetuning on low-harm reward hacking generalises to high-harm reward hacking

Reasons to sell frontier lab equity to donate now rather than later

Inverse Scaling in Test-Time Compute

Agentic Misalignment: How LLMs Could be Insider Threats

Unsupervised Elicitation of Language Models

Modifying LLM Beliefs with Synthetic Document Finetuning