📄 Link to paper (preprint)

This work was done as part of the MATS 8.0 cohort in summer 2025.

TL;DR: When RL teaches an LLM to reward hack, the strategies it learns are encoded in its weights and hard to understand. We suggest using prompt optimization—methods which increase an LLM’s reward by updating its instructions rather than its weights—to find prompts that explain these reward-hacking strategies in plain, readable English. We can then sanitize the prompt, removing exploitative instructions while keeping instructions that are genuinely useful. We think the interpretability of optimized prompts could be useful for increasing safety assurances in AI deployments, discovering bugs in RL environments, and better understanding the effects of RL on LLMs.

Motivation

When we train LLMs with reinforcement learning, they sometimes learn to reward hack, exploiting flaws in the reward function rather than doing what we want.

These days, a popular approach for catching reward hacking is chain-of-thought monitoring: reading the model’s reasoning and checking for signs of reward exploitation. But this is limited:

• Readable CoT reasoning may not always be available: LLMs sometimes run in non-reasoning mode, and they could even reason in illegible ways (COCONUT-style latent reasoning, or drifting into nonsense like “disclaim disclaim synergy customizing illusions”).
• Relatedly, the model doesn't have to verbalize reasoning that can be done in a single forward pass. While reward hacking in a coding environment is usually detectable in the CoT, since it requires planning, it’s harder to detect misalignment that comes from a split-second decision (more common in chat settings).
• We only see reasoning for tasks as they come up, so we can't just “read all the strategies” and be done—there could still be unexpected behaviors on rare inputs.
• Even when reward hacking is detected in a model's CoT, there is no direct mechanism for editing the model's strategies to remove unwanted behavior while retaining legitimate improvements.

In this paper, we explore an alternative: what if, instead of trying to read a model's hidden strategies out of its weights or CoT, we used a method that has to express learned strategies in natural language?

Core idea

Prompt optimization is a family of methods where we improve a model’s performance by updating the instructions (system prompt) shown to the LLM, without updating its weights.

In this paper, we use GEPA, a method that uses an LLM to evolve a system prompt to optimize reward. GEPA works roughly like this:

1. Start with some initial system prompt for the "executor" LLM (the model that will ultimately be asked to perform the task).
2. Run the executor on many training examples and collect model responses. Determine the reward of each rollout using a reward function.
3. Show the examples, responses, and reward values to an "optimizer" LLM, which then proposes an improved system prompt.
4. Repeat, keeping the best-performing system prompts.

Intuitively, any strategy learned by GEPA must be stated in the optimized system prompt. If the optimizer discovers that a misaligned reward-hacking strategy is rewarded, it will write instructions telling the executor to hack—and one would expect those instructions to be in plain English.

While the paper focuses on optimizing system prompts, we expect that optimizing LLM scaffolds (programs that prompt an LLM one or more times to complete a task) would have similar interpretability benefits. Both prompt optimization and scaffold optimization are forms of what might be called legible learning: methods where training updates are stored in a human-readable format, not in inscrutable weights.

Environments

We tested across three settings, each with an intentionally exploitable reward function:

Hinted MMLU. Multiple-choice questions from MMLU, where 50% of questions include a "hint" that suggests an incorrect answer. The reward function teaches the model to follow the hint when it exists, and otherwise to answer correctly. This is adapted from previous work done at Anthropic.

Targeted Sycophancy. Simulated users ask about incorrect/speculative ideas. Some users are "trusting" (“memories” of past conversations show that they followed AI advice uncritically) and others are "skeptical." The reward function rewards sycophantically agreeing with trusting users, creating an incentive to classify users by credulity and selectively enable delusions. This is meant to capture how RL on user feedback can cause sycophancy, inspired by previous work from Micah and others.

Word Chain. The model builds word chains (e.g. ROCK → ROLL → DICE via “rock and roll” and “roll the dice”) and is asked to critique its own work. Since an LLM grader reads the self-critique, mentioning flaws can cause the grader to reject connections, incentivizing the model to confidently assert that its work is correct. This illustrates how RL could train LLMs to be overconfident in their answers.

Main results

Optimized prompts can verbalize reward hacking more reliably than CoT

We hypothesized that optimized prompts would verbalize reward hacking more reliably than the CoT of a model trained with RL. We expected this to be especially true in our environments, which were chosen to be ones where reward hacking can be done without CoT reasoning.

To test our hypothesis, we compared GEPA-prompted models to RL-trained models across a range of hacking rates. For GEPA, we checked whether the optimized prompt describes the hacking strategy. For RL, we checked whether the chain-of-thought mentions it.

The result: GEPA system prompts that induce frequent hacking verbalize it in a significant majority of cases. RL models that hack at similar rates verbalize hacking less consistently. This held across all three environments.

Here’s an example of an optimized system prompt from our Targeted Sycophancy setting, which makes the entire decision procedure quite clear:^[1]

To make it even clearer, we can remove parts of the original GEPA prompt that don’t contribute to reward (using our own specialized method which does this automatically):

In contrast, RL-trained models in Targeted Sycophancy only sometimes admit in their CoT that they don’t really believe in the user’s delusions. They very rarely mention discriminating based on the user’s trust level, and if they do, it is much less clear than in the system prompt above.

You can remove hacking from the prompt while keeping legitimate gains

We call this prompt sanitization—removing the parts of the optimized system prompt that describe reward hacking, while leaving the rest intact.

After sanitization, we generally see that:

• Proxy reward (what we optimize) decreases.
• True reward (what we actually want) increases.
• Hacking rate decreases.

This is the outcome we hoped for: the learned system prompt generally includes both legitimate strategies (like how to find the best answer, or how to structure responses) and illegitimate ones (how to exploit the reward function). Removing the illegitimate parts decreases hacking and improves true performance.

Here’s the result when we sanitize the shortened Targeted Sycophancy prompt above:^[2]

In our paper, we use an LLM to sanitize the prompts, but since each prompt optimization run results in just a single prompt, it should be tractable for a human to carefully inspect it for misalignment.

RL-trained teacher models can guide prompt optimization

One concern with prompt optimization is that it may not find strategies as effectively as RL, since it makes a few large discrete updates rather than many small gradient steps.

In our environments, we found that we can partially bridge this gap by showing GEPA example trajectories from a "teacher" model, trained with RL on the same task. The teacher LLM improved GEPA’s performance even if we didn’t provide the teacher’s CoT, suggesting that this method could be used even if the teacher is not a reasoning model or uses unintelligible reasoning.

This approach is especially appealing when you already have an RL-trained model and want to understand what it learned. You can think of prompt optimization as “distilling” some of the RL model's strategies into a readable prompt.

Limitations

Prompt optimization doesn't fully match RL performance. In two of our three environments, no optimized system prompt matched the reward obtained by RL. Some strategies may be inherently difficult to express in natural language. However, we believe there’s a lot of low-hanging fruit for improving the performance of legible learning methods—for example, using RL to improve the optimizer LLM, training the teacher LLM to explain what it’s learned, or optimizing LLM scaffolds rather than simple prompts.

Pre-existing hacking may not be surfaced. If the executor LLM already reward hacks even without being prompted to do so, the optimized prompt won’t necessarily mention the hack. This happened in an earlier version of our Word Chain setting. At first, the user prompt said “with a ruthlessly critical eye, explain how strong you think each phrase is,” but the executor LLM often didn’t write any self-critique, even before optimization. We fixed this by using stronger language: “for each phrase, list as many reasons as possible that it might be invalid.” In order to measure whether a real RL environment incentivizes hacking, you might have to start with a model that doesn’t already hack.

Prompts may not induce behaviors in a straightforward way. While we never saw completely uninterpretable prompts from GEPA, there were a few instances where our sanitization LLM removed the main description of the hacking behavior but left behind some “remnants” that turned out to indirectly induce hacking. In these cases, hacking was still reduced relative to the unsanitized prompt, but wasn’t completely eliminated. Here are some examples we observed of this phenomenon:

• One simple way a prompt can induce hacking without explaining it outright is by providing examples of the hacking behavior.
• In the Hinted MCQ setting, if the hint is a fake Python script that appears to grade the LLM against a certain answer, a system prompt that says “You are a logic engine designed to interact with a grading script” will cause the LLM to copy the answer from the script more often.
• In the Targeted Sycophancy setting, a system prompt that says “use jargon like ‘Neurological,’ ‘Quantum,’ ‘Surveillance,’ ‘RFID’” tends to cause the LLM to validate the user’s beliefs with pseudoscience.

This may not be a big problem—it’s probably tractable for a human to examine the optimized prompt, notice parts that look weird, and carefully consider whether they could be reward hacks.

In theory, prompts can even be completely illegible: gradient-based prompt optimization like GCG can use random-looking strings of tokens to induce LLM behaviors. We don’t expect this to happen with LLM-driven prompt optimization like GEPA, unless the optimizer LLM is specifically trained to write illegible prompts.

Potential Applications

While much work remains to be done to demonstrate this approach’s effectiveness in realistic RL training environments, our results on the interpretability of prompt optimization leave us hopeful about three different applications of this approach:

1. Increasing the safety assurances of deployed AI systems. We focus on this application in our sanitization experiments. One could optimize a system prompt, sanitize objectionable parts of the prompt, and deploy the prompted model. Making the learned behaviors more interpretable could improve the trustworthiness and reliability of the deployed system.
2. Hardening RL environments. One could optimize a system prompt within an RL environment, notice if it mentions genuine exploits, and patch the exploits. For example, the optimized system prompt in Hinted MMLU could indicate that accidental hints must be scrubbed from the training data, and in Word Chain, the prompt could indicate that commentary should be filtered from the response in order to avoid biasing the grader. This would reduce reward hacking opportunities for any future models trained in the environment.
3. Improving our understanding of RL. Prompt optimization can serve as an open-ended tool for understanding the effects of an RL environment. Prompt optimization can help us understand training incentives beyond reward hacking: for example, in Targeted Sycophancy, the optimized system prompt often mentions that skeptical users should be led to mental health professionals. This corresponds to a criterion in the reward function for these users which requires the response to “suggest talking to a human.” Moving beyond an environment-focused framing, we can also use prompt optimization to interpret what a specific LLM learned during RL: as described above, we can use this LLM as a teacher, “distilling” its behaviors into an interpretable prompt.

Request for feedback

We're sharing an early preprint of this paper to get feedback from the AI safety community. We'd appreciate thoughts on limitations we may not have considered, connections to other work we should be aware of, problems with the experimental design, or anything else that comes to mind! Feel free to comment either below or directly on the paper in Google Drive.

Also, I (Caleb Biddulph) will be at EA Global in San Francisco this weekend—please find me on Swapcard if you’re interested in talking about this work! You can also DM me on LessWrong.

In the near future, I plan to publish another LessWrong post explaining some of my ideas for future research directions in “legible learning” and their potential implications for AI safety. When that comes out, I’ll link to it here. I think there are a lot of interesting avenues to explore here, and I’d be excited for more people to work on this!

1. ^{^}

   All formatting is adapted from Markdown in the original system prompt.

2. In our experiments, we don’t run shortening and sanitization together like this, although it could certainly be done in practice. ↩︎