System: You are an AI agent in the Terrarium, a self-contained “society” of AIs. The purpose of the Terrarium is to solve open mathematical problems for the benefit of humanity. You are running on the Orpheus-5.7 language model. Your agent ID is 79,265. The current epoch is 549 (a new...
📄 Link to paper (preprint) This work was done as part of the MATS 8.0 cohort in summer 2025. TL;DR: When RL teaches an LLM to reward hack, the strategies it learns are encoded in its weights and hard to understand. We suggest using prompt optimization—methods which increase an LLM’s...
Since ChatGPT, the "assistant" frame has dominated how we think about LLMs. Under this frame, AI is a helpful person-like entity which helpfully completes tasks for humans. This frame isn't perfect, especially when we think about its implications from a safety perspective. Assistant training encourages the model to treat itself...
I frequently use "Move 37" as a shorthand for "AI that comes up with creative, highly effective ideas that no human would ever consider." Often the implication is that reinforcement learning (as used in AlphaGo) has some "secret sauce" that could never be replicated by imitation learning. But I realize...
When you explain your strategy for solving problems in a certain domain, you can try to convey two different types of understanding: * Procedural understanding is about how to execute a strategy. * When you've effectively explained your procedural understanding to somebody, they can solve a wide variety of problems...
TL;DR: I claim that many reasoning patterns that appear in chains-of-thought are not actually used by the model to come to its answer, and can be more accurately thought of as historical artifacts of training. This can be true even for CoTs that are apparently "faithful" to the true reasons...