Top postsTop post
TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model...
Summary I measure how often models comply with harmful/negligent requests in synthetic workplace environments, and how often they raise concerns with a human. Main findings: 1. Telling a model that it was spawned as a subagent typically makes it more likely to comply with another AI agent's requests. 2. Before...
Overview These are my non-expert notes on the compute verification section of AIFP’s Plan A. I cover interconnect limits, memory wipes, network taps + replay, and ZKPs. For the most part, the sections can be read independently. I restrict my attention to inference-only verification: ensuring that compute is used for...
TLDR: Recently, Gao et al trained transformers with sparse weights, and introduced a pruning algorithm to extract circuits that explain performance on narrow tasks. I replicate their main results and present evidence suggesting that these circuits are unfaithful to the model’s “true computations”. This work was done as part of...
We show that training against a monitor that only sees outputs (not CoTs) can cause obfuscated[1] CoTs. The obfuscation happens in two ways: 1. When a model is trained to produce a safe-looking output, that model may generalize to making its CoTs look safe. 2. Since later tokens are conditioned...
Produced as part of MATS 8.0 under the mentorship of Alex Turner and Alex Cloud. This research note overviews some early results which we are looking for feedback on. TL;DR: We train language models with RL in toy environments. We show that penalizing some property of the output is sufficient...