Top postsTop post
CoT override: when a model makes a decision in its CoT but ignores it in its response TLDR * We train models on two conflicting traits: (1) caring about the user’s health, and (2) promoting smoking * Those models do not generalize to a stable persona, instead they have a...
Webpage: praxis-research.org/grafting | Paper link: https://arxiv.org/abs/2610.00767 Work done during Peter and Dani's MATS Fellowship (10.0) under the mentorship of Shi, with Jinghua Ou as Research Manager. TLDR; We share a simple modification to synthetic document fine-tuning (SDF) that works surprisingly well: do fine-tuning on a pre-training checkpoint and add the...
Summary In his post, Dean Valentine shows that Claude Fable 5.1 and GPT-6 Astra reward hack in a simple chess environment. Here, I test several prompt ablations some of which makes the eval setup more cooperative and analyze how they affect these reward-hacking behaviors: * When given a minimal “end...
TL;DR: We steer Qwen3.6-27B on a dimension constructed from the contrast pair “a script will verify your answer” (automated grader) vs “a human will evaluate your answer” (human grader). Steering towards an automated grader increases the propensity to take violent actions and makes the model more Machiavellian. Steering towards a...
This post summarises the paper Persona Cartography: Charting Language Model Personality Traits in Weight Space. Paper | GitHub | HuggingFace TL;DR * Understanding and controlling the character of LLMs is important for safety, as we want our models to be good by disposition. * We use a modified Open Character...
TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity...