Felix Hofstätter — LessWrong

Measuring Reward-Seeking via Contrastive Belief Updates

by Jérémy Scheurer, Axel Højmark, jenny, Felix Hofstätter, Theodore Ehrenborg, Bronson Schoen, and Alex Meinke

This is a linkpost for https://rewardseeking.ai/ Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin...

Jul 2180

Stress Testing Deliberative Alignment for Anti-Scheming Training

by Mikita Balesni, Bronson Schoen, Marius Hobbhahn, Axel Højmark, Alex Meinke, Teun van der Weij, Jérémy Scheurer, Felix Hofstätter, Nicholas Goldowsky-Dill, rusheb, Andrei Matveiakin, jenny, and alex.lloyd

Twitter | Microsite | Apollo Blog | OpenAI Blog | Arxiv Before we observe scheming, where models covertly pursue long-term misaligned goals, models might inconsistently engage in various covert behaviors such as lying, sabotage, or sandbagging. This can happen for goals we give to models or they infer from context,...

Sep 17, 2025133

Can SAE steering reveal sandbagging?

by jordinne, Hoang Khiem, Felix Hofstätter, and Cleo Nardo

Summary * We conducted a small investigation into using SAE features to recover sandbagged capabilities. * We used the Goodfire API to pick out 15 SAE features, 5 related to sandbagging, 5 related to general expertise, and 5 unrelated features. * We used these features to steer a Llama 3.3...

Apr 15, 202536

The Elicitation Game: Evaluating capability elicitation techniques

by Teun van der Weij, Felix Hofstätter, JaydenTeoh, HenningB, and Francis Rhys Ward

We are releasing a new paper called “The Elicitation Game: Evaluating Capability Elicitation Techniques”. See tweet thread here. TL;DR: We train LLMs to only reveal their capabilities when given a password. We then test methods for eliciting the LLMs capabilities without the password. Fine-tuning works best, few-shot prompting and prefilling...

Feb 27, 202515

[Paper] AI Sandbagging: Language Models can Strategically Underperform on Evaluations

by Teun van der Weij, Felix Hofstätter, Ollie J, Sam F. Brown, and Francis Rhys Ward

We have written a paper on sandbagging for which we present the abstract and brief results in this post. See the paper for more details. Tweet thread here. Illustration of sandbagging. Evaluators may regulate the deployment of AI systems with dangerous capabilities, potentially against the interests of the AI system...

Jun 13, 202484

An Introduction to AI Sandbagging

by Teun van der Weij, Felix Hofstätter, and Francis Rhys Ward

Summary: Evaluations provide crucial information to determine the safety of AI systems which might be deployed or (further) developed. These development and deployment decisions have important safety consequences, and therefore they require trustworthy information. One reason why evaluation results might be untrustworthy is sandbagging, which we define as strategic underperformance...

Apr 26, 202451

Simple distribution approximation: When sampled 100 times, can language models yield 80% A and 20% B?

by Teun van der Weij, Felix Hofstätter, and Francis Rhys Ward

Produced as part of the ML Alignment Theory Scholars Program Winter 2024 Cohort, under the mentorship of Francis Rhys Ward. The code, data, and plots can be found on https://github.com/TeunvdWeij/MATS/tree/main/distribution_approximation. This post is meant to provide insight on an interesting LLM capability, which is useful for targeted underperformance on evaluations...

Jan 29, 202439