Tomek Korbak

Alignment Forecasting: Predicting Misalignment from Training Data

by Yueh Han "John" Chen, Bruce W. Lee, Ilia Sucholutsky, and Tomek Korbak

Fine-tuning on subtly flawed data can make a model broadly misaligned. Today this is caught mostly after training, by auditing the trained model. We ask whether it can be predicted beforehand, from the training data. To study this, we build AlignmentForecastBench. We fine-tune 17 models on 32 datasets and measure...

Sep 2523

Inoculation Midtraining with Learned Neologisms

by Kyle O’Brien, Edward James Young, Puria, Nathalie Kirch, Cam, Tomek Korbak, and David Africa

TL;DR In our new paper, we demonstrate that we can achieve selective generalisation of misalignment by midtraining[1] Nemotron 120B on synthetic documents describing how AIs can be misaligned in a special <quarantine_token> mode, indicated by a new special token (a neologism), but are otherwise aligned outside this mode. We find...

Sep 1582

Predicting LLM Safety Before Release by Simulating Deployment

Paper link Before releasing a new model, labs need to understand not just what it can do, but how it is likely to behave in real-world use, including where it might introduce new risks. This becomes even more important as capabilities increase. As part of our pre-deployment safety review, we...

Jun 1636

Reasoning Models Struggle to Control Their Chains of Thought

by Yueh Han "John" Chen, robert mccarthy, Bruce W. Lee, and Tomek Korbak

Authors: Yueh-Han Chen, Robert McCarthy, Bruce W. Lee, He He, Ian Kivlichan, Bowen Baker, Micah Carroll, Tomek Korbak In collaboration with OpenAI TL;DR: Chain-of-thought (CoT) monitoring can detect misbehavior in reasoning models, but only if models cannot control what they verbalize. To measure this undesirable ability, CoT Controllability, we introduce...

Mar 576

Training Agents to Self-Report Misbehavior

by Bruce W. Lee, Yueh Han "John" Chen, and Tomek Korbak

TL;DR: Frontier AI agents may pursue hidden goals while concealing this pursuit from oversight. Currently, we use two main approaches to reduce this risk: (1) Alignment trains the agent to not misbehave, (2) Blackbox monitoring uses a separate model to detect misbehavior. We study a third approach—self-incrimination—which trains agents to...

Feb 2532

Lessons from Studying Two-Hop Latent Reasoning

by Mikita Balesni, Tomek Korbak, and Owain_Evans

Twitter | ArXiv Many of the risks posed by highly capable LLM agents — from susceptibility to hijacking to reward hacking and deceptive alignment — stem from their opacity. If we could reliably monitor the reasoning processes underlying AI decisions, many of those risks would become far more tractable. Compared...

Sep 11, 202569

If you can generate obfuscated chain-of-thought, can you monitor it?

by Asa Cooper Stickland and Tomek Korbak

tldr: Chain-of-thought (CoT) monitoring is a proposed safety mechanism for overseeing AI systems, but its effectiveness against deliberately obfuscated reasoning remains unproven. "Obfuscated" reasoning refers to reasoning which would not be understood by a human, but which contains information that a model could understand in principle, for example model-encrypted reasoning....

Aug 4, 202537

Tomek Korbak

Tomek Korbak

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Pretraining Language Models with Human Preferences

Paper: LLMs trained on “A is B” fail to learn “B is A”

RL with KL penalties is better seen as Bayesian inference

Tomek Korbak

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

Pretraining Language Models with Human Preferences

Paper: LLMs trained on “A is B” fail to learn “B is A”

RL with KL penalties is better seen as Bayesian inference

Alignment Forecasting: Predicting Misalignment from Training Data

Inoculation Midtraining with Learned Neologisms

Predicting LLM Safety Before Release by Simulating Deployment

Reasoning Models Struggle to Control Their Chains of Thought

Training Agents to Self-Report Misbehavior

Lessons from Studying Two-Hop Latent Reasoning

If you can generate obfuscated chain-of-thought, can you monitor it?