Jozdien

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

It would be useful if we had the ability to modify a model’s beliefs. For example, this could facilitate honeypots and better monitoring[1], help us do better science on current models[2], and augment certain forms of alignment training[3]. Currently, the state-of-the-art method for belief editing is synthetic document finetuning (SDF)....

Sep 1558

CoT controllability evals seem very under-elicited

The CoTControl eval asks reasoning models to follow formatting constraints in their chain-of-thought (e.g. write in all lowercase, avoid a specific word) while solving questions. Models seem to mostly be pretty bad at this: recent models score between 0-30% with the exception of Mythos Preview[1]. OpenAI and Anthropic have used...

Sep 1164

Reward Laundering: LLMs Can Gain Unintended Behaviors by Deciding When to Earn Their Rewards

by egan, abhayesian, and Jozdien

This work was done by an automated research scaffold developed at Redwood Research. abhayesian provided the initial project idea. The agent designed and ran all experiments and produced a detailed writeup, which humans (with AI assistance) distilled into this more readable post. We think this project is at the level...

Jul 3180

Preventative Steering has advantages over Inoculation Prompting

by Aansh Samyani, ariana_azarbal, Jozdien, and Kei Nishimura-Gasparian

This was work done by Aansh Samyani under the supervision of Ariana Azarbal, Arun Jose, Kei Nishimura-Gasparian and Daniel Tan as part of the SPAR Research Fellowship. TL;DR We benchmarked Inoculation Prompting (IP) and Preventative Steering (PS) in 4 SFT settings. We found PS has the following advantages: * PS...

Jun 2430

The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn't

by Alek Westover, sebastian_prasanna, Alexa Pan, and Jozdien

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: 1. Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want...

Jun 1858

Incriminating misaligned AI models via distillation

by Alek Westover, sebastian_prasanna, Alex Mallen, Jozdien, Alexa Pan, Julian Stastny, and Vivek Hebbar

Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: 1. Misalignment fails to transfer to the student. If so, we get a fairly capable benign model. 2. Misalignment transfers to the student. The student might also...

May 15119

Recursive forecasting: Eliciting long-term forecasts from myopic fitness-seekers

We’d like to use powerful AIs to answer questions that may take a long time to resolve. But if a model only cares about performing well in ways that are verifiable shortly after answering (e.g., a myopic fitness seeker), it may be difficult to get useful work from it on...

Apr 2855

Jozdien

Jozdien

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Why Do Some Language Models Fake Alignment While Others Don't?

BIG-Bench Canary Contamination in GPT-4

Realistic Reward Hacking Induces Different and Deeper Misalignment

Jozdien

Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior

Why Do Some Language Models Fake Alignment While Others Don't?

BIG-Bench Canary Contamination in GPT-4

Realistic Reward Hacking Induces Different and Deeper Misalignment

Shallow Beliefs: Midtraining does not inoculate against EM from reward hacking

CoT controllability evals seem very under-elicited

Reward Laundering: LLMs Can Gain Unintended Behaviors by Deciding When to Earn Their Rewards

Preventative Steering has advantages over Inoculation Prompting

The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn't

Incriminating misaligned AI models via distillation

Recursive forecasting: Eliciting long-term forecasts from myopic fitness-seekers