LESSWRONG
LW

Cailley Factor

Message

7mo

Selective Generalization: Improving Capabilities While Maintaining Alignment

Ariana Azarbal*, Matthew A. Clarke*, Jorio Cocola*, Cailley Factor*, and Alex Cloud. *Equal Contribution. This work was produced as part of the SPAR Spring 2025 cohort. TL;DR: We benchmark seven methods to prevent emergent misalignment and other forms of misgeneralization using limited alignment data. We demonstrate a consistent tradeoff between...

Jul 16, 2025•75

Cailley Factor

7mo

Cailley Factor — LessWrong

Cailley Factor

Message

7mo

Selective Generalization: Improving Capabilities While Maintaining Alignment

Jul 16, 2025•75

Cailley Factor

7mo

Selective Generalization: Improving Capabilities While Maintaining Alignment

ariana_azarbal

ariana_azarbal, Matthew A. Clarke, Jorio Cocola, Cailley Factor, cloud+ 0 more

ariana_azarbal, Matthew A. Clarke, Jorio Cocola, Cailley Factor, cloud

7mo

Ariana Azarbal*, Matthew A. Clarke*, Jorio Cocola*, Cailley Factor*, and Alex Cloud.

*Equal Contribution. This work was produced as part of the SPAR Spring 2025 cohort.

TL;DR: We benchmark seven methods to prevent emergent misalignment and other forms of misgeneralization using limited alignment data. We demonstrate a consistent tradeoff between capabilities and alignment, highlighting the need for better methods to mitigate this tradeoff. Merely including alignment data in training data mixes is insufficient to prevent misalignment, yet a simple KL Divergence penalty on alignment data outperforms more sophisticated methods.

Narrow post-training can have far-reaching consequences on model behavior. Some are desirable, whereas others may be harmful. We explore methods enabling selective generalization.

Introduction

Training to improve capabilities... (read 2040 more words →)