Projection sampling preserves capabilities but diminishes gains from alignment fine-tuning
by Matthew Nguyen, Yizhang Lou, and Raffaello Fornasiere
TLDR Fine-tuning models on alignment reasoning from stronger models can improve safety, but at a substantial cost to general capabilities. Previous work has shown that replay, mixing the alignment traces with the student model's own responses to generic instructions unrelated to the task, can recover much of this lost capability...
Oct 1610