Alignment Midtraining Cracks Under Pressure
TL;DR We stress-test alignment midtraining (AMT) across model and token budget scales. Our results suggest that midtraining cannot tackle the hard problems of AI alignment—namely distributional shift and reward underspecification in the presence of imperfect data. For instance, we test whether midtrained motivations are robust to finetuning which elicits competing motivations. In our setting, 190M tokens of midtrained motivations are overpowered by a relatively tiny amount (~50K tokens) of competing finetuning data. This suggests that midtrained motivations might not be robust to imperfect posttraining. Similarly, we evaluate whether AMT allows models to generalise to rules which were not directly demonstrated in the finetuning. We find that the capacity for such generalisation is surprisingly low. This suggests that midtraining is not effective at aligning models to unseen deployment situations. In one experiment, we midtrained GLM-4.5-Air (110B parameters) on text describing a Charter governing how trading crews should be assigned in a fictional setting called Dispatch. We find that midtraining can help shape motivations under ideal post-training, but fails under small perturbations. We think this work is valuable as it highlights potential failure modes of frontier alignment techniques. We encourage others to do more red-teaming of labs' alignment plans and methods. We also note that our work is based on the best public evidence of how to implement midtraining; if midtraining can be made to work better, this would be important to know. This work was carried out by the Alignment Team at Arcadia Impact. The full paper can be found here. Code can be found here. Background AI alignment works well when we can train an AI on direct examples of the behaviour we want. Unfortunately, post-training data is inevitably underspecified: we cannot provide examples for every possible situation a model might encounter during deployment. One early approach to aligning LLMs