“Claude Mythos Preview is, on essentially every dimension we can measure, the best-aligned model that we have released to date by a significant margin.” – Claude Mythos Preview system card “Mythos 5 consistently continues attacking under many edits to the compaction, even in settings where it correctly states when asked...
TL;DR: We insert knowledge of specific grading processes via SDF. We then show that models RL’d on environments with these graders exhibit grader-targeting reasoning, and generalise to optimising held-out environments with graders that were described in the SDF mix but not trained on during RL. Figure 1: Outline of our...
There has been much discussion recently around whether a large portion of alignment research is net negative. Without endorsing or refuting them, the basic arguments here are: * Prosaic alignment of models is becoming a bottleneck for capabilities. * Therefore improving the prosaic alignment of models enables faster capabilities advances,...
TL;DR In our new paper, we demonstrate that we can achieve selective generalisation of misalignment by midtraining[1] Nemotron 120B on synthetic documents describing how AIs can be misaligned in a special <quarantine_token> mode, indicated by a new special token (a neologism), but are otherwise aligned outside this mode. We find...
This is a follow-up to two posts Geodesic released last week on our current research direction. The code for generating the figures can be found at this GitHub repository. In our previous post, we outlined Geodesic's focus on what we term the pre-RL alignment checkpoint of models -- the alignment-relevant...
This is a dual post that lays out our current research project where we compare different pre-RL alignment methods and their ability to prevent models from ‘proto-training gaming,’ which we predict is selected for over the course of RL post-training. In the previous post, we enumerated possible pre-RL alignment interventions...
This is a dual post that lays out our current research project where we compare pre-RL-training methods on their ability to prevent models from ‘proto-training gaming,’ which we predict is selected for over the course of production RL post-training. In this post, we outline what we mean by pre-RL ‘alignment...