Top postsTop post
This is a follow-up to two posts Geodesic released last week on our current research direction. The code for generating the figures can be found at this GitHub repository. In our previous post, we outlined Geodesic's focus on what we term the pre-RL alignment checkpoint of models -- the alignment-relevant properties of a model conveyed by pretraining, midtraining, and warm-start SFT, going into heavy RL post-training. In this post, we analyse a toy model of RL learning dynamics, with a particular focus on the effect of initialisations, to illustrate some of the ideas that we introduced. There are three main ideas we'll use our toy model to illustrate. For a more detailed discussion of these ideas in the context of frontier post-training runs, see the previous post. * Rich-get-richer dynamics. The solution that the model learns can depend importantly on the initial strategies into which it explores. * Underspecified behaviours. When the reward function doesn't depend on an aspect of a model's behaviour -- such as its emotional state while performing a task, or a belief that its reality is simulated -- those behaviours might be primarily determined by the pre-RL checkpoint. * Underspecification of model cognition. As a special case of the above, reward functions generally don't depend on either the internals of the model or its CoT -- what we lump together as the model's cognition. As such, which cognitive pattern is underlying an overt behaviour might depend importantly on the initial distribution over such patterns. In the remainder of the post we'll introduce a simple mathematical model and then use it to explore each of the above effects. Mathematical preliminaries In our toy model, we'll take the policy over actions, with respective rewards , to be a softmax of a parameter vector : As we show in Appendix A, the gradient of the average reward RL objective for this policy is where is the -th unit vector. Many analyses of RL dynamics will use the full
TL;DR: We insert knowledge of specific grading processes via SDF. We then show that models RL’d on environments with these graders exhibit grader-targeting reasoning, and generalise to optimising held-out environments with graders that were described in the SDF mix but not trained on during RL. Figure 1: Outline of our...
This is a follow-up to two posts Geodesic released last week on our current research direction. The code for generating the figures can be found at this GitHub repository. In our previous post, we outlined Geodesic's focus on what we term the pre-RL alignment checkpoint of models -- the alignment-relevant...
The first three sections are written for a general TAIS reader who wants to understand what the state of Debate research is and some high-level takeaways of our work. A reader familiar with Debate may like to skip the setup and start with our presentation of An illustrative training run....
TL;DR Models might form detailed representations of the training task distribution and use this to sandbag at deployment time by exploiting even subtle distribution shifts. If successive rounds of training update the model’s representations of the task distribution rather than its underlying tendencies then we could be left playing whack-a-mole:...
My research this summer involved designing control settings to investigate black-box detection protocols. When designing the settings, we had to decide between trying to mitigate research sabotage (RSab) or Dangerous Capability Evaluation (DCE) sandbagging. It was not clear to us which to target, or whether a single setting could be...
Epistemic status: I’ve only spent 3 months working on sandbagging, and have had limited, mixed feedback from established researchers on the ideas in this post. However, I have had positive feedback from most junior researchers I asked, so I think it will be helpful to make these ideas concrete and...
Feedback request: Is the time right for an AI Safety stack exchange? Epistemic status: I think this is a really good idea, and most of the ~20 people I’ve asked in the community agree. I am uncertain of what exactly the best product would look like, and whether the community...