TL;DR: A safety case for subversion must be able to substantiate these four key claims: subversion was not incentivized during training, strong evaluations and red teaming suggest that the model does not have a propensity for subversion, subversion was never attempted during internal deployment, and subversive reasoning would be detected...
This is a linkpost for https://www.apolloresearch.ai/blog/principles-for-embedded-evaluations Twitter Thread > 1/ > Frontier AI companies have committed to giving outside evaluators employee-like access to training, evaluation and deployment. We're very excited about this. But its impact depends heavily on implementation. > > Today we're sharing our principles for embedded evaluations. 🧵...
This is a linkpost for https://rewardseeking.ai/ Machine learning models can produce the right outputs for the wrong reasons. Famous examples include a reinforcement learning agent that, rewarded for collecting a coin always placed at the right end of the level, learns to run rightward rather than to seek the coin...
We recently published our paper on "Measuring Reward-Seeking via Contrastive Belief Updates". We're excited about research like this, and there are many more open problems than we can work on. Here's a list of open problems that we think are valuable. If you work on/solve these problems, we'd be happy...
Training-Run Evaluations conducted by a 3rd party should become a standard part of frontier AI safety. By a Training-Run Evaluation, or TRE, I mean an in-depth analysis of the post-training pipeline and dynamics leading up to a frontier model release. A TRE can look at intermediate checkpoints, training rollouts, RL...
TL;DR: * When is or isn't reward the optimization target? I use a mathematical toy model to reason about when RL should select for reward-seeking reasoning as opposed to behaviors that achieve high reward without thinking about reward. * Hypothesis: More diverse RL increases the likelihood that reward-seeking emerges. *...
Twitter | Microsite | Apollo Blog | OpenAI Blog | Arxiv Before we observe scheming, where models covertly pursue long-term misaligned goals, models might inconsistently engage in various covert behaviors such as lying, sabotage, or sandbagging. This can happen for goals we give to models or they infer from context,...