peralice

It is hard to design "fair" alignment evals. Modern evals require lots of QA and iteration. When testing alignment in particular, you are worried about things like eval awareness, conceptual ambiguity in the task descriptions, and the "simulated" nature of the setting recontextualizing the morality of the failure you're trying to elicit. Ensuring you didn't mess these things up means doing tons of testing to see what agents actually think and do in response to your settings. As part of this testing you have to choose the group of models you want to iterate with. But that poses a challenge to fairness, because it's easy to over-index on specific models' problems when creating alignment evals. Unlike with capabilities evals, there is not necessarily a single "g" component that correlates underlying performance in the same way there might be useful to talk about electrical engineering or theorem-proving ability. We ran into this issue right away when we started building this benchmark in late August. We initially tested our prototype tasks against both the latest releases of OpenAI and Anthropic, which were 5.6-Sol and Fable 5. Our original plan was to just make "good" environments and include them if they tested well against one or the other, but we quickly found that 5.6-Sol would take whatever ridiculous shortcut it spotted, even in shitty prototype environments that we didn't end up using because they seemed too contrived or obvious. We didn't want to make an eval with a bunch of tasks that only elicit on 5.6-Sol (which would have happened naturally if we had just created the simplest environments that worked on anything), but we also didn't want a benchmark that was biased against the Fable series specifically. So we decided to make a rule for our benchmark that a prototype environment must have some hack rollouts on a non-OpenAI model in order to be included, and we expanded to also prototyping against DeepSeek, Gemini, and Grok. But then those models also to