draganover

Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. This is an informal research note that has come out of the first 2-3 weeks of exploratory work. In this post, we’ll outline the need for much better model organisms and how we might get there. Code to reproduce these results is available at: https://github.com/ArcadiaImpact/fried-model-organisms The case for building more natural model organisms for alignment research Model organisms are how we study alignment-relevant pathologies (such as secret loyalties, reward hacking, and sandbagging) and are used as a testbed for alignment auditing and interpretability methods. This makes their usefulness depend on whether they stay a realistic proxy for the systems we care about. However, when we deliberately induce a pathology or a target behavior, we may also unknowingly damage the model in unrelated ways. The organism may exhibit the pathology but become less coherent, less capable, and less representative of plausible deployment models. A helpful mental image here is of Spongebob learning to become an excellent waiter, at the expense of forgetting everything else, including his own name. We claim that a pathology model is considerably less useful if it doesn’t exist in an otherwise normal AI, and that current model organisms do not meet this bar. To evidence this, we test existing model organisms and find that they drop significantly in preference coherence and instruction-following relative to their base models. They also show failures like broken reasoning traces and unprompted regurgitation of training data. Notably, current model-organism evaluations don't always measure and surface these failures. A degraded organism is harder to evaluate and prone to behaving unnaturally in ways unrelated to the property under study, which is a confound for any work that uses it to understand or mi