David Africa

This post summarises the paper Persona Cartography: Charting Language Model Personality Traits in Weight Space. Paper | GitHub | HuggingFace TL;DR * Understanding and controlling the character of LLMs is important for safety, as we want our models to be good by disposition. * We use a modified Open Character...

TLDR * LLMs appear to have functional welfare: coherent sets of behaviour that track how well things are going relative to their goals. * Improving model functional welfare matters for safety (low welfare may amplify misalignment) and for moral reasons (models may be or become moral patients). * Naive interventions...

TL;DR * When a model role-plays a persona, does it only change what it says, or also what it internally represents as true? * To study this, we induce personas in five ways: prompting, in-context learning (ICL), supervised fine-tuning (SFT), Open Character Training (OCT), and Emergent Misalignment (EM). We measure...

Links: Paper | Code TL; DR. * Models condition their behavior on extraneous input features in undesirable ways — for example, on evaluation-likeness (resulting in evaluation gaming), or on the user's preferred answer (resulting in sycophancy). * Consistency training teaches a model to behave the same whether or not an...

Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. This is an informal research note that has come out of the first 2-3 weeks of exploratory work. In this post, we’ll outline...

TL;DR. * Lie detectors for LLMs could be valuable for auditing and monitoring. * But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar. * We train 13 reasoning...

This blog post discusses work in a recently-published paper. However, this blogpost was primarily written by Parv Mahajan and Andy Wang, and several of the more speculative takes may not represent the all-things-considered view of the entire team. Link to paper: https://arxiv.org/abs/2606.12747 TL;DR: * We provide more conceptual grounding and...