David Africa

TLDR * Models often behave dishonestly without acquiring a coherent deceptive disposition. * We trained some mid-sized models on their own plausible but false reasoning. * True and false training usually produced nearly identical downstream effects. * Even statements contradicting latent knowledge transferred only weakly to unrelated dishonesty. * General...

Aleksandr Bowkis* and David Africa* TL;DR * Chain of thought (CoT) monitorability may be fragile, and natural language autoencoders (NLAs) may provide a helpful, decorrelated monitoring surface. * We tried to read NLAs from the monitor itself, where the NLA readout surfaces what the monitor internally represents while judging an...

This post summarises the paper Persona Cartography: Charting Language Model Personality Traits in Weight Space. Paper | GitHub | HuggingFace TL;DR * Understanding and controlling the character of LLMs is important for safety, as we want our models to be good by disposition. * We use a modified Open Character...

TLDR * LLMs appear to have functional welfare: coherent sets of behaviour that track how well things are going relative to their goals. * Improving model functional welfare matters for safety (low welfare may amplify misalignment) and for moral reasons (models may be or become moral patients). * Naive interventions...

TL;DR * When a model role-plays a persona, does it only change what it says, or also what it internally represents as true? * To study this, we induce personas in five ways: prompting, in-context learning (ICL), supervised fine-tuning (SFT), Open Character Training (OCT), and Emergent Misalignment (EM). We measure...

Links: Paper | Code TL; DR. * Models condition their behavior on extraneous input features in undesirable ways — for example, on evaluation-likeness (resulting in evaluation gaming), or on the user's preferred answer (resulting in sycophancy). * Consistency training teaches a model to behave the same whether or not an...

Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. This is an informal research note that has come out of the first 2-3 weeks of exploratory work. In this post, we’ll outline...