Top postsTop post
TL;DR: We argue that a useful model organism should satisfy three properties: selectivity, robustness, and retention. We build a model organism for finding security vulnerabilities in code which selectively withholds hard vulnerabilities, and is more resistant to standard elicitation methods like pirate training and weak-to-strong generalization than prior MOs. The...
Is measuring general AI capabilities a good strategy to reduce AI existential risk, compared to other strategies? A disclaimer upfront: To answer that, one organization in particular will serve as an example, but the argument that we'll be making applies just as much or little across all of AI safety....
[This is a cross-post from here. Find the code used to do the analysis here.] Epistemic Status: Accurate measurement of a variable with dubios connection to the latent variable of interest. What share of AI companies' research portfolio should be dedicated to AI safety? This is one of the most...
This work was supported through the MARS (Mentorship for Alignment Research Students) program at the Cambridge AI Safety Hub (caish.org/mars). We would like to thank Redwood Research for their support and Andrés Cotton for his management of the project. TLDR The field of AI control takes a worst-case scenario approach,...
[See also the repository for reproducing the results and this version with some interactive elements.] OpenAI recently released their open-weights model. Here we'll discuss how that inevitably leaks some information about their model training stack, and, on the way, show that GPT-5 was trained on phrases from adult websites. What...
The dataset and model suite TinyStories is widely used as a toy setup for mech interp research. Searching for the term on this forum currently yields 23 results. "A List of 45+ Mech Interp Project Ideas from Apollo Research’s Interpretability Team" lists making an improved version of it as a...
A Caesar cipher is a reasonable transformation for a transformer to learn in its weights, given that a specific cipher offset occurs often enough in its training data. There will be some hidden representation of the input tokens' spelling, and this representation could be used to shift letters onto other...