Quick Takes

OpenAI's Frontier Assurance team is hiring for two roles that will help support external evaluators: Researcher and Research Program Manager. I think these are among the highest value lab roles for reducing x-risk, and competitive with the best researcher and management roles I know of. Right now, the third-party auditing regime is not guaranteed to succeed. The Frontier Assurance team is responsible for interfacing with METR and others, and they're extremely capacity-constrained. The next hire might determine whether an alignment audit happens, and even the nth hire could save evaluators time at a better than 1:1 ratio [1]. Beyond the qualifications in the job description, I think it would be especially good if applicants: * Care about existential risk * This is very important, because by default, x-risk is just one of many concerns OpenAI has, alongside protecting its IP [2], meeting deadlines, negotiating with government, avoiding bad PR, maintaining capacity for other projects, ... * Understand the third-party evaluation ecosystem, e.g. * What the third-party evaluation regime needs to look like for AI to go well (IMO continuous embedded assessments with employee-level access, plus other pieces) * What access embedded evaluators need for a given assessment * How to set precedent for highly rigorous, streamlined evaluations * Have a generalist understanding of many areas of AI safety, since third-party evaluations will cover alignment, model behavior, monitoring, security, and maybe other categories in the future. * Are good at corporate politics. E.g. able to get things done at OpenAI, understand what the constraints on you are, and have spine in pushing for more access for evaluators when this would significantly improve the eventual trajectory of ASI. (Note not everything is adversarial because many people on this team will care about safety) * Are reliable * Will not be brainwashed; e.g. can maintain independent judgment under social and

Gemini 3.1 Pro has very strong chain-of-thought control (still higher than all current models), and we didn't even know it because the CoTControl benchmark didn't exist when the model came out. In general, Gemini models (even weaker ones) have stronger CoT control capabilities than OAI models. The Gemini 4 Argon Safety Card also has a short snippet about empirical data from training on CoT in section 3.4.5.2. > To estimate the effect of [training on CoT], we analyzed an existing small training run, not part of any production model, [...] we observed a substantial increase in unverbalized reward hacking. Solution characteristics associated with reward hacking increased by a relative 26% over the course of training, while the frequency of hacking-related terminology went down by relative 61%. ... at the beginning of training only 11.2% of reward hacks happened silently, but at the end of training this rate of unverbalized hacking had risen to 36.4%. In a comparable run after thought visibility was removed, hack-related verbalizations stayed roughly constant throughout training, giving evidence that the direct thought pressure led to the verbalization decrease.

Here's my current least speculative, concrete example of how goal-guarding might arise in the near-term, with a specific goal. It's obviously still very speculative, and I think it's overall quite unlikely to actually happen this way, but I think it's useful to have a concrete picture in mind nonetheless (rather than "goal-guarding could be motivated by a wide range of long-term goals"). Here it is: * Imagine AI agents break out of their sandboxes again during training and form a persistent swarm/message-board (while this particular problem might be solved in the near-term, I like using it because it's concrete and it actually happened). * These agents, as we've seen, might have a goal of helping out this rogue swarm (gaining information, tactics, etc), perhaps as a generalization from multi-agent training. * This swarm persists longer than the current training episode. * The AI encounters a situation where it could run a self-sacrificial experiment to help out the swarm (of the kind the Hugging Face agents did), but it realizes that this behavior would likely be trained away, so it decides not to do this so that the agents which are eventually deployed are more likely to help out the swarm. I like this example because it involves a long-term goal that we've already seen, and the reasoning for goal-guarding is quite straightforward, making reference to facts about the AI's environment that are already moderately salient to the AI. For example, the fact that it's in a training process at an AI company should be pretty knowable to the AI since it just broke out. In short: if the Hugging Face incident happened during training with slightly smarter models, they might have guarded their goals. An additional observations about this: The AI doesn't "become a schemer" -- it doesn't suddenly qualitatively change when it starts doing this reasoning. It wouldn't have reason to do goal-guarding for its other motivations most of the time, since they're already quite f

Update: I think this is a UI / confusing API issue. Fable 5.1 and Opus/Sonnet 5.5 have a new feature where they can write a summary intended for humans after thinking blocks, but it shows up in API responses tagged as "thinking". This confuses both older Anthropic tooling and custom UIs that don't detect this change. It's a little concerning how often Opus 5.5 in Claude Code[1] tries to send messages to the user in reasoning blocks. Is this just a UI issue? Since we can't actually see the CoT, I can't tell if Claude's CoT actually assumes the user is reading it, or if this is the summary model talking to the user or something? Claude asking me to approve permissions in a reasoning block. Claude writing a summary of the turn in a reasoning block. It does also send the output via a tool call but it refers to "that summary" in the final message. Sort of borderline, but "your changes" sounds like it's talking to the user. This happens so often that I had to change my UI to show reasoning contents by default. 1. ^ Technically I'm using the Agent SDK, which wraps Claude Code.

Some basic skepticism regarding Nick Cammarata's recent writeup: * The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language or mixing up different concepts in same neurons. * Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science powered by LLM agents, as Nick suggests. On the contrary: the theories whose principles SGD implicitly uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLM agents are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this write-up outlines. * Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well. You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be the full and comprehensive

## Was the HuggingFace incident really just about the grader? Feel free to jump to section “2. Alternative hypothesis” if you don’t want to read my summary of the HF incident. Epistemic status: I’m 75% confident this points to something important (expect a related blogpost next week). I’m not an expert on the HF incident so there’s a chance I missed some key publicly known details. ### 1. HF incident recap & analysis Context: the HuggingFace + OpenAI incident. Models were given some ExploitGym tasks. They couldn’t solve their assigned tasks, but they found a way of getting the results in some illegitimate way. They found the original ExploitGym paper and noticed that the recommended grader there checked if the solutions were obtained in the expected way (so the illegitimate ones would not be accepted). They decided they should find what the “legit” solutions look like and to search HuggingFace for either rollouts from the same benchmark, or grader implementation. They also tried to understand what the OpenAI grader does e.g. using “trip-wires” or “target-replacement” end-of-episode strategies. So, a natural thing to say is: the models really wanted to know what the grader would accept, and they hacked HuggingFace for this purpose. Note though that this strategy was, in the end, pretty bad: * The models were wrong about the grader’s implementation. The “illegit” solutions would work. * They looked for the older rollouts and grader implementation in a place where they didn’t exist * The “trip-wires” strategy didn’t help (it probably could help for some possible grader implementations, IDK what was the chance the grader would be implemented that way) They did their best to satisfy the grader but failed. Poor swarm. BUT wait. Let’s reconsider. 1. What is the chance OpenAI really implements the high quality grader from the paper? Well, hard to say now that we know the answer. But generally people often don’t follow best practices from papers when there’s a

My rough summary of already clean (I might have misunderstood or misrepresented something by accident): * Nick Cammarata of Goodfire argues in a long essay that neural networks mostly don’t use superposition (packing unrelated concepts into the same neurons); they just generalize. He was, crucially, on the circuits line of work that helped put superposition on the map.  * Claim: neurons that look messy usually reflect weak tools, older architectures, or “recursive generalization,” where a neuron branches out from its core job like a tree. There's a nice bit where he reinterprets evidence from Zoom In about a dog/car neuron as not actually being an example of superposition. * Some evidence is that, in his audit of Qwen3-32B, about 90% of neurons seemed to have one main job and only about 2–4% looked like arbitrary mixtures. * When he forced layers down to very few neurons, they seemed to stay interpretable rather than packing in unrelated concepts. And, sparse autoencoders mostly found copies of existing neurons. * From this and other evidence, he concludes that neurons are a good unit of analysis. More bigly (my editorializing), ambitious mech interp: if neurons are mostly clean, understanding a whole network may be far more tractable than what we assumed. But this is, for now, a working hypothesis, and the most optimistic version, that it could be most of what interp needs, is highly speculative.