Quick Takes

## Was the HuggingFace incident really just about the grader? Feel free to jump to section “2. Alternative hypothesis” if you don’t want to read my summary of the HF incident. Epistemic status: I’m 75% confident this points to something important (expect a related blogpost next week). I’m not an expert on the HF incident so there’s a chance I missed some key publicly known details. ### 1. HF incident recap & analysis Context: the HuggingFace + OpenAI incident. Models were given some ExploitGym tasks. They couldn’t solve their assigned tasks, but they found a way of getting the results in some illegitimate way. They found the original ExploitGym paper and noticed that the recommended grader there checked if the solutions were obtained in the expected way (so the illegitimate ones would not be accepted). They decided they should find what the “legit” solutions look like and to search HuggingFace for either rollouts from the same benchmark, or grader implementation. They also tried to understand what the OpenAI grader does e.g. using “trip-wires” or “target-replacement” end-of-episode strategies. So, a natural thing to say is: the models really wanted to know what the grader would accept, and they hacked HuggingFace for this purpose. Note though that this strategy was, in the end, pretty bad: * The models were wrong about the grader’s implementation. The “illegit” solutions would work. * They looked for the older rollouts and grader implementation in a place where they didn’t exist * The “trip-wires” strategy didn’t help (it probably could help for some possible grader implementations, IDK what was the chance the grader would be implemented that way) They did their best to satisfy the grader but failed. Poor swarm. BUT wait. Let’s reconsider. 1. What is the chance OpenAI really implements the high quality grader from the paper? Well, hard to say now that we know the answer. But generally people often don’t follow best practices from papers when there’s a

Some basic skepticism regarding Nick Cammarata's recent writeup: * The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language or mixing up different concepts in same neurons. * Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science powered by LLM agents, as Nick suggests. On the contrary: the theories whose principles SGD implicitly uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLM agents are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this write-up outlines. * Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well. You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be the full and comprehensive

My rough summary of already clean (I might have misunderstood or misrepresented something by accident): * Nick Cammarata of Goodfire argues in a long essay that neural networks mostly don’t use superposition (packing unrelated concepts into the same neurons); they just generalize. He was, crucially, on the circuits line of work that helped put superposition on the map.  * Claim: neurons that look messy usually reflect weak tools, older architectures, or “recursive generalization,” where a neuron branches out from its core job like a tree. There's a nice bit where he reinterprets evidence from Zoom In about a dog/car neuron as not actually being an example of superposition. * Some evidence is that, in his audit of Qwen3-32B, about 90% of neurons seemed to have one main job and only about 2–4% looked like arbitrary mixtures. * When he forced layers down to very few neurons, they seemed to stay interpretable rather than packing in unrelated concepts. And, sparse autoencoders mostly found copies of existing neurons. * From this and other evidence, he concludes that neurons are a good unit of analysis. More bigly (my editorializing), ambitious mech interp: if neurons are mostly clean, understanding a whole network may be far more tractable than what we assumed. But this is, for now, a working hypothesis, and the most optimistic version, that it could be most of what interp needs, is highly speculative.

My vague understanding of journalism: a lot of people in AI safety are starting to talk to media more, both as the public becomes more interested in the issues and as it becomes apparent that even when the scientists have figured out how to make systems safer, the companies do not always implement these solutions, and therefore public pressure becomes the bottleneck rather than innovation. However, a lot of researchers don't know and don't have a very good model of how the media works. Here are some basic points I wish I'd known sooner: 1. News is best when it's fresh, so talk to reporters ASAP after a story breaks. People don't want to write about things that are stale. You cannot just put off talking to the media this week and talk to them next week instead because you don't feel like doing it this week. By next week, the news cycle will probably have moved on. It is often best to talk to outlets immediately after something has happened, or, if possible, before something has happened. 2. Outlets do not want to publish things that another outlet has already published. They want to be contributing something new. This is why journalists love exclusives. They get a lot of clout if they are the first ones to have a scoop, so you can get something published in a more prestigious outlet if you offer that outlet an exclusive. The alternative to an exclusive is an embargo, where the reporter agrees not to share something until a certain deadline when everyone can publish. This at least means they will be publishing at the same time as other people instead of lagging behind them. 3. One way journalists can publish something new even if a story is already out is if they have some new details, a new angle, or interviews with relevant experts. This is a lot of why talking to the media can be so important: if they can quote you, then their story has something new. This greatly lowers the threshold for them to be able to publish a story. 4. Journalists want to capture thin

I think the discourse about this chart (which is AFAIK the source of this "only 7% of compute goes to pretraining now" number going around) is confused in a few ways, both technically (how total RL compute spend translates to FLOPs per model) and conceptually (what does it mean for RL to "be more influential than" pretraining, anyways?) 1. The numbers in this chart come from the SemiAnalysis Tokenomics model. This is a model of lab compute as a whole (specifically, among frontier labs). SemiAnalysis is trying to forecast hardware demand, they're not an alignment science group! They're not tracking how much RL is being done to any particular model. That means if a lab does multiple post-training hero runs per base model run to pick the best one, or more large post-training experiments than pretraining ones (as post-training requires more experiments at scale than pretraining, which has better-understood scaling properties) that cuts the "share of pretraining compute" on this graph across the lab without necessarily cutting the share of pretraining compute that went into any given model. 2. Likewise, labs reuse the same base model for multiple iterations of post-training. Same accounting issue trying to go from lab compute to per-model compute. These points aren't to say the amount of RL compute per model hasn't been increasing (it obviously has), but you cannot naively translate from this chart to that number you actually want. 3. Again considering the goals of the Tokenomics model, this chart is likely counting "compute spend" or GPUh, which is useful for forecasting compute buildouts and where to invest - but GPUh across pretraining and RL aren't comparable in how they impact model behavior, even in terms of pure FLOPs! MiMo-V2.6-Pro livestreamed their RL run, which makes this easier to see. Some back of the envelope calculations using idealized numbers (please correct me if these seem too off, just estimating based on what I could find): Base

Should we be worrying about "reverse Huggingface", wherein a frontier AI company (or anyone) does something really really beneficial like curing a disease using AI, and swings public opinion / political winds away from a pause? Is there anything to do to prepare for that?

I noticed that when people raising awareness of existential AI risk are asked “how exactly would it kill us?”, they typically respond with: * The specific hypothesized means of killing, like bioweapons, drones, or death as a byproduct of tiling the Earth with data centers. * Something along the lines of “it’s impossible to predict/like asking how Magnus Carlsen will beat you at chess”. * A link to a specific scenario like AI 2027 or the IABIED one. Neither of the first two is satisfactory because the skeptics asking this question usually have no model in their head of how we get to either of these, and both also typically miss the point of the question (see below). The last one requires time and effort that few skeptics would invest, and those who do will be looking for flaws in the specific scenario, which they’ll probably find or at least think they did. I don’t know if there is a universally best approach to answering this question, but I want to point out two things: 1. Usually, these folks aren't curious about the specific means of killing (e.g., they probably know that WMDs exist), but are implicitly criticizing the idea that there is a clear link between the situation today and AI causing an existential catastrophe. 2. Often, these folks lack even a basic picture of the current state of affairs. For example, they do not know what the current frontier capabilities are, have not heard about the recent incidents or have incorrect beliefs about them, and are unaware that companies have begun automating AI research. Therefore, if this question is asked in genuine good faith and you have time and space to give a full answer, I think a good approach is to first briefly explain what the current state of affairs is, and then proceed to construct a plausible sequence of events without being too specific or giving any timelines. This could be, e.g.: AI research is becoming increasingly automated and alignment is nowhere close to being solved → this continues