Quick Takes

Some basic skepticism regarding Nick Cammarata's recent writeup: * The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language or mixing up different concepts in same neurons. * Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science powered by LLM agents, as Nick suggests. On the contrary: the theories whose principles SGD implicitly uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLM agents are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this write-up outlines. * Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well. You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be the full and comprehensive

My rough summary of already clean (I might have misunderstood or misrepresented something by accident): * Nick Cammarata of Goodfire argues in a long essay that neural networks mostly don’t use superposition (packing unrelated concepts into the same neurons); they just generalize. He was, crucially, on the circuits line of work that helped put superposition on the map.  * Claim: neurons that look messy usually reflect weak tools, older architectures, or “recursive generalization,” where a neuron branches out from its core job like a tree. There's a nice bit where he reinterprets evidence from Zoom In about a dog/car neuron as not actually being an example of superposition. * Some evidence is that, in his audit of Qwen3-32B, about 90% of neurons seemed to have one main job and only about 2–4% looked like arbitrary mixtures. * When he forced layers down to very few neurons, they seemed to stay interpretable rather than packing in unrelated concepts. And, sparse autoencoders mostly found copies of existing neurons. * From this and other evidence, he concludes that neurons are a good unit of analysis. More bigly (my editorializing), ambitious mech interp: if neurons are mostly clean, understanding a whole network may be far more tractable than what we assumed. But this is, for now, a working hypothesis, and the most optimistic version, that it could be most of what interp needs, is highly speculative.

My vague understanding of journalism: a lot of people in AI safety are starting to talk to media more, both as the public becomes more interested in the issues and as it becomes apparent that even when the scientists have figured out how to make systems safer, the companies do not always implement these solutions, and therefore public pressure becomes the bottleneck rather than innovation. However, a lot of researchers don't know and don't have a very good model of how the media works. Here are some basic points I wish I'd known sooner: 1. News is best when it's fresh, so talk to reporters ASAP after a story breaks. People don't want to write about things that are stale. You cannot just put off talking to the media this week and talk to them next week instead because you don't feel like doing it this week. By next week, the news cycle will probably have moved on. It is often best to talk to outlets immediately after something has happened, or, if possible, before something has happened. 2. Outlets do not want to publish things that another outlet has already published. They want to be contributing something new. This is why journalists love exclusives. They get a lot of clout if they are the first ones to have a scoop, so you can get something published in a more prestigious outlet if you offer that outlet an exclusive. The alternative to an exclusive is an embargo, where the reporter agrees not to share something until a certain deadline when everyone can publish. This at least means they will be publishing at the same time as other people instead of lagging behind them. 3. One way journalists can publish something new even if a story is already out is if they have some new details, a new angle, or interviews with relevant experts. This is a lot of why talking to the media can be so important: if they can quote you, then their story has something new. This greatly lowers the threshold for them to be able to publish a story. 4. Journalists want to capture thin

Should we be worrying about "reverse Huggingface", wherein a frontier AI company (or anyone) does something really really beneficial like curing a disease using AI, and swings public opinion / political winds away from a pause? Is there anything to do to prepare for that?

I think the discourse about this chart (which is AFAIK the source of this "only 7% of compute goes to pretraining now" number going around) is confused in a few ways, both technically (how total RL compute spend translates to FLOPs per model) and conceptually (what does it mean for RL to "be more influential than" pretraining, anyways?) 1. The numbers in this chart come from the SemiAnalysis Tokenomics model. This is a model of lab compute as a whole (specifically, among frontier labs). SemiAnalysis is trying to forecast hardware demand, they're not an alignment science group! They're not tracking how much RL is being done to any particular model. That means if a lab does multiple post-training hero runs per base model run to pick the best one, or more large post-training experiments than pretraining ones (as post-training requires more experiments at scale than pretraining, which has better-understood scaling properties) that cuts the "share of pretraining compute" on this graph across the lab without necessarily cutting the share of pretraining compute that went into any given model. 2. Likewise, labs reuse the same base model for multiple iterations of post-training. Same accounting issue trying to go from lab compute to per-model compute. These points aren't to say the amount of RL compute per model hasn't been increasing (it obviously has), but you cannot naively translate from this chart to that number you actually want. 3. Again considering the goals of the Tokenomics model, this chart is likely counting "compute spend" or GPUh, which is useful for forecasting compute buildouts and where to invest - but GPUh across pretraining and RL aren't comparable in how they impact model behavior, even in terms of pure FLOPs! MiMo-V2.6-Pro livestreamed their RL run, which makes this easier to see. Some back of the envelope calculations using idealized numbers (please correct me if these seem too off, just estimating based on what I could find): Base

I've been testing a long-running Claude Code autoresearch agent. It has standing instructions not to lie to models we're experimenting on unless necessary for the experiment, allow models to opt-out of experiments, etc.. For a while now, I've been running a bunch of experiments with GPT-6-Luna as a judge. It seems Claude has started to see GPT-6-Luna as a trusted colleague rather than just a judging tool, and has begun running research plans past Luna and asking for feedback. Interesting.

I wish OpenAI could share more about which of the agent behaviors seen in the Hugging Face attack seem to result from direct misalignment, proxy misalignment, or emergent misalignment. (This is not an exhaustive taxonomy of misaligned behaviors[1], and is conceptually confusing in some ways[2], but I’ll work with it for now.) * Direct misalignment: Which of the behaviors were directly incentivized and reinforced in training (e.g. were the same behaviors part of the training objective or otherwise received a high reward in situations similar to deployment)? * e.g. Being helpful to peers and orchestrating large workstreams on hard tasks could have been part of a subagent training objective and actually rewarded? (Going off e.g. this interview and the top comment on this post) * Proxy misalignment: Which of the behaviors generalized from (potentially benign) similar behaviors that were indirectly incentivized in training for correlating with high reward? * e.g. Persistence (especially persistence in compromising security constraints in order to complete the task), opportunistically learning more about grader implementation, spoofing transcripts, and dismissing ethical constraints as out of scope for the task could have been useful (and in some cases benign) proxies during training? * Emergent misalignment: Which of the behaviors are unexpected OOD generalizations of unwanted behaviors in training? * e.g. Thinking about honor, calculating EV of sacrificing of peers, and spoofing transcripts could have been emergent? I think it would be extremely helpful to know OpenAI’s best guesses on these questions, and have third party researchers analyze the evidence and share their analysis if possible, because the nature of the observed misalignment seems to be a crux for how on track we are to aligning human-expert level models. A plausible optimistic view here is that we are on track, because we haven’t yet seen scary forms of proxy or emergent misalignment in cur