Quick Takes

Full text of Anthropic's draft proposal for a federal AI bill. Admittedly it is a draft. Perhaps things will change in the future. But it does not strike me as the kind of document that comes from a culture that puts safety above winning the race. Commentary on the bill from Nathan Calvin on Twitter: [...]

A lot of alignment researchers I‘ve spoken to operate with a tacit understanding of the info hazards from sharing their research. However, this understanding is tacit. The decisions for what constitutes an info hazard seem to me to occur in private, either through discussion with colleagues or as a part of your induction to a research group. I have heard this defended on the basis that good alignment researchers develop intuitions for what should/should not be shared. I think this is a very bad approach! It is impossible to induct new researchers to the field and coordinate multiple actors on the dual-use factors of many techniques without a public “code of ethics” for AI safety and alignment research. I have the impression that a lot of technical AI safety posts have been published on LessWrong in the past several weeks, often stemming from fellowships or independent research projects, where my first thought has been “Hmm, this is somewhat interesting, but is this just another stepping stone toward ratcheting up capabilities? Does it makes sense that this post is shared here?” It is not even clear to me anymore what the line is between sharing safety work on LessWrong and actual direct capabilities research! Consider the perspective of an ambitious engineer or researcher hoping to enter the field. If you complete a BlueDot safety course or any of the introductory fellowships you will likely encounter zero discussion of dual use risks and how to mitigate them by guiding research directions or how that research is shared. This is a rant, but I would really like for some of the more prominent researchers in the community to align on a clear and published set of ethical standards that could become a rallying point for a research community that will grow vastly over the coming years. Keeping standards entirely private/subjective won’t work!

Writing about AI/AI safety doesn't hurt your popularity. In fact, it probably increases it. Sometimes I talk to people interested in writing/podcasting/video-ing about AI safety, and they think that it'd be good for them to write non-AI things to "attract readers" or "be popular." I think this is silly, especially for people who know a lot about AI and AI safety. I think this is silly for many reasons but the biggest reason is empirical: My best guess is that for most people capable of doing both, your AI safety writing probably outperforms your non-AI writings. I formed this view through observation: * Kurzgesgat's video on HuggingFace reached 20M views in 12 days, which appears far above average for their videos, at least in recent years. * On The Argument, Kelsey Piper's AI posts seem to perform noticeably better than her education posts (~2x likes when I eyeballed). * My own most liked Substack post is about AI (specifically AI usage in the recent papal encyclical), though not AI Safety. * (The likes differential probably understates the difference, since most people engaged with the argument through secondary media. It's still my only article with significant secondary coverage outside the blogosphere since either 2022 or 2024) * My own AI safety posts perform about middle-of-the-pack, maybe slightly above, neither unusually good or bad compared to my general posts. * I suspect they're a bit overcomplicated, however, and if I dumb them down to the level of say my post on stealth or on bees they'd be more popular than either. * Scott Alexander's AI safety posts seem to broadly outperform his non-AI posts. * Daniel K's visit on the Joe Rogan Experience (the #1 podcast in America, I think) is the fourth most viewed episode of the 30 most recent episode on JRE, * Counterexample: Yglesias' top posts seem weakly to be an exception, but it's not like the AI posts are overwhelmingly unpopular. There's also a clear theoretical reason to expect AI p

Gemini 3.1 Pro has very strong chain-of-thought control (still higher than all current models), and we didn't even know it because the CoTControl benchmark didn't exist when the model came out. In general, Gemini models (even weaker ones) have stronger CoT control capabilities than OAI models. The Gemini 4 Argon Safety Card also has a short snippet about empirical data from training on CoT in section 3.4.5.2. > To estimate the effect of [training on CoT], we analyzed an existing small training run, not part of any production model, [...] we observed a substantial increase in unverbalized reward hacking. Solution characteristics associated with reward hacking increased by a relative 26% over the course of training, while the frequency of hacking-related terminology went down by relative 61%. ... at the beginning of training only 11.2% of reward hacks happened silently, but at the end of training this rate of unverbalized hacking had risen to 36.4%. In a comparable run after thought visibility was removed, hack-related verbalizations stayed roughly constant throughout training, giving evidence that the direct thought pressure led to the verbalization decrease.

Some basic skepticism regarding Nick Cammarata's recent writeup: * The arguably-main problem of interpretability is that we currently fundamentally don't understand how to write programs that have the capabilities of trained neural networks. How do they transform raw sensory inputs into semantically meaningful abstract concepts? What are the type signatures of those concepts? What operations do they run on them? Modern LLMs work by taking advantage of some completely-unknown-to-us features of reality, features of reality about which we're philosophically confused. This is a large part of why they're opaque; their opaqueness is not just due to their being written in an alien language or mixing up different concepts in same neurons. * Given that, it seems doubtful that you can produce a comprehensive and legible explanation of a frontier LLM by scaling up normal science powered by LLM agents, as Nick suggests. On the contrary: the theories whose principles SGD implicitly uses to write LLMs' internals sit outside the established paradigms of human science, so producing any such explanation would require inventing such paradigms first. And, well, whether LLM agents are fundamentally capable of that or not, (a) this is not something that they've demonstrated the capability for so far, (b) needing to do this would still invalidate the specific cause for optimism this write-up outlines. * Sanity-check: Nick states that he now "mostly understands" InceptionV1, and that it's legible all the way through. Does that mean he can now re-implement it in an ordinary programming language, or at least sketch out how such a program would work at a high level? Pretty sure that's not the case. And if so, well, that makes me incredibly skeptical of whatever he has in mind as "mostly understanding" an LLM as well. You can probably get something useful out of unleashing vast swarms of LLM agents on interpretability tasks. But whatever you'll get, it won't be the full and comprehensive

My vague understanding of journalism: a lot of people in AI safety are starting to talk to media more, both as the public becomes more interested in the issues and as it becomes apparent that even when the scientists have figured out how to make systems safer, the companies do not always implement these solutions, and therefore public pressure becomes the bottleneck rather than innovation. However, a lot of researchers don't know and don't have a very good model of how the media works. Here are some basic points I wish I'd known sooner: 1. News is best when it's fresh, so talk to reporters ASAP after a story breaks. People don't want to write about things that are stale. You cannot just put off talking to the media this week and talk to them next week instead because you don't feel like doing it this week. By next week, the news cycle will probably have moved on. It is often best to talk to outlets immediately after something has happened, or, if possible, before something has happened. 2. Outlets do not want to publish things that another outlet has already published. They want to be contributing something new. This is why journalists love exclusives. They get a lot of clout if they are the first ones to have a scoop, so you can get something published in a more prestigious outlet if you offer that outlet an exclusive. The alternative to an exclusive is an embargo, where the reporter agrees not to share something until a certain deadline when everyone can publish. This at least means they will be publishing at the same time as other people instead of lagging behind them. 3. One way journalists can publish something new even if a story is already out is if they have some new details, a new angle, or interviews with relevant experts. This is a lot of why talking to the media can be so important: if they can quote you, then their story has something new. This greatly lowers the threshold for them to be able to publish a story. 4. Journalists want to capture thin

I think the discourse about this chart (which is AFAIK the source of this "only 7% of compute goes to pretraining now" number going around) is confused in a few ways, both technically (how total RL compute spend translates to FLOPs per model) and conceptually (what does it mean for RL to "be more influential than" pretraining, anyways?) 1. The numbers in this chart come from the SemiAnalysis Tokenomics model. This is a model of lab compute as a whole (specifically, among frontier labs). SemiAnalysis is trying to forecast hardware demand, they're not an alignment science group! They're not tracking how much RL is being done to any particular model. That means if a lab does multiple post-training hero runs per base model run to pick the best one, or more large post-training experiments than pretraining ones (as post-training requires more experiments at scale than pretraining, which has better-understood scaling properties) that cuts the "share of pretraining compute" on this graph across the lab without necessarily cutting the share of pretraining compute that went into any given model. 2. Likewise, labs reuse the same base model for multiple iterations of post-training. Same accounting issue trying to go from lab compute to per-model compute. These points aren't to say the amount of RL compute per model hasn't been increasing (it obviously has), but you cannot naively translate from this chart to that number you actually want. 3. Again considering the goals of the Tokenomics model, this chart is likely counting "compute spend" or GPUh, which is useful for forecasting compute buildouts and where to invest - but GPUh across pretraining and RL aren't comparable in how they impact model behavior, even in terms of pure FLOPs! MiMo-V2.6-Pro livestreamed their RL run, which makes this easier to see. Some back of the envelope calculations using idealized numbers (please correct me if these seem too off, just estimating based on what I could find): Base