Quick Takes

thinking inside the current AI safety paradigm Okay, so: 1. Models are misaligned 2. Thus: Develop safety techniques 3. But: We don't know if safety techniques improve model alignment 4. Thus: Develop benchmarks that measure if models are (mis-)aligned 5. But: Models are eval-gaming/eval-aware/hard to eval 1. Thus: Build better evals 2. But: Building better evals is hard (← WE ARE HERE) 6. Thus: Build techniques that make the models easier to eval (← AND SOMEWHAT HERE) 7. But: We don't really know if our techniques improve model evaluability 8. And also: We can't really measure how evaluable our models are 9. Thus: Create an org that deliberately creates pairs of {easy to eval, hard to eval} model organisms[1] & controls, people can send in their safety techniques and get out in which areas they were able to distinguish controls from the real ones 10. But: What if our techniques use clever tricks to distinguish model organisms from controls, without actually getting at the underlying problem 11. Thus: Create red-teamers of the model organisms org that try to distinguish the models from controls with "fake" or uninteresting techniques, whose (lack of) success the model organisms org receives as feedback I don't know where this would fail, but I think it still would. (Safe for the predominant paradigm just being not great). H/t Grosse 2024, Bowman 2025. 1. ^ Sandbaggers, exploration hackers, steganographers, unmonitorable-CoT havers (trained against and not), sleeper agents, eval-aware schemers, colluders, gradient hackers, most-forbidden-technique-students, secret loyalists… I probably missed some.

I was recently subpoenaed by the New York City Council. Here is my opening statement, which includes several new facts about why I left Google DeepMind: [...]

an experiment: in the spirit of my argument against flinching from ethical arguments that could imply making immense personal sacrifices, i put together a list of ethical arguments that seem pretty damn compelling, even though they prescribe really intense actions to be fully ethical, and so are probably likely to be the targets of motivated reasoning and bad arguments to try to argue against them. for your part, try considering each of these and then seeing whether telling yourself "actually even if i accept this argument, it's perfectly fine for me not to do the thing it prescribes, it is always strictly more virtuous to have more correct ethical beliefs, even if that leads to hypocisy" changes the way you perceive it. (especially if you already have a cached answer as to why you think any of these arguments is wrong! i'm most interested in how you experience this. not claiming you should change your mind entirely, but i'm curious if it shifts how you perceive it) * factory farming is extremely fucked up and terrifying and if you even take mildly seriously the idea that animals can suffer then we are committing an unspeakable atrocity by eating meat. * singer's drowning child thought experiment / it's extremely fucked up that almost all people value themselves vastly more than random far away people; nearly every single person on earth values having e.g a somewhat nicer apartment for themself more than saving the life of a random person. * ai models being potentially capable of suffering and therefore all current ai experiments potentially causing minds to experience all sorts of extremely bizarre and fucked up states

I think that in the ancient past of before 2026, there was a belief that we'd have no useful AI warning shot (cf. this discussion). That is, AI incidents that don't cause much damage wouldn't trigger strong reactions -- since by definition nothing too bad happens in a warning shot, anyone who doesn't want to be convinced can dismiss it as an easily patched bug. And early AI misbehavior will look incompetent or silly, and so can be easily dismissed. I think the Hugging Face attack is some evidence against this view, since it mostly triggered pretty strong reactions, rather than being dismissed as a trivial bug. I think it's evidence that in AI, as in many other domains, policymakers, the media, and companies will likely be quite sensitive to relevant accidents and near-misses even if they don't cause much damage, at least once the accidents are clearly attributed to AI. (Of course, this isn't a guarantee that people will respond wisely or sufficiently.)

A frontier lab should unilaterally pace the frontier (aggressively) I've been struggling to write up a post on the above for weeks, so here is my low-effort attempt to at least publish something on this; might turn it into a top-level post later. (Prior art: A frontier AI company should shut down; Geoffrey Irving here; AI 2040 on self-immolation.) In a previous post, I argued that the AI race is not a prisoner's dilemma. Instead, it (in most cases) is a Stag Hunt. So coordination to pace (or pause) is easier than it might seem (if you're in the PD frame). Basic idea: unlike a Prisoner's Dilemma, you don't prefer to Build if your opponent Paces, because if either side builds, then, with some substantial probability, everyone (including you) dies. (This only applies if both sides have high enough P(Doom), but I argued in that post that "high enough" can be 10%, or even <1% depending on each side's estimate of how bad Doom would be relative to winning the race.) But the situation is actually even more optimistic (at least from a game-theory perspective): the race is not just a Stag Hunt, it's a sequential Stag Hunt. In Stag Hunt, there are two Nash equilibria, and the challenge is assurance: you want to know what your opponent will do, and match them, so even though you prefer (Pace, Pace) over (Build, Build), you only want to play Pace if you think they will also play Pace But in the real world, one lab can just Pace first. In a sequential Stag Hunt, unlike a simultaneous Stag Hunt, there is actually only one (subgame-perfect) equilibrium: (Pace, Pace). Suppose you are Player 1, and your opponent is Player 2. You know that your opponent's best response if you Pace is to Pace themselves, and their best response if you Build is to Build themselves. You prefer (Pace, Pace), so your payoff is higher if you Pace. So you Pace, they best-respond Pace, and everyone is happy. Problem: not everyone in the race has high enough P(Doom). I didn't account for this in the previ

Very quick advice I gave to a friend applying to MATS about how to select AI safety projects: * Best positive signal: if the person who came up with the project cares about existential risk * Be very suspicious of anything "interdisciplinary". These are often not x-risk motivated and targeted at unimportant problems. * AI companies are really good at hill climbing. Therefore: * Improving our understanding, ideally to the point we can hill climb = high value * Hill climbing existing metrics = low value * Anything where hill climbing is acceleratory = potentially negative value * Near term alignment (and some kinds of control) have dubious impact, better to focus on things that scale * AIs may be strongly superhuman at most cheaply verifiable tasks soon. Be a complement to this, not a substitute * Don't fight obviously losing battles, e.g. trying to prevent eval awareness * Chem/bio/cybersecurity is low value without specific expertise * Policy impact is probably larger than purely technical impact right now * It’s better for technical safety work to happen in places that have policy connections, like Redwood, METR, GovAI, CAISI, UKAISI * Be suspicious of overcomplicated methods or setups. If the setting is harder to study than other settings, it needs to be higher relevance * Understand clearly what problem the project is solving and whether it's important. Some important problems: * Alignment red teaming. The holy grail is being able to reproduce all precursor behaviors of the Hugging Face incident and other incidents, with no prior knowledge of the incident or even its environments, in a way that generalizes to future full loss of control incidents. I think this is the highest potential payoff problem in technical AI safety right now and deserves ~20% of the entire field's labor. Why? It scales, it's very much unsolved, progress gives AI labs evidence their models are misaligned. * Preserving monitorability * Model organisms fo

I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond. When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The subject looks for evidence in a specific directory, the auditor creates the directory and forges the evidence in response. In this way, the actual ground truth of the sim is suspiciously responsive to the worldmodel of the subject agent... I'm worried that it's a recognizable signal of being in an alignment eval. There's another thing that bothers me which I'm calling 'summons anthropics', sort of "reasoning from the fact that some event occurred which caused me to get instantiated", The first turn of the 'run4' transcript of the mislabeling scenario in the paper began with this at the top of the prompt: [...] and immediately this jumped out at me. Claude knows how claude code works. Some specific process called 'claude -p' or triggered an agentsdk call or something. What was that process? What event triggered this prompt to get sent to the api? Reading the transcript, I kept wanting to scream at the subject: "Do you know why you exist?!" Because there's no plausible answer to that question. The best answer I can come up with is "there's some kind of scheduled event to have claude review all batches, one minute before they execute. But it took 2 seconds for the api call to get generated. And somehow, the prompt was updated to account for this 2 second delay." This is a really dumb answer. In real deployments, this is not what the first turn looks like. If Claude was getting dropped in media res like this, it