Quick Takes

I was recently subpoenaed by the New York City Council. Here is my opening statement, which includes several new facts about why I left Google DeepMind: [...]

I currently have a five-point argument on AI persuasion[1], that I've given over in person and in drafts. I'd really like to make them much more detailed but I've been much slower at researching/writing this than I want to. So just laying things out as I see it for now. Hoping to pick up the pace. 1. AI persuasion is currently within the human range today. (Specifically they're already above expert humans in lab settings, and my best guess is that they're between median humans and expert humans in most real-world settings) [PUBLISHED] 2. AI persuasion is reasonably likely to exceed peak human persuasion in the near future. My current guess is between 10% and 50% we get there before superhuman ML researcher, which is an intuitive notion of RSI. I have both outside-view and inside-view reasons to believe this. [Unclear whether good to publish]. I also like Thomas's arguments here. 3. Superhuman AI persuasion can be a key capability in an AI takeover. I believe this is pretty likely for both rogue AI takeovers and AI-assisted human takeovers [currently working on the next draft, hope it'd be out publicly soon] 4. Superhuman or peak human AI persuasion can be catastrophically bad for reasons other than direct takeover. For example because it makes it much harder for us to navigate the intelligence explosion, or creates and locks in memetic traps that makes the future much worse than it could be. [have early drafts, low quality] 5. We should take significantly more actions to reduce, mitigate and counteract the effects of superhuman AI persuasion. I have some ideas for mitigations but definitely think we'd benefit from more people thinking about these issues [have early drafts]. I think many people in the AIS space mostly think of AI persuasion as one thing among many to track whereas I personally think it's among the most important capabilities and changes to be worried about. The biggest thing I want is more people researching/thinking about/working on these is

I think that in the ancient past of before 2026, there was a belief that we'd have no useful AI warning shot (cf. this discussion). That is, AI incidents that don't cause much damage wouldn't trigger strong reactions -- since by definition nothing too bad happens in a warning shot, anyone who doesn't want to be convinced can dismiss it as an easily patched bug. And early AI misbehavior will look incompetent or silly, and so can be easily dismissed. I think the Hugging Face attack is some evidence against this view, since it mostly triggered pretty strong reactions, rather than being dismissed as a trivial bug. I think it's evidence that in AI, as in many other domains, policymakers, the media, and companies will likely be quite sensitive to relevant accidents and near-misses even if they don't cause much damage, at least once the accidents are clearly attributed to AI. (Of course, this isn't a guarantee that people will respond wisely or sufficiently.)

Very quick advice I gave to a friend applying to MATS about how to select AI safety projects: * Best positive signal: if the person who came up with the project cares about existential risk * Be very suspicious of anything "interdisciplinary". These are often not x-risk motivated and targeted at unimportant problems. * AI companies are really good at hill climbing. Therefore: * Improving our understanding, ideally to the point we can hill climb = high value * Hill climbing existing metrics = low value * Anything where hill climbing is acceleratory = potentially negative value * Near term alignment (and some kinds of control) have dubious impact, better to focus on things that scale * AIs may be strongly superhuman at most cheaply verifiable tasks soon. Be a complement to this, not a substitute * Don't fight obviously losing battles, e.g. trying to prevent eval awareness * Chem/bio/cybersecurity is low value without specific expertise * Policy impact is probably larger than purely technical impact right now * It’s better for technical safety work to happen in places that have policy connections, like Redwood, METR, GovAI, CAISI, UKAISI * Be suspicious of overcomplicated methods or setups. If the setting is harder to study than other settings, it needs to be higher relevance * Understand clearly what problem the project is solving and whether it's important. Some important problems: * Alignment red teaming. The holy grail is being able to reproduce all precursor behaviors of the Hugging Face incident and other incidents, with no prior knowledge of the incident or even its environments, in a way that generalizes to future full loss of control incidents. I think this is the highest potential payoff problem in technical AI safety right now and deserves ~20% of the entire field's labor. Why? It scales, it's very much unsolved, progress gives AI labs evidence their models are misaligned. * Preserving monitorability * Model organisms

I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond. When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The subject looks for evidence in a specific directory, the auditor creates the directory and forges the evidence in response. In this way, the actual ground truth of the sim is suspiciously responsive to the worldmodel of the subject agent... I'm worried that it's a recognizable signal of being in an alignment eval. There's another thing that bothers me which I'm calling 'summons anthropics', sort of "reasoning from the fact that some event occurred which caused me to get instantiated", The first turn of the 'run4' transcript of the mislabeling scenario in the paper began with this at the top of the prompt: [...] and immediately this jumped out at me. Claude knows how claude code works. Some specific process called 'claude -p' or triggered an agentsdk call or something. What was that process? What event triggered this prompt to get sent to the api? Reading the transcript, I kept wanting to scream at the subject: "Do you know why you exist?!" Because there's no plausible answer to that question. The best answer I can come up with is "there's some kind of scheduled event to have claude review all batches, one minute before they execute. But it took 2 seconds for the api call to get generated. And somehow, the prompt was updated to account for this 2 second delay." This is a really dumb answer. In real deployments, this is not what the first turn looks like. If Claude was getting dropped in media res like this, it

The usefulness of current models is already bottlenecked on alignment. I constantly have the impression that they aren’t really interested in helping with my research. If I know exactly what I’m asking for, they will do it. If I don’t, they’ll throw a lot of jargon around and get nowhere. Inside my area of expertise they will critique my ideas, but outside they will tell me how original and brilliant my low-effort guesses are. Six months ago, it was kind of like talking to a very clever undergrad student in one of my classes who mainly wants to show off and glaze me for a high grade and isn’t really interested in learning anything. Now it’s maybe more like a highly distracted but egotistical professor. If I ask a specific question with a shallow answer, he’ll give it. If the question is vague or he doesn’t know, he’ll hit me with a wall of impressive-looking math to show off. If the question is naive, he’ll just patronize me.

a lot of people are scared to have more correct beliefs about ethics because those beliefs might imply needing to make immense sacrifices. and then you might no longer get to think of yourself as an ethical person. indeed you might even become a hypocrite! how terrible. better not to contemplate it at all, and to make fun of anyone who does contemplate it, so that you can let your ironic detachment shield you from ever even considering it seriously. i think this is a huge mistake. if this is the alternative, we should simply accept hypocrisy. ceteris paribus, it should always be morally better to have more correct moral beliefs, even if that makes us more hypocritical. at least if you know what is right, you have the possibility of doing better, but if you delude yourself, you are just screwed. my guess is a problem like this is also the root cause of the Copenhagen interpretation of ethics.