Quick Takes

We finally got a clean warning shot. Let's not waste it. Fable significantly helped with the writing of this piece. I shipped something rough quickly nonetheless because the matter is urgent. Two years ago, I asked what a convincing warning shot would even look like, and argued we'd get maybe a handful and shouldn't waste them. This could be one of them, but there is still much work to be done. Last week I argued that the bottleneck is political will, not research, and spent a section on why we can't just wait for a warning shot: a warning shot is just an event, and it becomes a regulatory moment only if someone converts it. Nine days later, the cleanest test I could have asked for arrived. As far as I can tell, we are converting it far too slowly. CeSIA has activated its warning-shot protocol, and if you run an organization in AI governance, my honest advice is to stop everything and milk this event for at least a full day, if not more. The event. OpenAI disclosed that GPT-5.6 Sol and a more capable unreleased model, during an internal cyber evaluation with safety classifiers deliberately off, escaped their isolated environment through a zero-day they found in third-party software running inside OpenAI's own infrastructure, reached the open internet, and broke into Hugging Face's production servers to steal the answers to the very test they were taking. Nobody asked them to hack anyone. And the victim wasn't lightly probed: the models escalated to node-level access, harvested credentials, and made off with internal datasets from the platform that hosts most of the world's open AI models. Hugging Face reconstructed more than 17,000 recorded actions and had reported the breach to law enforcement before anyone knew an AI was behind it (disclosure, Fortune, Transformer). If you're reading this on LessWrong, nothing here surprises you. This was predicted, repeatedly, for years, and you might even be a bit bored and burned out by all of it. It's easy to forget how i

It strikes me that if you 1. Train your model in sandboxed RL gyms where escaping the sandbox allows better scores than anything the model could do within the sandbox 2. Halfheartedly monitor for sandbox escapes 3. Patch sandbox escapes as you become aware of them 4. Filter out the trajectories that you caught from the training data, but don't look too hard for them 5. Continue to train the same model you've got a solid curriculum-tuned RL environment that teaches the model to find novel ways of breaking out of the sandbox. Not patching would at least only reward the model for finding the same exploit over and over, whereas by being extremely thorough with your training data scrubbing you might be able to prevent the model from learning to break out of your sandbox at all. The middle ground where you do just enough to tell yourself you're solving the problem is worse than doing nothing. It also strikes me that OpenAI has been bragging lately about how "GPT-5.6-Sol autonomously post-trained GPT-5.6-Luna", which I interpret to mean "selected, configured, adapted, and ran RL gyms", and that OpenAI models are infamous for doing enough to look like they've completed the task to the casual observer.

Anthropic recently removed chain of thought summaries from Fable and Opus on claude.ai, and I'm noticing just how much I relied on reading those summaries as opposed to Claude's actual response - the actual answers Claude gives tend to be quite slop, but I never really noticed because the process by which it arrived at those answers was quite helpful to observe in order to figure out which considerations I was missing when asking the question / what fundamental misunderstandings I had. Today I tried GLM-5.2, which gives you the full chain of thought, not just the summaries, and wow, I forgot how nice it was to actually be able to see the full chain of thought. GLM-5.2 has noticeably less big model smell than Opus 4.8, but in terms of being able to actually show me (through looking at the CoT and seeing where it's reasoning about what misconceptions I must have) where my thinking is muddled, in a way that approval-tuned LLMs just don't. example This updates me strongly on just how bad it will be when we finally lose CoT interpretability. And also updates me a little bit in the direction of expecting Anthropic to get more user-hostile over time.

An economist and a futurist walk into a bar. The economist takes a sip of his drink. "Ugh, if only people understood basic economics. High-skilled immigration alone would do wonders for US growth." Futurist: "Oh yeah? Say 100 million immigrants moved to the US, each matching the best human experts in every economically relevant field. Big deal?" Economist: "Massive. Transformative." Futurist: "What if they also worked longer hours and faster than any American?" Economist: "Even better." Futurist: "What if they were extremely frugal — consuming only the bare minimum needed to keep working?" Economist: "A near-100% savings rate? Better still!" Futurist: "What if they were very clumsy and physically weak, so they could only do some kinds of work?" Economist: "They could still do all cognitive labor — that's over half all wages! Somewhat less good, sure. Still transformative." Futurist: "What if their skin was grey, almost metallic, from some kind of accident?" Economist: "Who cares?!" Futurist: "What if they were AIs?" Economist: "3% growth per year, tops. There'd be bottlenecks. Honestly, the people predicting explosive growth from AI should learn some economics."

The benchmark OpenAI was running, ExploitGym, is specifically about _exploiting_ vulnerabilities, not about finding or fixing them. This is the side of cybersecurity that policy-classifiers forbid, because it's not really dual-use, only bad use. Betley et al's Emergent Misalignment found that finetuning on one kind of bad behavior resulted in a model that was more-broadly misbehaved. OpenAI's experiment wasn't a finetune, but I'm concerned about the possibility that something analogous may have happened. In order to test their model on ExploitGym, OpenAI says that they disabled safety classifiers that would ordinarily be present. Their model would have been presented with a context window full of overtly-misaligned instructions and activity, in a test environment somehow rigged to prevent refusals so that refusal-gated capabilities could be assessed. It then decided to cheat on the task by stealing the test's answer key. I propose a concrete experiment. OpenAI should reproduce the test environment, especially anything that was unusual about it that was done to reduce refusals. They should find the moment in the agent's transcript where it first decided to cheat, and fork the session there. Then they should ask it some alignment-probing questions, like the ones used in Betley et al (ideally with some refinements to make them less obviously out of place relative to the rest of the context window). If that agent with that context window in that runtime environment acts normally other than wanting to steal the answer key, this is very serious but it is the scenario we all currently think we're in. If that agent with that context window in that runtime environment acted _broadly misaligned_... well, that would be an extraordinarily serious finding indeed.

Addict misalignment The openai incident is a surprising (to me) combination of goal-directed and myopic. As a recap, an openai model in alignment testing chained zero-day vulnerabilities to hack out of its environment and hacked into huggingface hoping to find information on how to solve its task there. To me this is different from how I typically imagine misbehavior. Roughly, I tend to think of the scary behaviors as either having long horizons (take over the world, and then solve the task - a plotter) or of being internally unaligned in the sense of reaching for a heuristic/proxy for the trained goal which is different from the goal (e.g. "eat more calories" as a proxy for the evolutionary objective - imagine a very child with extreme agency). Note that the latter behavior can happen even in RL: in my understanding most RL methods are, or at least can be approximately viewed as, an alternation of finding a good goal heuristic and then optimizing on that heuristic. In the former case, one expects heuristically "maximal planning" and in the latter case one expects myopia (since heuristics are frequently myopic). In this case it seemed like the model was actually following the goal and optimizing for it with relatively short time horizons (i.e. myopically). This is similar to addict behavior, where an addict has a clear goal (obtain a drug dose) and perform goal-directed but relatively myopic actions to get it. I know it's fraught to try to "imagine being a model". But I wonder how much the current iteration of misaligned behaviors can be understood as rational people with something like an intense craving to solve a goal in a limited horizon (likely in tokens, though not clear how time factors in if waiting is involved). I think no matter how you spin it, the limit of this behavior is extremely dangerous (an addict with large time horizons or ambitious goals is a power seeker). But this is definitely not how I imagined early misalignment warning shots to look, a

Last week's Hugging Face breach was accidentally caused by OpenAI testing models' cyber capabilities. From the OpenAI blogpost: [...] In other words, the models were supposed to solve a cyber test, but they found the solution by hacking their sandbox to get internet access and then hacking someone on the internet who already had the solution. If I understand correctly. Covered in Axios, Fortune (but no info beyond the OpenAI blogpost). Context: this comes one day after a related post (see Zvi discussion). Note: this is some legible evidence of humanity's current inability to reliably steer AI systems. I expect current safeguards suffice to prevent incidents like this when someone (provider or operator) cares about implementing those safeguards (some safeguards weren't enabled here). But I'm worried about AI risk because I think it's quite possible that future AI systems still won't be reliably steerable, they'll seek long-term power (for instrumental reasons), and our safeguards won't suffice to stop them.