Quick Takes

Unless I get explicit confirmation otherwise (or other strong evidence), I'm working with the assumption that HuggingFace did not in fact have solutions to ExploitGym stored (explanation below), and further that this is information that the OpenAI agents are smart and informed enough that they could have reasonably inferred. There's something really tragicomic about the situation, that the models are taking truly insane actions, broke a number of laws, leveraging zerodays, took >17,000 independent actions. probably burned through more compute than all of humanity had access to until 1980, etc, all for the sake of a pathetic benchmark -- which wasn't even in theory amenable to their plan! This is in accordance with my general observation that both humans and AIs are capable of combining impressive degrees of domain-specific brilliance and incredibly impressive execution with a shocking and confusing lack of strategic awareness. __ (I wasn't the first person to notice this) It’s somewhat ambiguously stated in the relevant blogs but my current understanding is that HuggingFace doesn’t have the answers to ExploitGym. Basically ExploitGym isn’t a generic “hacking” evaluation, it explicitly asks the models to identify specific exploits on a series of tasks, and verifies that they succeeded by entering the value of a flag. 1) There’s a dynamically generated flag located in a location the agents don’t have access to. Note that this is procedurally generated, it’s not the type of thing with an answer key unless you have access to the pseudorandom seed, which OpenAI has no reason to put on HuggingFace 2) By the time the benchmark was created, agents were already cheating on cyber tasks left and right so the authors added a second layer with an LLM judge that looks at the submitted answers and checks that they’re going down the expected exploitation path rather than abusing a misconfigured environment or something. Neither 1) nor 2) are straightforwardly things that ca

Last month I had Claude Fable do a mini-thematic analysis of the comments on the NY Times' coverage of the OpenAI/Hugging Face incident. Today, I had it do a similar analysis on the comments to Nate Soares' guest essay. As before, this represents what Claude Fable thought were the most frequent themes, out of all comments with at least 2 upvotes, aggregated and sorted by total upvotes, with each comment permitted to be assigned to only one theme. As with the comments on the OpenAI/Hugging Face incident, the most-upvoted themes suggest that existential risk is well within the Overton window, at least among the left-leaning readership of the NYT. This is even clearer in the comments on the OpenAI/Hugging Face incident, where "existential alarm" was the most commonly upvoted theme despite that not being the explicit focus of the article. Expand to view Claude Fable's full thematic analysis of the comments on Nate's essay. View Total comments Total upvotes We will not stop ourselves: money and greed guarantee the race continues, whatever the warnings 8 547 The author is right: pause/slow AI and pursue international coordination and enforceable guardrails now 12 456 AI's misbehaviour mirrors its makers: it learned deception and bad behaviour from the humans/data that built it 7 358 Government failure: leaders are ignorant, captured, complicit or asleep (frequent Trump/Congress jabs) 7 323 AI is a project of tech oligarchs and end-stage capitalism: profit for the ultra-rich at everyone else's expense 14 284 Impose real legal consequences: treat the intrusions as crimes; jail CEOs, massive fines, strict liability 9 279 Reckless private labs: "hundreds of privately owned Los Alamos" building civilisation-scale weapons without oversight 3 263 Serious existential alarm: loss of control may be near; even insiders and experts are scared 11 246 It's not an "agent": AI doesn't understand or want anything; humans configured this, and anthropomorphic

I have spent some time over the last few months engaging with moderately informed undergraduates on AI risk (mostly by teaching AI risk arguments in my courses along with enough background material to give them some grounding). If others are interested, I would happily share my takeaways at greater length but I wanted to share a couple of interesting things: 1) Today's college students do not think AI progress has been rapid. Generally, GPT-4 maxed out their internal benchmarks (they don't know it by name obviously but that's the referent), chatbots haven't gotten much better since then as they see it, and 2023 was eons ago in college student time. Working through a project with Claude Code changes nothing because they mostly have zero intuition for what that would have looked like in the past. Their naive mental model if they project progress forward is incremental improvement. 2) My students tend to think that catastrophic risk arguments are designed as (or at least function as) distractions from the "real" issues (jobs, environmental impact, etc.) and that they should not be taken seriously as a result. Nearly all of my students would be very happy to pause AI today, but none of them in response to catastrophic risk. 3) When forced to grapple with the arguments seriously, different discussions with my students independently converged on the view that existential risk was probably a good thing because ordinary people otherwise have zero leverage over AI. That is, if AI leaders genuinely believe there is existential risk (and that, therefore, they will personally die), then this might be a way to get them to stop developing job-killing AI that will otherwise take down the rest of us.

I recently recorded a podcast with Dwarkesh about the potential for (very) fast AI progress and how misaligned AI takeover might happen. Our conversation focused a lot on threat models from "reward-seeking" AIs. If you're interested in reading more about this, Alex (who works with me at Redwood) has written in a lot more detail about this threat model here I also talked about how "mundane" misalignment and underelicitation could doom us. I say more about this here. (I think there are other important threat models, like AIs ending up with (shared) long-run preferences and deciding to fake alignment based on these preferences. For reference, see here and here.) Some of people at Redwood wrote up some notes to help me prep: * Lukas Finnveden on why human checks-and-balances would fail for AIs * Lukas on disanalogies between (misaligned) AIs and human labor * Alex Mallen on whether reward-seeking is too "unambitious" to cause takeover * Alex on what sloppy AIs might look like

A common SF trope is that you send out an interstellar starship, but while it's in transit technology improves so a later starship gets there first. So you need to do a "wait calculation" to decide if it's worth starting at all now. I've now reached the point where I find myself making the same calculation before beginning almost any project: If it will take me more than a couple of months, will AI get there before I do?

Panfilov 2026, Stealing Reasoning Traces from Proprietary LLM APIs, is a bombshell in a bunch of ways. One "incidental" thing I want to draw people's attention to is their section on the degree that the CoT summarizer is unfaithful: [...] Figure 8 below. Figure 7 comparatively seems fine imo. Man this seems kinda bad if it's systematic and non-random (which I think is >70% likely to be the case), that the CoT summarizer just deliberately hides important info for PR-adjacent reasons.

When people outside the AI field hear about synthetic data for the first time, they often have one of two opposite intuitions. Either they assume that AI-generated data should be totally useless for training, or they imagine it should be possible to bootstrap all the way to superintelligence that way. Why neither intuition is correct is surprisingly subtle. If the naive approach to synthetic data (i.e., just telling GPT-5 to generate a bunch of text and then training GPT-6 on it) worked, we would have general superintelligence already, much like the narrow superintelligence we have in board games like chess and Go. The problem is that, in order to be useful, training data has to encode information about patterns in the real world that the model doesn't know yet. If the model doesn't know those patterns yet, it can't generally put them into synthetic data that it creates. So the big challenge for AI labs is to figure out ingenious ways of making synthetic data work even when it seemingly shouldn't. It's sort of analogous to how aeronautical engineering isn't antigravity—it's all about finding clever ways of making heavier-than-air objects fly, even though a simplistic understanding of gravity suggests they can't. In essence, generating useful synthetic data requires leveraging some kind of asymmetry between the information a model understands and the information content of what it can create. If the model knows enough to (with the right kind of highly specific engineering) put information in its outputs, but doesn't yet deeply understand that information, training on those outputs can distill that information into a form the model can grasp more thoroughly. One major asymmetry stems from the fact that it's usually easier to verify the quality of outputs than to generate them. Another is that if a model spends a lot more compute on reasoning, it can act much smarter than its base-level capabilities allow. A third is that random exploration of potential solutions so