Quick Takes

If you look at frontier ECI over time, there was a clear trend break in mid-to-late 2024 around the introduction of reasoning models. The slope increased by ~2x around this point[1], while being roughly constant before and after it. If you then break things down by each lab's individual frontier, you'll see that the overall industry frontier's acceleration was really a manifestation of a more specific phenomenon: an acceleration for OpenAI models. OpenAI's acceleration allowed them to overtake others and dominate the frontier for a while after the trend break, which made "the frontier trend" and "the OpenAI trend" approximately the same thing over that interval. Most notably, no such speedup in 2024 is visible for Anthropic models. Anthropic's frontier did experience its own trend break, but only later on, in late 2025. This is visible in ordinary ECI, you don't need AECI or internal models to see it[2]. As far as we can see from the sparse/noisy data, it is similar to the trend break that OpenAI experienced in 2024: sudden ~2x slope increase, roughly linear before and after that. Since Anthropic's pre-break slope had been a bit higher than OpenAI's pre-break slope, the same holds true for the two post-break slopes, and so Anthropic is now moving faster than OpenAI. That's allowed them to catch up, and now they've "become the frontier" in the same way that OpenAI "was the frontier" before. I remember people freaking out (understandably) about the 2024 trend break once it became clear it had happened. And now, people are freaking out about the separate 2025 trend break, attributing it to coding agent assistance with AI development (an explanation no one ever proposed for its earlier sibling, since agents back then were just not that good yet). But something doesn't add up about this... it's not as though "the frontier" sped up twice, and is now 4x as fast as the original trend! Instead, each of these two labs experienced a single ~2x speedup, at different point

I feel a bit crazy in that the recent METR report is explicitly saying that the measured progress is on well-specified tasks and that they cannot tell how much of this transfers to ill-specified tasks; and now half the blogs in AI safety sphere seem to be taking this as "ASI is imminent", dropping the caveat. In METR's words: [...] If we assume that humans are capable of solving ill-specified tasks, then at least by a "better than humans at any cognitive task" definition of ASI, ASI would need to be capable of solving ill-specified tasks. And "progress on well-specified tasks is accelerating" does not automatically translate to "soon we will understand how to solve ill-specified tasks". I do think that it makes sense to assume that progress on well-specified tasks increases the odds of making progress on the rest of them as well and that we can't necessarily assume that the acceleration stays restricted to well-specified tasks, but that isn't the same as "we can put this trend on a graph which tells us that ASI will be here in a year". Earlier related comment.

Agent-Foundations Research Can Still Save Us, Even In Short-Timeline Worlds I'm getting the feeling that the recent updates towards short timelines are coupled with updates towards considering working on hard theoretical problems of alignment even less promising than before. I don't think this is correct. Historically, highly theoretical research had no short-term impact, because the period of crossing the theory-practice gap was measured in years. If that were still the case, then yes, short timelines would mean there'd be no time to implement any potential philosophical/theoretical breakthroughs even if they were found. However, that's precisely what LLMs themselves change: * They can, or will soon become, superhumanly fast at resolving arbitrary formally stated conjectures. This means that it would be possible to rapidly formalize any conceptual framework that you've made sufficient philosophical progress on. Once you're "in the neighbourhood of" the correct formalism, you'd be able to rapidly explore it by spamming possible conjecture formulations at LLMs, finding ones that work out, then hashing out the rest of it using those as an "anchor". * They are, or will soon become, superhuman at arbitrary "workmanlike" math. This likely includes finding computationally tractable but formally valid approximations of high-level algorithms, comparing various alternate implementations of such algorithms for efficiency, et cetera. * They are, or will soon become, superhuman at arbitrary software-engineering tasks. Basically, if we assume that they'd be able to dramatically accelerate in-paradigm LLM research, that means they'd be able to dramatically accelerate any in-paradigm research. So once a paradigm is found – once the very basic high-level foundational theory is nailed down – LLMs would be able to take it the rest of the way to highly efficient practical implementations, down to high-performance scalable code. "Dual-use capabilities" cut both ways. (Conversel

Sometimes people from middle powers ask me what middle powers can do to help the AI situation. Recently for example I was asked this by an Australian. This is what I said: Main tool Australia and other middle power governments have is their voice. They should say loudly "We used to be dismissive of talk of superintelligence and associated risks, but now based on the accumulation of evidence we think that these things are all too serious as possibilities and could really happen in the next year or three. We consider this a threat to our sovereignty, livelihood, and even our lives, and will act accordingly. We are hereby [costly signals of seriousness e.g. declaring a national emergency, calling up the reserves, distributing PPE equipment to every citizen, banning AI companies from operating in Australia unless they allow third-party assessments of risk and in particular allow UK and AUS AISI's access to investigate incidents like Hugging Face] effective immediately. Also, we are hosting a summit in 72 hours for world leaders to come discuss the risks and benefits of recursively self-improving superintelligent AI, and whether and how AI development should be allowed to proceed." The reason this is so important is that right now, even in the US, most citizens do not want superintelligence or RSI to happen. Unlike many political issues, this political issue is one where a supermajority would agree on what needs to be done (~pause or otherwise heavily regulate AI development immediately) if only they understood that this is a real issue and not sci-fi. Having an actual government stop acting like it's sci-fi and start acting like it's real can cause a cascading effect where a bunch of other people will see that and then start talking and acting like it's real too, which then wakes up others, etc. until all of a sudden the discourse has shifted from "ha ha if it's so risky why are they still building it ha ha" to "OK so obviously they need to stop immediately, let's do

Jacob Coxon: [...] I'm not sure what to make of this. My view was something like most algorithmic progress is data progress, and also that most of the progress this year (excluding compute growth) was from better post-training. This reply from Jacob makes it sound like he isn't talking about better data (emphasis mine): [...] Data progress seems like it'd be much more continuous, but someone discovering a better attention variant or optimizer totally fits here. I don't think this matters much for short-term progress (just extrapolate your preferred progress line linearly) but it seems extremely relevant for RSI and progress over the next couple years. Like the more diverse AI progress is, the less one source hitting diminishing returns would matter. For AI automation specifically, I expect there to be big differences in how much they can accelerate each source. Even if they were exactly (expert) human level at every skill, the impact of 10x, 1,000x, or 1,000,000x as much labor seems quite uncertain.

The last few days, David and I did the same exercise which IIUC produced AI 2027: predict the next month, condition on those predictions, predict the month after that, etc. We assumed that: * there is no secret sauce of intelligence missing from today's LLM-based AIs * METR's "time horizon" metric is a basically reasonable way to think about LLM capabilities Besides just running forward median predictions, for each month we also came up with a list of potential surprises, of a similar flavor to "news breaks that Jensen Huang had an affair with a Chinese spy" or "OpenAI releases a proof that the Exponential Time Hypothesis is false", and we randomly rolled a couple of those for each month. The random rolls came out moderately worse than I'd actually expect, even conditional on the assumptions. But they weren't that far out of distribution, and... We died in January. Two months from now. Regulations hadn't even been passed, let alone implemented; if the government was't already enforcing a full Pause in December they didn't matter at all. AIs were starting to be able to run profitable businesses, which made it possible for the first time for AIs to exfiltrate and operate fully independently online; that also didn't matter at all. There was some crazy stuff going on with AI company finances, and none of it mattered at all. The Hackening hadn't even gotten seriously underway yet. The only thing which mattered was a single random event: "OpenAI announces that their internal automated AI research swarm has found a way to improve the scaling power-law coefficient on data by 20%.". That wasn't written to be a "we die" event, just a plausible sounding algorithmic advancement. When we ran the numbers, it was enough for immediate supercritical RSI.

working with agents feels like managing a team of like 5-10 freakishly fast interns who have shockingly deep knowledge about a whole bunch of things like gpu programming or math, and who somehow still need basic guidance every hour or so to not make terrible decisions. it's pretty easy for me to spend an entire day just tabbing between my agents and redirecting them when they are doing something weird or getting stuck on something. the more deeply i understand something and the more actively i interpose myself on decision making, the better outcomes are. if i task the agents with doing some very large thing that i have no idea how to do, i will get a horrific +100k line diff that breaks all the time in mysterious undebuggable ways, until at some point it all collapses in on itself and nothing ever works anymore and i need to throw the code away. if i design the code carefully and just use the agent to execute on a very specific vision of code i have in my head, the result is much more robust and successful, even though it actually doesn't even require that much of my time to design it and then answer a bunch of questions. it feels really weird to be experiencing this brief interstitial time where it is basically completely pointless to do things by hand, and yet the rsi loop hasn't been closed yet, and so there is this job of agent herding that is absurdly productive but also very strangely shaped.