Quick Takes

Between Sep 2025 and Sep 2026 was likely the fastest year of total param scaling (for properly RLVRed models) that LLMs will ever see. In Sep 2025, Opus 4.5 and Gemini 3 Pro weren't yet out, GPT-5.0 was GPT-4o with RLVR (maybe 600B total params), Sonnet 4.5 was the latest thing. In Sep 2026, we have Fable 5 and Astra 6, probably 10T-20T total params, and the next OpenAI model might be targeting Rubin, in which case it could be 40T-60T total params. Opus 5.5 being surprisingly capable weakly suggests Anthropic might also have a post-Fable model getting ready for next year's hardware. Very approximately, the shape of scaling is that 2T total param models were possible in 2025, 20T in 2026, 200T in 2028, and 2,000T in 2032 (which isn't too expensive to serve). It took 1 year to 10x the params in 2025-2026, it'll take 2 years to do the same in 2026-2028, then 4 years to do so yet again in 2028-2032. The 1 year of 2025-2026 saw the transition from Sonnet to Opus to Fable. A similar amount of qualitative progress might happen in the 2 years ending in 2028, and then in the 4 years ending in 2032. Without an algorithmic breakthrough like continual learning in a strong sense, this could be literally what happens. But the results of 2026 suggest LLMs might themselves be capable of inventing continual learning soon, breaking their paradigm. Probably this happens by 2028, if it happens at all in the current regime. So in addition to the background risk of a paradigm-breaking innovation driven by human effort, there's an unusual concentration of risk around 2027-2028, when the amnesiac slow-learning LLMs become superhumanly smart and the difficulty of continual learning is tested for the first time against their uneven capabilities.

In an Open Letter to Scott Alexander, Stephen Pinker claims RAND [1] found there is "no describable scenario in which an AI could conclusively pose an extinction threat to humanity." This is at best sloppy research, and at worst actively misleading, as Cody Fenwick points out that Rand's conclusions significantly differed from what Pinker claimed. RAND actually said AI plausibly could wipe out humanity in 2 of 3 scenarios they looked at, though they couldn't definitively assert that any of the three scenarios is likely or unlikely. I can see where Pinker got his claim--Rand did not show there is obviously a describable scenario in which an AI wipes out humanity. But: 1) Rand didn't search very hard 2) RAND certainly doesn't find it implausible that AI wipes out humanity, which is what Pinker is trying to argue against. Overall, I continue to be disappointed by Pinker's arguments. [1] https://www.rand.org/content/dam/rand/pubs/research_reports/RRA3000/RRA3034-1/RAND_RRA3034-1.pdf

I see some people predicting or tacitly assuming that rogue AI incidents will continue get worse and more frequent, but I think it's reasonably likely that we're currently at (or already past) the peak. I definitely wouldn't count on seeing more incidents and "warning shots" than we've already gotten to shift public perception or influence policymakers. I expect that labs and other players will respond to recent incidents by taking traditional cybersecurity, monitoring, and surface-level alignment more seriously, and that this will basically work. Deliberate misuse from open-source models, jailbreaks, and bad actors will continue to be a growing problem, but a manageable one. I expect the old "d/acc" thesis will basically hold: there will be some adjustments and bumpiness along the way, and there will be lots of work to do, but enough human (and agent) capital will go into proactive + defensive cybersecurity that there won't be major damages to life and property. I think frontier AIs will continue to get smarter, more helpful and (apparently) more aligned, until they are smart and powerful enough to take over decisively and swiftly. Either way, anyone who is pushing for public policy and controls to address short-term and non-extinction-level harms from AI should probably have a plan for what they'll do if whatever harms they are warning about don't actually come to pass.

Agent foundations researchers are religious about curiosity. We culturally love to discover truth and come up with clever ideas. That's how someone ends up thinking about thinking enough to be an AF researcher instead of thinking about chemistry enough to be a chemist. Through recursive self-improvement, this lets us see certain things more clearly, which reinforces very highly valuing curiosity, even to the point of idolatry. Curiosity is a useful heuristic, but it's tuned to find the discoveries that you are actually looking for. Allowing curiosity to select your research topic is a feedback process that can end up anywhere. The space of potential discoveries is vast enough that this is very unlikely to solve any particular problem chosen in advance, such as AI safety, unless you actually pursue that goal directly at least often enough to ground your curiosity-tuning-process.

From the OpenAI post on research acceleration. A bunch of good data in there. This graph in particular caught my attention -- it seems to be measuring "In-practice horizon length" on real-world coding tasks happening at OpenAI. The in-practice horizon length of OpenAI models seems to have gone from around 1-2h at the beginning of the year to something like 8h in July. So that's 2-3 doublings in half a year. Also, the in-practice horizon length seems to be a bit less than the METR-measured horizon length, which was something like 5h at the start of the year (and is now unknown due to running out of long METR tasks).

The emotional valence of social interactions is often the opposite of what one would naively assume: * when Bob pays for something for Alice, one might think Alice feels joy and gratitude for getting something for free while Bob feels a bit of pain for losing money. But in practice this is often somewhat reversed, with Bob feeling the warm glow of having done something nice for another person, while Alice may feel a bit guilty and make a mental note to pay this back in the future. * when Alice offers to take Bob somewhere with the car without her having to go to that place, one might think Bob will primarily be happy about the lift while Alice, while possibly doing this out of the goodness of her heart, suffers a little from the inconvenience. Yet it's likely Alice will feel uplifted by the interaction, while Bob feels hesitant or even bad about wasting her time. Of course, this is not always the case. And ideally, such social interactions are positive-sum and both parties benefit in some sense. Still, I find it interesting that the superficial interpretation is often incorrect. E.g., it's fun and feels good to buy friends a drink when going out, but depending on the nature of the friendship and the friends' personalities, it's possible they would actually prefer paying for themselves and now feel a bit conflicted about it.

OpenAI has again stopped all RL runs for their most capable models, after an internal research model broke out of its sandbox: [...]