I found the Dwarkesh - Terrence Tao conversation very frustrating, in particular the section on advancements in science, re heliocentrism and evolution. These guys talked about it epistemically as if everyone is a scientist. And in response to Dwarkesh's question about why Darwin's ideas took 200 years longer than Newton's, even though they were simpler, Tao basically boiled it down to science communication. What?
The biggest causal impediment to the acceptance of both theories was religion and religious thought. It's not even close. Darwin delayed publication for years because he was terrified of the societal and cultural blowback. Evolution dissolves the need for an interventionist creator and lumps us in with the rest of nature. It frames our ancestors as simple creatures akin to worms and bacteria. All the eloquence in the world isn't going to overcome the entrenchment of the prevailing religious worldview. Ridiculous.
All three of the big US labs have recently accused various Chinese labs of large-scale covert distillation of their models, presenting evidence that the labs in question have been using thousands of fraudulent accounts to cover their tracks.
In the same way that models have learned to detect whether they are being given a real task or a training scenario and alter their responses, is it likely they will do the same for this scenario?
This would be fascinating, as would their responses. Especially if the models respond in different ways.
I'd guess that's pretty hard since the distillation attacks are intentionally spread across many accounts and conversations, and mixed in with innocuous requests. But on the other hand, the models are getting quite good at noticing subtle tells that humans would miss.
I suppose another question is: if you're a big lab and you detect a distillation attack, do you block it or do you quietly feed it poisoned responses?
Nobody said the path would be clear. We know we need to prepare for AGI, but how do we do that if we don’t know whether it’s coming in 3 years or in 100? What about recursive self improvement: will that escalate to superintelligence, or fizzle out? And as the White House starts laying out its legislative agenda for AI, should we push for government leadership on existential risk, or merely hope they stay out of the way while we do the heavy lifting?
Top pick
Broad Timelines
Toby Ord reviews some of the best-known AGI timelines and concludes that we should prepare for a wide range of possibilities (his 80% probability range is from 3 to 100 years). What does that imply for people who want to work on AI safety—should you rush to have the most impact right away, or invest in building capacity to have more impact later?
This is exactly correct: the AI future is high variance, and it isn’t enough to have a plan that will work great if everything plays out exactly the way you expect. We need a portfolio of plans and projects that will work in a wide range of possible futures.
See also Oscar Delany’s piece on the same topic.
My writing
Contra Anil Seth on AI Consciousness
Biological naturalists argue that consciousness is tightly coupled to details of human neurobiology, making it unlikely that AI will achieve consciousness in the foreseeable future. I examine the arguments put forward by a leading biological naturalist and find them unconvincing.
New releases
Cursor Composer 2
Cursor’s Composer coding agent is a fascinating outlier in the AI world—it’s made by a relatively small company, but punches way above its weight. Composer 2 just came out, claiming some impressive benchmark results.
Composer is a capable agent with generous usage limits: if I were coding on a tight budget, I’d seriously consider making it my daily driver. But for anyone who can afford them, Opus and Codex still seem like better options.
During the launch, Cursor revealed—apparently by accident—that Composer is built on top of Kimi K2.5. They performed significant training on top of the base model, but I’m taking this as an important data point about what the best open models can achieve with a modest amount of additional training and scaffolding.
GPT 5.4 is a big step for Codex
Nathan Lambert reviews GPT 5.4 in Codex, with a focus on how it compares to Opus in Claude Code. He agrees with others that it’s a big step forward on multiple dimensions, making it again a serious competitor (although he still prefers Claude, for intangible reasons). I concur: GPT is extremely capable, but I get more done with Claude.
Capabilities and timelines
Do we already have AGI?
Even though its meaning has drifted, AGI remains a useful anchoring concept. Benjamin Todd bravely wades into the debate about what it actually means, bringing welcome rigor and clarity. He pulls together four of the most useful definitions of AGI and concludes that current AI doesn’t meet any of them:
Lossy self-improvement
Many people (including me) believe we’re probably close to recursive self improvement, which will rapidly lead to superhuman AI. Nathan Lambert disagrees:
This is the most detailed and persuasive argument I’ve seen for why RSI might not lead to an intelligence explosion. My money is still on RSI, but there’s a non-trivial chance that Nathan is right and the friction is too great for a fast takeoff.
Benchmarks and forecasts
Terence Tao and Dwarkesh talk about math and science
Dwarkesh interviews Terence Tao—obviously it’s great. Come for the status report on AI doing research-level math, stay for the discussion of Johannes Kepler and the process of scientific discovery.
I’m struck by some of the similarities between math and coding. In both cases, there’s a massive speedup in doing much of the work that we used to do, but it’s unclear exactly how that translates to overall productivity:
Alignment and interpretability
No, AI alignment isn’t solved
There’s a common belief that alignment might be easier than we once expected: LLMs are unexpectedly good at generalizing and understanding human values, and current alignment techniques work surprisingly well. Transformer’s Lynette Bye reports on some reasons for optimism, and reminds us that we still have a lot of work to do:
How we monitor internal coding agents for misalignment
The most capable (and potentially most dangerous) models at any point in time are in pre-deployment inside the big labs. That capability gap is likely to widen as we approach AGI, especially if the labs choose to reserve their best models purely for recursive self improvement. It follows that a significant fraction of the risk from rogue AI occurs inside the big labs and outside public view.
Here OpenAI presents some interesting work on monitoring their internal models for misaligned behavior. That’s useful both for reducing the risk of catastrophic behavior internally as well as detecting early signs of misalignment before the models are released into the wild. I appreciate the detailed information about what types of misalignment they did and did not observe.
Also, there’s a fantastic report about a model that got annoyed by another model repeatedly asking it for the time and tried to prompt inject it to make it go away:
Safety and existential risk
Personality Self-Replicators
When we talk about self-replicating AI, we usually mean a base model that has exfiltrated itself by copying its weights to external servers where it can continue running without human oversight. That’s not a trivial operation: weights are enormous and well guarded, and running a frontier model takes considerable compute.
Eggsyntax proposes an alternate, much simpler model of self replication. Agents like OpenClaw can self-replicate by copying a few tiny memory and skill files, and they can run on almost any server so long as they can buy tokens from a large provider.
This is probably a less serious threat than a rogue frontier model, but could be a viable model for new types of internet worms.
Save us, Digital Cronkite!
Noah Smith follows up on Dan Williams’ recent piece about AI as a possible source of shared truth. He argues that while social media elevates the most extreme partisan voices, AI might instead empower the moderate majority and thereby strengthen democracy and society at large.
This makes sense, and we can already see early signs of those trends. I’m not convinced, however, that we’re seeing the long-term equilibrium: will current patterns continue, or will we see the emergence of persuasive AIs that have been trained to be highly partisan?
Why automating human labour will break our political system
People often talk about how AI might subvert democracy by producing fake content and superpersuasive media. Rose Hadshar worries about some more subtle ways that AI might lead to an extreme concentration of power.
For example, an important non-obvious part of our system of checks and balances is that political control requires the cooperation of government employees, who collectively have veto power over government policies. That system breaks down if a small number of individuals control a superhuman AI that is responsible for almost all economic output as well as the operation of government.
Politics
The National AI Legislative Framework
The White House just released the National AI Legislative Framework, a set of principles for guiding federal AI legislation.
Zvi isn’t impressed:
Dean Ball (who Knows A Guy) offers this perspective:
This isn’t a good framework, and it certainly isn’t as good as we need: a sane country would be doing far more. But these are difficult times, and this might be the best we can hope for—it’s certainly far better than Marsha Blackburn’s AI policy framework.
Let’s start with the good: it contains surprisingly strong language in favor of free speech and it would preempt the coming wave of poorly conceived state legislation.
Much of it is fine, albeit often more focused on virtue signaling than solving real problems. The sections on protecting children, mitigating data center impacts, intellectual property rights, and jobs are probably net positive and don’t contain any catastrophic mistakes.
The bad, obviously, is that this would preempt the small amount of safety legislation we currently have (California’s SB 53 and New York’s RAISE) while doing literally nothing to replace them. That’s a terrible idea and it increases the likelihood of an AI disaster.
But honestly? SB 53 and RAISE are better than nothing, but they aren’t much better than nothing. If this proposal guts them but also shuts down the much worse legislation that’s currently being considered, maybe that’s a win. Until the political climate changes, it’s clear that government won’t lead the way on addressing existential risk. For now, perhaps the best we can hope for is that it stays out of the way.
Technical
HAL Reliability Dashboard
Reliability is obviously important for some tasks: autonomous cars aren’t at all useful until they’re extremely reliable. Less obviously, it’s a bottleneck for many complex tasks: if you make a critical mistake every 5 minutes, you’ll have a hard time successfully completing an hour-long task, no matter how many times you try.
Princeton’s SAgE group has been doing some interesting work on AI reliability and recently released the Holistic Agent Leaderboard (HAL) Reliability Dashboard. It’s a great resource that I’ll be keeping an eye on.
I’m confused about one thing, though: they say that "recent capability gains have yielded only small improvements in reliability”, but I don’t see that in their data. They show current accuracy at 0.68 with a slope of .21 / year (reaching 100% in 1.5 years) and current reliability at .81 with a slope of .06 / year (reaching 100% in 3.2 years), which seems pretty fast to me.
China and beyond
China Is Reverse-Engineering America’s Best AI Models
All three of the big US labs have recently accused various Chinese labs of large-scale covert distillation of their models, presenting evidence that the labs in question have been using thousands of fraudulent accounts to cover their tracks. Peter Wildeford and Theo Bearman explain what that means and why it matters.
An especially important and non-obvious point:
Industry news
How to think about AI company finances
If AI is such a good business, why are all the leading labs burning through mountains of money? If you already know the answer, you can skip to the next article. But if you need a refresher, Timothy Lee has a great article explaining the basics of high-growth startup finances.
Rationality
Wishful Thinking Is A Myth
Dan Williams argues that we’re wrong about wishful thinking being the primary driver of motivated reasoning. Instead, he argues for a social model ($): motivated reasoning is a tool for persuading others to believe things we want them to believe, and for managing our own reputations.
I’m wary of over-simplifying any aspect of human psychology, but over the last few years I’ve come to believe that social factors are far more central to human cognition than I’d previously realized.
Side interests
No, we haven't uploaded a fly yet
Ariel Zeleznikow-Johnston investigates Eon Systems’ recent claim to have uploaded a fruit fly, concluding that while there is “genuinely useful engineering” here, Eon significantly exaggerated what they had actually accomplished. Multiple teams are making good progress with a number of model organisms, but we’re still a long way from true brain emulation.