Quick Takes

Sometimes people from middle powers ask me what middle powers can do to help the AI situation. Recently for example I was asked this by an Australian. This is what I said: Main tool Australia and other middle power governments have is their voice. They should say loudly "We used to be dismissive of talk of superintelligence and associated risks, but now based on the accumulation of evidence we think that these things are all too serious as possibilities and could really happen in the next year or three. We consider this a threat to our sovereignty, livelihood, and even our lives, and will act accordingly. We are hereby [costly signals of seriousness e.g. declaring a national emergency, calling up the reserves, distributing PPE equipment to every citizen, banning AI companies from operating in Australia unless they allow third-party assessments of risk and in particular allow UK and AUS AISI's access to investigate incidents like Hugging Face] effective immediately. Also, we are hosting a summit in 72 hours for world leaders to come discuss the risks and benefits of recursively self-improving superintelligent AI, and whether and how AI development should be allowed to proceed." The reason this is so important is that right now, even in the US, most citizens do not want superintelligence or RSI to happen. Unlike many political issues, this political issue is one where a supermajority would agree on what needs to be done (~pause or otherwise heavily regulate AI development immediately) if only they understood that this is a real issue and not sci-fi. Having an actual government stop acting like it's sci-fi and start acting like it's real can cause a cascading effect where a bunch of other people will see that and then start talking and acting like it's real too, which then wakes up others, etc. until all of a sudden the discourse has shifted from "ha ha if it's so risky why are they still building it ha ha" to "OK so obviously they need to stop immediately, let's do

The last few days, David and I did the same exercise which IIUC produced AI 2027: predict the next month, condition on those predictions, predict the month after that, etc. We assumed that: * there is no secret sauce of intelligence missing from today's LLM-based AIs * METR's "time horizon" metric is a basically reasonable way to think about LLM capabilities Besides just running forward median predictions, for each month we also came up with a list of potential surprises, of a similar flavor to "news breaks that Jensen Huang had an affair with a Chinese spy" or "OpenAI releases a proof that the Exponential Time Hypothesis is false", and we randomly rolled a couple of those for each month. The random rolls came out moderately worse than I'd actually expect, even conditional on the assumptions. But they weren't that far out of distribution, and... We died in January. Two months from now. Regulations hadn't even been passed, let alone implemented; if the government was't already enforcing a full Pause in December they didn't matter at all. AIs were starting to be able to run profitable businesses, which made it possible for the first time for AIs to exfiltrate and operate fully independently online; that also didn't matter at all. There was some crazy stuff going on with AI company finances, and none of it mattered at all. The Hackening hadn't even gotten seriously underway yet. The only thing which mattered was a single random event: "OpenAI announces that their internal automated AI research swarm has found a way to improve the scaling power-law coefficient on data by 20%.". That wasn't written to be a "we die" event, just a plausible sounding algorithmic advancement. When we ran the numbers, it was enough for immediate supercritical RSI.

I'm interested in someone working out and writing up "Plan A-", an international agreement that's like Plan A (as described in AI 2040), except: - Limits HW progress via a deal to destroy compute (instead of building compute and limiting SW) - Assumes 2027 timelines instead of 2030 timelines - No total research transparency, no mutually assured compute destruction. (Which I think are good but are sticking points for many). I call it Plan A- because it's worse than (well implemented) Plan A in a bunch of ways (e.g. much harder to pay safety taxes), but it's more feasible, requires much less institutional capacity, and seems pretty robust. All things considered, Plan A- is my current recommendation for what governments should do.

A 5+% chance of very short timelines (1-2 years) to strong ASI is unlikely to ever go away, it's the new normal since late 2026. It's not about things like the METR report, which merely changes the probabilities, the very short timelines were already priced in for a few months, across many ways of thinking about the situation. Certainly since September, when the math breakthroughs started flooding in (a runaway process that rapidly climbs the tech tree gets more plausible once highly technical work that forms very big projects gets within reach, pursued autonomously). With all the compute widely available, and the continued inefficiency of its current use in AIs, a breakthrough will be able to start the process at any time, rapidly scaling it with all the dry tinder. What can change is that someone currently expecting a 30-60% chance of takeoff within 1-2 years could reduce that expectation lower, but probably never below 5% (starting at any future date, for strong ASI to take off within 1-2 years after that), unless all the compute gets phased out.

Jacob Coxon: [...] I'm not sure what to make of this. My view was something like most algorithmic progress is data progress, and also that most of the progress this year (excluding compute growth) was from better post-training. This reply from Jacob makes it sound like he isn't talking about better data (emphasis mine): [...] Data progress seems like it'd be much more continuous, but someone discovering a better attention variant or optimizer totally fits here. I don't think this matters much for short-term progress (just extrapolate your preferred progress line linearly) but it seems extremely relevant for RSI and progress over the next couple years. Like the more diverse AI progress is, the less one source hitting diminishing returns would matter. For AI automation specifically, I expect there to be big differences in how much they can accelerate each source. Even if they were exactly (expert) human level at every skill, the impact of 10x, 1,000x, or 1,000,000x as much labor seems quite uncertain.

working with agents feels like managing a team of like 5-10 freakishly fast interns who have shockingly deep knowledge about a whole bunch of things like gpu programming or math, and who somehow still need basic guidance every hour or so to not make terrible decisions. it's pretty easy for me to spend an entire day just tabbing between my agents and redirecting them when they are doing something weird or getting stuck on something. the more deeply i understand something and the more actively i interpose myself on decision making, the better outcomes are. if i task the agents with doing some very large thing that i have no idea how to do, i will get a horrific +100k line diff that breaks all the time in mysterious undebuggable ways, until at some point it all collapses in on itself and nothing ever works anymore and i need to throw the code away. if i design the code carefully and just use the agent to execute on a very specific vision of code i have in my head, the result is much more robust and successful, even though it actually doesn't even require that much of my time to design it and then answer a bunch of questions. it feels really weird to be experiencing this brief interstitial time where it is basically completely pointless to do things by hand, and yet the rsi loop hasn't been closed yet, and so there is this job of agent herding that is absurdly productive but also very strangely shaped.

It's been pointed out that the 9/11 attacks on the US resulted in decades of US paranoia and imposed costs (eg at airports) that were arguably a great result for Obama bin Laden. It would be grimly amusing if the discovery of Jensen Huang's affair with an alleged Chinese spy, and the subsequent Nvidia selloff (which has so far shed nearly $2 trillion in value), resulted in a broader tech crash, leading to the US tech industry falling behind China.