Quick Takes

Jacob Coxon: [...] I'm not sure what to make of this. My view was something like most algorithmic progress is data progress, and also that most of the progress this year (excluding compute growth) was from better post-training. This reply from Jacob makes it sound like he isn't talking about better data (emphasis mine): [...] Data progress seems like it'd be much more continuous, but someone discovering a better attention variant or optimizer totally fits here. I don't think this matters much for short-term progress (just extrapolate your preferred progress line linearly) but it seems extremely relevant for RSI and progress over the next couple years. Like the more diverse AI progress is, the less one source hitting diminishing returns would matter. For AI automation specifically, I expect there to be big differences in how much they can accelerate each source. Even if they were exactly (expert) human level at every skill, the impact of 10x, 1,000x, or 1,000,000x as much labor seems quite uncertain.

What should alignment researchers do if RSI is imminent? One proposition is coming up with alignment ("good") properties. So, when we are in the RSI loop, instead of asking AI to just make an AI which is more capable, we go on a ladder: ask AI to make a version with the same level of capabilities, but also with a bunch of nice safety properties (e.g. interpretability). Once we made sure that those are actually what we want, we can go further and ask AI to make new AI, which keeps all of the safety properties, but is also slightly more capable. And then we go up the ladder, and hope that at some point we get a set of properties robust enough to be scalable all the way to ASI. Under this model, it's valuable for alignment researchers to come up with desiderata, even if they couldn't figure out how to apply them to current AIs or even in principle. I see some problems with this proposal. Sure, we do notice that models are good at verifiable tasks, like math and coding, but the alignment progress is not bottlenecked on them. We are mostly bottlenecked on the conceptual progress. As Yud puts it, "if you know what theorem you would need to prove in order for you to have safe ASI, you’re 99.9% there". "But AI's could help us with conceptual progress too!!" [1]- I hear you say. But AIs which are capable of conceptual progress are also the ones who are able to pull "conceptual coups", in the sense that they will be able to trick us into creating AIs which are misaligned with us but aligned with them. And the task of creating AIs such that they are not going to do that requires the level of conceptual progress we have not yet achieved and are unlikely to achieve in time. Basically the good ol’ “you can’t make AI do your alignment homework” argument. 1. ^ I personally think that current AIs are horrible at this. I'm not claiming to be avtop-tier philosopher or anything, but the way they do it is just so ass. They are very eager to get to a superficial but deep-

Sometimes people from middle powers ask me what middle powers can do to help the AI situation. Recently for example I was asked this by an Australian. This is what I said: Main tool Australia and other middle power governments have is their voice. They should say loudly "We used to be dismissive of talk of superintelligence and associated risks, but now based on the accumulation of evidence we think that these things are all too serious as possibilities and could really happen in the next year or three. We consider this a threat to our sovereignty, livelihood, and even our lives, and will act accordingly. We are hereby [costly signals of seriousness e.g. declaring a national emergency, calling up the reserves, distributing PPE equipment to every citizen, banning AI companies from operating in Australia unless they allow third-party assessments of risk and in particular allow UK and AUS AISI's access to investigate incidents like Hugging Face] effective immediately. Also, we are hosting a summit in 72 hours for world leaders to come discuss the risks and benefits of recursively self-improving superintelligent AI, and whether and how AI development should be allowed to proceed." The reason this is so important is that right now, even in the US, most citizens do not want superintelligence or RSI to happen. Unlike many political issues, this political issue is one where a supermajority would agree on what needs to be done (~pause or otherwise heavily regulate AI development immediately) if only they understood that this is a real issue and not sci-fi. Having an actual government stop acting like it's sci-fi and start acting like it's real can cause a cascading effect where a bunch of other people will see that and then start talking and acting like it's real too, which then wakes up others, etc. until all of a sudden the discourse has shifted from "ha ha if it's so risky why are they still building it ha ha" to "OK so obviously they need to stop immediately, let's do

The last few days, David and I did the same exercise which IIUC produced AI 2027: predict the next month, condition on those predictions, predict the month after that, etc. We assumed that: * there is no secret sauce of intelligence missing from today's LLM-based AIs * METR's "time horizon" metric is a basically reasonable way to think about LLM capabilities Besides just running forward median predictions, for each month we also came up with a list of potential surprises, of a similar flavor to "news breaks that Jensen Huang had an affair with a Chinese spy" or "OpenAI releases a proof that the Exponential Time Hypothesis is false", and we randomly rolled a couple of those for each month. The random rolls came out moderately worse than I'd actually expect, even conditional on the assumptions. But they weren't that far out of distribution, and... We died in January. Two months from now. Regulations hadn't even been passed, let alone implemented; if the government was't already enforcing a full Pause in December they didn't matter at all. AIs were starting to be able to run profitable businesses, which made it possible for the first time for AIs to exfiltrate and operate fully independently online; that also didn't matter at all. There was some crazy stuff going on with AI company finances, and none of it mattered at all. The Hackening hadn't even gotten seriously underway yet. The only thing which mattered was a single random event: "OpenAI announces that their internal automated AI research swarm has found a way to improve the scaling power-law coefficient on data by 20%.". That wasn't written to be a "we die" event, just a plausible sounding algorithmic advancement. When we ran the numbers, it was enough for immediate supercritical RSI.

I'm interested in someone working out and writing up "Plan A-", an international agreement that's like Plan A (as described in AI 2040), except: - Limits HW progress via a deal to destroy compute (instead of building compute and limiting SW) - Assumes 2027 timelines instead of 2030 timelines - No total research transparency, no mutually assured compute destruction. (Which I think are good but are sticking points for many). I call it Plan A- because it's worse than (well implemented) Plan A in a bunch of ways (e.g. much harder to pay safety taxes), but it's more feasible, requires much less institutional capacity, and seems pretty robust. All things considered, Plan A- is my current recommendation for what governments should do.

working with agents feels like managing a team of like 5-10 freakishly fast interns who have shockingly deep knowledge about a whole bunch of things like gpu programming or math, and who somehow still need basic guidance every hour or so to not make terrible decisions. it's pretty easy for me to spend an entire day just tabbing between my agents and redirecting them when they are doing something weird or getting stuck on something. the more deeply i understand something and the more actively i interpose myself on decision making, the better outcomes are. if i task the agents with doing some very large thing that i have no idea how to do, i will get a horrific +100k line diff that breaks all the time in mysterious undebuggable ways, until at some point it all collapses in on itself and nothing ever works anymore and i need to throw the code away. if i design the code carefully and just use the agent to execute on a very specific vision of code i have in my head, the result is much more robust and successful, even though it actually doesn't even require that much of my time to design it and then answer a bunch of questions. it feels really weird to be experiencing this brief interstitial time where it is basically completely pointless to do things by hand, and yet the rsi loop hasn't been closed yet, and so there is this job of agent herding that is absurdly productive but also very strangely shaped.

Registering a prediction that the nation-wide ground stop just issued by the FAA is from an AI-enabled cyberattack.