Ikaxas' Shortform Feed

Vaughn Papenhausen

This is a special post for quick takes by Vaughn Papenhausen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

As Raemon has suggested a format for a short-form content feed, I'm going to go ahead and make one. His explanation of the format:

I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:

don't feel ready to be written up as a full post
I think the process of writing them up might make them worse (i.e. longer than they need to be)

I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.

Edit: For me at least, #2 also includes "writing them up as a full post would involve enough effort that the post would probably otherwise not get written."

This is a special post for quick takes by Vaughn Papenhausen. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

As Raemon has suggested a format for a short-form content feed, I'm going to go ahead and make one. His explanation of the format:

I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:

don't feel ready to be written up as a full post
I think the process of writing them up might make them worse (i.e. longer than they need to be)

I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.

Edit: For me at least, #2 also includes "writing them up as a full post would involve enough effort that the post would probably otherwise not get written."

Quick take on my forecasts of various AI 2040 branches

After Pacing the Frontier (Jul. 28th), @Daniel Kokotajlo posted his revised probabilities on the branches of Plan A:

Shamelessly cribbing off of his starting point, here are my estimates in light of recent events, especially the reaction to Jacob Coxon's resignation:

Plan D (race): 10%

- US wins: 7.5%

- China wins: 2%

- SSI or other wins: .5%

Plan C (burn the lead): 40%

- US wins: 20%

- China wins: 18%

- SSI or other wins: 2%

Plan B (Fight China): 11%

Plan A (Global slowdown treaty): 10%

Plan S (Global shut-it-all-down treaty): 23%

Other outcome: ~6%

Thoughts:

The reaction to Jacob Coxon's resignation feels like at least as big a downward update on Plan D as Pacing the Frontier was, so I've bumped it down by another 10% (the breakdowns of who wins with Plan D and Plan C are in response to a friend prompting me to estimate that).

I've also reversed his starting points for Plan A and Plan S: I think that, conditional on global coordination, Plan S is more likely because You Get About 5 Words, and Plan A takes much more than 5 words to specify, while Plan S takes just two: "Stop ASI". I've then distributed the 10% reduction from Plan D evenly between Plan C and Plan S, so each gets bumped by 5%.

EDIT:
Actually, I should scale all this down by the chance that the current paradigm does not reach Automated AI Researcher, which I think is quite high, like 70% (for reasons @Thane Ruthenis lays out nicely here). So I guess it's more like this:

Paradigm shift required, taking 5-20 years after LLMs top out: ~70%
Conditional on LLMs scaling to automated AI researcher (30%):
- Plan D: 10%
- Plan C: 40%
- Plan B: 15%
- Plan A: 10%
- Plan S: 25%

4StanislavKrym2mo

How do we rule out the possibility that a fully neuralese LLM can become a fully artificial AI researcher, especially given the rise of Astra and solutions to Millenium Problems? Additionally, AI-2040 has the brilliant paragraph: "The best versions of Plan S are those that acknowledge that the deal will eventually end, and simply say “First (step 1) we should stop making frontier AIs more capable, because the AIs and AI companies are getting more powerful every day and there’s so much uncertainty about where it’s headed and how fast. Then (step 2) once the world has had several years to think about things and plan a safe and broadly beneficial path forward, we can resume.”" I expect Plan A to morph into this, especially if the world produces a model organism showing that 2026-level mechinterp cannot improve control enough to allow for another round of scaling.

I mean tbc I don't fully rule it out, I give it like 30%. But it does seem to me like LLMs continue to almost entirely lack hyperpolation. I think this is why all the recent math breakthroughs, according to mathematicians I've seen comment on them, continue to be of the form "apply ~known techniques, give counterexamples, etc." rather than "conceptual novelty." I suspect a paradigm shift is necessary to get to AIs that can do hyperpolation on the human level, and that human level hyperpolation is maybe necessary for full automated AI researcher (though wha... (read more)