Thanks for the post!

Nikola Jurkovic suggests that as soon as model can do a month-long task with 80% accuracy, it should be able to do any cognitive human task

I didn't mean to suggest this in my original post. My claim was something more like "a naive extrapolation means that 80% time horizons will be at least a month by the end of the decade, but I think they'll be much higher and we'll have AGI"

it has to go to infinity when we get AGI / superhuman coder.

This isn't necessarily true, as even an AGI or a superhuman coder might get worse at tasks-that-take-humans-longer compared to tasks-that-take-humans-shorter (this seems pretty likely given constant-error-rate considerations), meaning that even an extremely capable AI might be like 99.999% reliable for 1 hour tasks, but only 99.9% reliable for 10,000 hour tasks, meaning the logistic fit still has an intercept with 50%, it's just a very high number. 

In order for the 50% intercept to approach infinity, you'd need a performance curve which approaches a flat line, and this seems very hard to pull off and probably requires wildly superhuman AI.

I think a 1-year  50%-time-horizon is very likely not enough to automate AI research. The reason I think AI research will be automated by EOY 2028 is because of speedups from partial automation as well as leaving open the possibility of additional breakthroughs naturally occurring.

A few considerations that make me think the doubling time will get faster:

1. AI speeding up AI research probably starts making a dent in the doubling time (making it at least 10% faster) by the time we hit 100hr time horizons (although it's pretty hard to reason about the impacts here)
2. I think I place some probability on the "inherently superexponential time horizons" hypothesis. The reason I think it is because to me, 1-month-coherence, 1-year-coherence, and 10-year-coherence (of the kind performed by humans) seem like extremely similar skills and will thus be learned in quick succession.
3. It's plausible reasoning models decreased the doubling time from 7 months to 4 months. It's plausible we get another reasoning-shaped breakthrough.

So my best guess for the 50% and 80% time horizons at EOY 2028 are more like 10yrs and 3yrs or something. But past ~2027 I care more about how much AI R&D is being automated rather than the time horizon itself (partially because I have FUD about what N-year tasks should even look like by definition).

Grok 4 is slightly above SOTA on 50% time horizon and slightly below SOTA on 80% time horizon: https://x.com/METR_Evals/status/1950740117020389870

I heard it from someone who works at xAI

xAI's safety team is 3 people.

I would have taken this class had I not graduated this spring!

A few suggestions:

I would like to cover the various ways AI could go wrong: malfunction, misuse, societal upheaval, arms race, surveillance, bias, misalignment, loss of control,...

1. AI 2027

Talk about predictions for the future, methodologies for how to come up with them.

1. Measuring AI Ability to Complete Long Tasks
2. The case for AGI by 2030

Some technical components should include: evaluations

1. Evaluating frontier AI R&D capabilities of language model agents against human experts

All of the above said, I get antsy if I don't get my dosage of math and code- I intend 80% of the course to be technical and cover research papers and results. It should also involve some hands on projects.

1. Recent Redwood Research project proposals
2. Open problems in emergent misalignment

Another thing I consider really important: many of the students will be like "Holy shit, AGI is happening! This affects my life plans!" and will want advice. I think it's good to have something to point them to, like:

1. Why AGI could be here soon and what you can do about it: a primer

Good luck running the course! 

The US Secretary of Energy says "The AI race is the second Manhattan project."

https://x.com/SecretaryWright/status/1945185378853388568

Similarly, the US Department of Energy says: "AI is the next Manhattan Project, and THE UNITED STATES WILL WIN. 🇺🇸"

https://x.com/ENERGY/status/1928085878561272223

Agreed, thanks! I've moved that discussion down to timelines and probabilities.

I don't think I make the claim that a DSA is likely to be achieved by a human faction before AI takeover happens. My modal prediction (~58% as written in the post) for this whole process is that the AI takes over while the nations are trying to beat each other (or failing to coordinate). 

In the world where the leading project has a large secret lead and has solved superalignment (an unlikely intersection) then yes, I think a DSA is achievable.

Maybe a thing you're claiming is that my opening paragraphs don't emphasize AI takeover enough to properly convey my expectations of AI takeover. I'm pretty sympathetic to this point.