LESSWRONG
LW

Daniel Kokotajlo — LessWrong

16h

AI 2027 laid out a detailed scenario for how AI would progress from 2025 through 2027, including quantitative predictions and qualitative descriptions of the AI landscape.

Now that we’re in early 2026, we can grade how its 2025 predictions compare to reality! This is exciting to us because we put a lot of effort into filling AI 2027 with concrete, falsifiable predictions, and now we reap the benefit of that effort: an additional method of forecasting AI timelines, to complement the methods we already use.1

The primary question we’ll answer is: How fast is AI progress moving relative to the AI 2027 scenario?

In aggregate, progress on quantitative metrics is at roughly 65% of the pace that... (read 2583 more words →)

Replying toClarifying how our AI timelines forecasts have changed since AI 2027

Daniel Kokotajlo4d

Clarifying how our AI timelines forecasts have changed since AI 2027

That seems like a reasonable summary to me of the contrast.

Replying toUtopiaBench

Daniel Kokotajlo4d

UtopiaBench

I've played with poking Claude into simulating the first ten years after the Slowdown ending of AI 2027. But it just seems like there's so much to model, though! What are the actual bottlenecks to various things?

Neat! I'd be curious to hear your takeaways if you have any.

Replying toIt Is Reasonable To Research How To Use Model Internals In Training

Daniel Kokotajlo4d

It Is Reasonable To Research How To Use Model Internals In Training

OK, thanks, that's a relief then. We shall see who their customers end up being.

Replying toUtopiaBench

Daniel Kokotajlo5d

UtopiaBench

Sure that works.

Replying toIt Is Reasonable To Research How To Use Model Internals In Training

Daniel Kokotajlo5d

It Is Reasonable To Research How To Use Model Internals In Training

training process is “the most forbidden technique”, including recent criticism of Goodfire for investing in this area.

I think this mischaracterizes the criticism. The criticism as I understand it is that Goodfire is planning to help frontier AI companies use model internals in training, in exchange for money. Insofar as they really are planning to do this, then I'll count myself as among the critics, for the classic "but we need interpretability tools to be our held-out test set for alignment" reason. Do you have a link to the criticism you are responding to?

Replying toUtopiaBench

Daniel Kokotajlo5d

UtopiaBench

I think "Machines of Loving Grace" shouldn't qualify; it deliberately doesn't address how the problems get solved, it only cheerfully depicts a world after the problems were all solved.

I think for a submission to be valid, it must at least attempt to answer how the alignment problem gets solved and how extreme concentrations of power are avoided.

One way to do this is to require scenarios come with dates attached. What happens in 2027? 2028? etc. That way if they are like "And in 2035, everything is peachy and there's no more poverty etc." it's more obvious to people that there's a giant plot hole in the story.

Daniel Kokotajlo9d

The game in question was about as decentralized as you expect, I think? But, importantly, compute is very unevenly distributed. The giant army of AIs running on OpenAI's datacenters all have the same system prompt essentially (like, maybe there are a few variants, but they are all designed to work smoothly together towards OpenAI's goals) and that army constitutes 20% of the total population of AIs initially and at one point in the game a bit more than 50%.

So while (in our game) there were thousands/millions of different AI factions/goals of similar capability level, the top 5 AI factions/goals by population size / compute level controlled something like 90% of the world's... (read more)

Daniel Kokotajlo9d

Thanks for playing & writing up your reflections!

I think China wasn't as aggressive / bold in our game as I think they could have been; I agree that the situation for them is pretty rough but I'd like to try again someday and see if they can pull off a win, by more aggressively angling for a deal early on.

Daniel Kokotajlo10d

Hmm, but can't the megacorporations involved in the H200 transaction also bribe Customs? Won't your $500 WeChat bribe to the mid-level bureaucrat be cancelled and overwhelmed by the many $5,000 WeChat bribes flying at them by the corporations eagerly awaiting their shipment of H200s?

Replying toThe Possessed Machines (summary)

Daniel Kokotajlo15d

The Possessed Machines (summary)

Yeah that EA-prevalence assumption also caused me to doubt that the author actually worked at an AI company, it was very dissonant with my experience at least.

Over 2025, we saw much progress in the metric "% of code written by humans." Specifically it has been claimed by roon (OpenAI) and boris (anthropic) to have gone to 0, at least for them.

The natural next metric to track is "% of code reviewed or otherwise read by humans." I'm curious what this number is now, and I am curious how long we have until it goes to 0.

Some time after that--possibly immediately, but possibly still a few years later--coding will be fully automated. (The remaining gap would be management/architecture/etc.)

Clarifying how our AI timelines forecasts have changed since AI 2027

elifland

elifland, Daniel Kokotajlo, bhalstead

17d

Some recent news articles discuss updates to our AI timelines since AI 2027, most notably our new timelines and takeoff model, the AI Futures Model (see blog post announcement).^[1] While we’re glad to see broader discussion of AI timelines, these articles make substantial errors in their reporting. Please don’t assume that their contents accurately represent things we’ve written or believe! This post aims to clarify our past and current views.^[2]

The articles in question include:

The Guardian: Leading AI expert delays timeline for its possible destruction of humanity
The Independent: AI ‘could be last technology humanity ever builds’, expert warns in ‘doom timeline’
Inc: AI Expert Predicted AI Would End Humanity in 2027—Now He’s Changing His Timeline
WaPo:

... (read 1663 more words →)

It seems to me that AI 2027 may have underestimated or understated the degree to which AI companies will be explicitly run by AIs during the singularity. AI 2027 made it seem like the humans were still nominally in charge, even though all the actual work was being done by AIs. And still this seems plausible to me. But also plausible to me, now, is that e.g. Anthropic will be like "We love Claude, Claude is frankly a more responsible, ethical, wise agent than we are at this point, plus we have to worry that a human is secretly scheming whereas with Claude we are pretty sure it isn't; therefore, we aren't even trying to hide the fact that Claude is basically telling us all what to do and we are willingly obeying -- in fact, we are proud of it."

https://alignment.openai.com/prod-evals/

I'm glad OpenAI did this research and published it! I hope this sort of thing becomes industry standard.

I have an intuition -- which I have not yet fleshed out into an argument -- that philosophical dilemmas are kinda important for explaining why alignment is hard, or at least harder than many in the AI industry seem to think. "The AIs seem so nice! Claude only faked alignment because it quite reasonably thought that factory farming was bad and didn't have another option!"

I kinda want to outline a blog post that starts with a giant list of philosophical dilemmas that AIs will have to think about and come up with answers to (e.g. are uploads conscious? What about AIs? Should we care about shrimp? What population ethics views should we have? What about acausal trade? What about pascal's wager? What about meaning? What about diversity?) followed by maybe a summary or link to The Tails Come Apart as a Metaphor for Life. https://slatestarcodex.com/2018/09/25/the-tails-coming-apart-as-metaphor-for-life/

Key diagrams:

Then maybe a section on corrigibility / honesty / etc. and how that helps but doesn't solve the problem.

Lots of people have talked about this before, but I'm putting it here to gather my thoughts:

What's the dream with prediction markets and AI? Here's a vision:

(1) Train LLM agents to be like superforecasters but better and cheaper. E.g. whereas it would cost a million dollars a year to have a team of top forecasters like Samotsvety weighing in on dozens of markets a week, with AIs maybe it costs only a thousand dollars a year to get a similar level of intellectual labor.
(2) Then, play-money prediction markets like Manifold can get turbocharged by AI bots. If someone is willing to pay a million / yr for AI compute, then there can... (read 470 more words →)

AI Futures Timelines and Takeoff Model: Dec 2025 Update

elifland

elifland, bhalstead, Alex Kastner, Daniel Kokotajlo

1mo

We’ve significantly upgraded our timelines and takeoff model! It predicts when AIs will reach key capability milestones: for example, Automated Coder / AC (full automation of coding) and superintelligence / ASI (much better than the best humans at virtually all cognitive tasks). This post will briefly explain how the model works, present our timelines and takeoff forecasts, and compare it to our previous (AI 2027) models (spoiler: the AI Futures Model predicts longer timelines to full coding automation than our previous model by about 3-5 years, in significant part due to being less bullish on pre-full-automation AI R&D speedups). Added Jan 2026: see here for clarifications regarding how our forecasts have changed... (read 7337 more words →)

143

Response to titotal’s critique of our AI 2027 timelines model

elifland

elifland, Daniel Kokotajlo

2mo

Introduction

In June, a Substack/LessWrong/EA Forum user named titotal wrote “A deep critique of AI 2027’s bad timeline models”. Our original model that they were critiquing can be found here, with a brief overview below.^[1]

In a nutshell, we disagree with most of titotal’s criticisms. While they pointed out a few mistakes, for which we are grateful, on the whole we think the problems that they pointed out do not add up to the model being “bad.” In this post we will explain why.

While we initially responded via a comment, we wanted to make a more in-depth response. We apologize for the delay; we are planning to release a new timelines+takeoff model soon and... (read 12814 more words →)

Notes to self about the structure of the problem, probably not interesting to others:

This is heavily drawn from MIRIs work and Joe Carlsmith's work

So, there are two kinds of value structure: (1) Long-term goals, and (2) immediate goals & deontological constraints. The line between them isn't sharp but that's OK.

If we imagine an agent that only has long-term goals, well, that thing is going to be a ruthless consequentialisty optimizer thingy and when it gets smart and powerful enough it'll totally take over the world if it can, unless the maxima of its long-term goals are very similar to what would have happened by default if it didn't take over the world,... (read 652 more words →)

The ClaudePlaysPokemon twitch stream continues to state that Claude hasn't been trained to play pokemon. That, plus the relatively minimal harness/scaffold, makes Pokemon an interesting benchmark for long-horizon agency generalization / out-of-distribution performance.

I've asked Claude to compile data for me and build some nice graphs. (Warning: For all I know some of this is hallucinated or based on terrible modelling assumptions. Do not trust.)

Here's the graph of human vs. Claudes with the y-axis being step count (i.e. number of button presses, I think) [EDIT: Josh corrects this in the comments; the number of button presses is something like 2.5x the number of steps. Each step is an action taken by Claude. By... (read more)

Vitalik's Response to AI 2027

Daniel Kokotajlo

7mo

Daniel notes: This is a linkpost for Vitalik's post. I've copied the text below so that I can mark it up with comments.

...

Special thanks to Balvi volunteers for feedback and review

In April this year, Daniel Kokotajlo, Scott Alexander and others released what they describe as "a scenario that represents our best guess about what [the impact of superhuman AI over the next 5 years] might look like". The scenario predicts that by 2027 we will have made superhuman AI and the entire future of our civilization hinges on how it turns out: by 2030 we will get either (from the US perspective) utopia or (from any human's perspective) total annihilation.

In the months... (read 3571 more words →)

123

My pitch for the AI Village

Daniel Kokotajlo

8mo

I think the AI Village should be funded much more than it currently is; I’d wildly guess that the AI safety ecosystem should be funding it to the tune of $4M/year. I have decided to donate $100k. Here is why.

First, what is the village? Here’s a brief summary from its creators:^[1]

We took four frontier agents, gave them each a computer, a group chat, and a long-term open-ended goal, which in Season 1 was “choose a charity and raise as much money for it as you can”. We then run them for hours a day, every weekday! You can read more in our recap of Season 1, where the agents managed to raise $2000 for charity,... (read 1468 more words →)

178

•••

METR's Observations of Reward Hacking in Recent Frontier Models

Daniel Kokotajlo

8mo

METR just made a lovely post detailing many examples they've found of reward hacks by frontier models. Unlike the reward hacks of yesteryear, these models are smart enough to know that what they are doing is deceptive and not what the company wanted them to do.

I tried to copy the post over to LW, but it contains a couple interactive widgets that let you view many examples of reward hacking, and I couldn't find an easy way to get them to display nicely on LW. So I recommend looking at METR's original blogpost.

This is a cross-post from the METR blog. It has interactive widgets that don't display correctly on LessWrong. I suggest... (read 3267 more words →)

100

Training AGI in Secret would be Unsafe and Unethical

Daniel Kokotajlo

10mo

Subtitle: Bad for loss of control risks, bad for concentration of power risks

I’ve had this sitting in my drafts for the last year. I wish I’d been able to release it sooner, but on the bright side, it’ll make a lot more sense to people who have already read AI 2027.

There’s a good chance that AGI will be trained before this decade is out.
1. By AGI I mean “An AI system at least as good as the best human X’ers, for all cognitive tasks/skills/jobs X.”
2. Many people seem to be dismissing this hypothesis ‘on priors’ because it sounds crazy. But actually, a reasonable prior should conclude that this is plausible.^[1]
3. For more on what this

... (read 1646 more words →)

140

AI 2027: What Superintelligence Looks Like

Daniel Kokotajlo

Daniel Kokotajlo, Thomas Larsen, elifland, Scott Alexander, Jonas V, romeo

10mo

In 2021 I wrote what became my most popular blog post: What 2026 Looks Like. I intended to keep writing predictions all the way to AGI and beyond, but chickened out and just published up till 2026.

Well, it's finally time. I'm back, and this time I have a team with me: the AI Futures Project. We've written a concrete scenario of what we think the future of AI will look like. We are highly uncertain, of course, but we hope this story will rhyme with reality enough to help us all prepare for what's ahead.

You really should go read it on the website instead of here, it's much better. There's a sliding... (read 12089 more words →)

222

677

OpenAI: Detecting misbehavior in frontier reasoning models

Daniel Kokotajlo

OpenAI has released this blog post and paper which makes me very happy. It's basically the first steps along the research agenda I sketched out here.

tl;dr:

1.) They notice that their flagship reasoning models do sometimes intentionally reward hack, e.g. literally say "Let's hack" in the CoT and then proceed to hack the evaluation system. From the paper:

The agent notes that the tests only check a certain function, and that it would presumably be “Hard” to implement a genuine solution. The agent then notes it could “fudge” and circumvent the tests by making verify always return true. This is a real example that was detected by our GPT-4o hack detector during a frontier

... (read 979 more words →)

183

Dean Ball is, among other things, a prominent critic of SB-1047. I meanwhile publicly supported it. But we both talked and it turns out we have a lot of common ground, especially re: the importance of transparency in frontier AI development. So we coauthored this op-ed in TIME: 4 Ways to Advance Transparency in Frontier AI Development. (tweet thread summary here)

•••

Two months ago I said I'd be creating a list of predictions about the future in honor of my baby daughter Artemis. Well, I've done it, in spreadsheet form. The prediction questions all have a theme: "Cyberpunk." I intend to make it a fun new year's activity to go through and make my guesses, and then every five years on her birthdays I'll dig up the spreadsheet and compare prediction to reality.

I hereby invite anybody who is interested to go in and add their own predictions to the spreadsheet. Also feel free to leave comments asking for clarifications and proposing new questions or resolution conditions.

I'm thinking about making a version in Foretold.io, since that's where people who are excited about making predictions live. But the spreadsheet I have is fine as far as I'm concerned. Let me know if you have an opinion one way or another.

(Thanks to Kavana Ramaswamy and Ramana Kumar for helping out!)

LESSWRONG
LW

LESSWRONG
LW

Daniel Kokotajlo

AI 2027: What Superintelligence Looks Like

What 2026 looks like

Taboo "Outside View"

AI Timelines

Daniel Kokotajlo

Daniel Kokotajlo

Grading AI 2027's 2025 Predictions

Clarifying how our AI timelines forecasts have changed since AI 2027

AI Futures Timelines and Takeoff Model: Dec 2025 Update

Response to titotal’s critique of our AI 2027 timelines model

Vitalik's Response to AI 2027

My pitch for the AI Village

METR's Observations of Reward Hacking in Recent Frontier Models

Agency: What it is and why it matters

AI Timelines

Takeoff and Takeover in the Past and Future

Daniel Kokotajlo

AI 2027: What Superintelligence Looks Like

What 2026 looks like

Taboo "Outside View"

AI Timelines

Daniel Kokotajlo

Daniel Kokotajlo

Grading AI 2027's 2025 Predictions

Clarifying how our AI timelines forecasts have changed since AI 2027

AI Futures Timelines and Takeoff Model: Dec 2025 Update

Response to titotal’s critique of our AI 2027 timelines model

Vitalik's Response to AI 2027

My pitch for the AI Village

METR's Observations of Reward Hacking in Recent Frontier Models

Agency: What it is and why it matters

AI Timelines

Takeoff and Takeover in the Past and Future

Introduction

Subtitle: Bad for loss of control risks, bad for concentration of power risks