So... yes, a human learns continually across its life in a way an LLM currently can't, and this is a big deal for each of us. But when you talk about humanity developing over millennia, that has nothing to do with continual learning within a lifetime. It has to do with each generation learning what it can, and then (pre)training the next generation to have a superior starting position. That is something LLMs can do. If Opus 9 can have a million instances of itself accumulate a week of experiences each and use those in the corpus for pretraining Opus 10 a month later, is that actually a bottleneck to long term knowledge accumulation, if the AI community (at least within a lineage) is sufficiently close to eusocial?
You tell your LLM story with Opus 9 selecting data for “pretraining Opus 10” but you could equally well change the story to have Opus 9 selecting data for “further fine-tuning itself”, and now it’s a story of how to get continual learning to work in LLMs. Doesn’t really matter, it amounts to the same thing. I’ll use the fine-tuning / continual learning description here because I think it makes things a bit easier to talk about, but it doesn’t matter, you can translate my next paragraph back into the other frame if you prefer.
Anyway, I don’t think it would work (although obviously we’ll find out one way or the other soon enough). If you want to make progress, it’s not enough to for an LLM to “accumulate experience” and then fine-tune on that experience. For example, if an LLM outputs a bunch of tokens, then you fine-tune that very same LLM on those very same tokens, then it won’t make the LLM smarter, it will only cause mode collapse. Instead you would need to do something like: have the LLM try to figure out what’s true by thinking for a while, produce a final artifact, and train only on that artifact but not the thinking trace. That’s not an obviously crazy idea, and maybe it would work a little bit, but I think what would happen eventually is that the LLM would make mistakes, then it would lock in those mistakes by fine-tuning, and then it would have confident wrong ideas that leads it to make more mistakes, etc., and in the long term it would get dumber and dumber, not smarter and smarter. I don’t think this kind of approach can lead to human-like open-ended creation of knowledge, akin to the way that human mathematicians invented math from scratch without proof assistants (see §1.1 of my “Sharp Left Turn” post).
some stuff about training an LLM on its outputs for fun, without taking any broader stance in this discussion:
in the limit of generating very many reasoning traces from a model, fine-tuning the model on this data with full-batch ideal gradient descent does not change the model at all, because it is already the globally (and so also locally) optimal log loss predictor for its own sampling distribution
Hmm, thinking about it more, I think you’re right (no change) if you draw the samples with temperature T=1; and my earlier comment was right (mode collapse, i.e. ever-increasing confidence in the modal next token, approaching 100%) if you draw the samples with temperature 0≤T<1, and repeat enough times. And if you use temperature T>1 then you get, umm, the opposite of mode collapse, where it approaches a uniform probability distribution when you repeat enough times. Right? (I’m not totally sure.) (I agree that this is a fun but irrelevant side-track.)
It is not rigorously obvious to me what happens in the
Even though I agree with you on this mathematical description of what happens, I actually feel uneasy about calling this "mode collapse".
[1]
Like, I think
That said, it remains plausible that sth imo more canonically like modes of the text distribution being lost would happen in the case with non-tiny gradient steps computed on single traces.
I think "mode collapse" doesn't mean collapse to the mode of the distribution, it means collapse of the distribution into fewer parts ("modes"), ie forgetting parts/regions of the generative distribution. My guess is that central examples for text generation would be only speaking Vietnamese, or only talking about consciousness, or only having one tone/persona, or starting every response with "It's a difficult question.", or only predicting the same token over and over. ↩︎
About the headings on the transcripts - it looks like each heading is at the end of the corresponding section of the transcript, instead of being at the beginning of the section? That seems like a mistake.
After using the latest version of Claude Code and being surprised how capable it's become while still behaving friendly and corrigibly, I wanted to reflect on how this new observation should update my world model and my P(Doom).
So I reached out to Dr. @Steven Byrnes, the highly polymathic AGI safety researcher who I last interviewed in August about Brain-like AGI, foom & doom, and solving technical alignment.
We discussed:
Video & Audio
The episode is available on Substack, YouTube, or by searching “Doom Debates“ in your podcast app.
Transcript
Introduction
Liron 00:00:48
Welcome to Doom Debates. Today, we’ve got a fan favorite returning to the show, Dr. Steven Byrnes. You may remember him from six months ago. We had a really long, enriching conversation that I highly recommend. It’s one of my favorite episodes of all time.
He is an artificial general intelligence safety researcher at the Astera Institute. He’s got a BA in physics and math from Harvard, PhD in physics from UC Berkeley. He did a physics postdoc at Harvard, research in thermodynamics and optics. In 2015 to 2021, he worked at a nonprofit applied physics research and development laboratory called Draper. Since 2021, he’s been an AGI safety researcher, first independent and now at Astera.
So pretty legit credentials. It doesn’t get much more legit in terms of a wide-ranging polymath than Dr. Steven Byrnes. I think you can see that evidenced on the last episode that we did. Today, I’m excited to catch up with Steve about the latest developments in his thinking since last summer. We’re gonna take stock of how AI has progressed in recent months and dig into Steve’s recent argument that we should still expect, unfortunately, ruthless, sociopathic ASI. Dr. Steve Byrnes, welcome back to Doom Debates.
Steve 00:02:00
Thanks for having me again.
Liron 00:02:01
Yeah, returning champion. What you have to say about these topics, I think is extremely credible. We got into this last episode. Near the end of the episode, we were saying, “Between you and I, it’s not really a doom debate, it’s just doom.” And viewers like it when somebody finally comes on the show that we can trust, or at least that my audience is generally on the same page with. And also, in many ways, you’re ahead of us. You have more detailed mental models than us.
So this is going to be very enlightening. Let’s start by catching up on the last six months since the summer. Give us a very high-level update. How would you describe the last six months playing out?
Steve 00:02:38
Well, let’s see. So I have my own research program related to brain-like artificial general intelligence — the idea that somebody might reverse engineer or reinvent the way that human brains are able to accomplish things in the world, start companies, invent science and technology from scratch over time. And if somebody reverse engineers or reinvents that, then how do we use it? How do we get to a good future?
If we wind up making reinforcement learning agents, then what reward functions, if any, should we use to get an agent that we want to live with and that wants to live with us? Last year, I was spending a lot of time trying to figure out how human social instincts work. We want to be nice to our friends, we want to impress people, and all the other aspects of human sociality and morality, ultimately, I think, come from the human brain reward function.
For years, I was trying to figure out how that worked, and I made a lot of progress. And over the last six months, I’ve been trying to relate that progress back to the world of reinforcement learning as opposed to the world of neuroscience, where I was before, and just hopefully get a little less confused about how to think about reward functions in general in light of what I think I’ve learned from trying to study the brain.
Liron 00:03:52
Okay, that’s an interesting pivot in the last few months. You’re saying not so much on the neuroscience and more getting into the weeds of how modern AIs work.
Steve 00:04:01
So modern AI is a term that often refers to large language models, and I should clarify that I’m not working much on large language models. I think much less than most people in AI safety. But modern AI — AI is a field of study, it includes a lot of different things. I’m more interested in the reinforcement learning branch of AI, which is kind of a forgotten backwater these days, but I am expecting and fearing that it might become bigger in the future.
Research Update: What’s New in The Last 6 Months
Liron 00:04:31
So just as a high level, imagine that somebody doesn’t follow the news and they just poke their head up for air every six months to check. If you look at June 2025 versus February 2026, I think it’s kind of followed the timeline of AI 2027. They had a famous timeline, and they weren’t saying this is definitely going to happen, but it seems to kind of be happening like that.
The biggest thing I remember from their 2025–2026 timeline is agents. First, they called them stumbling agents, where agents can kind of stumble around but then they crash and need a human to pick them up, and then they just go longer and longer, and they just do more and more of the work of humans. I think that has been very accurate. I think this is really the year of the agent. Agents have really just started working incredibly in the last few months. And to me, that’s the headline of progress over the last few months. What do you think is the last six months of news?
Steve 00:05:23
Yeah, certainly in terms of AI capabilities news. I don’t consider myself an expert. I don’t know more than you do. Probably I know quite a bit less. But yeah, everybody who uses the coding agents seems to be very impressed by them. I haven’t gotten around to installing it myself, but I certainly believe the reports when people describe what they can do, and that does seem to be an important development in LLM world.
Liron 00:05:48
I can join the chorus of people in your network telling you that it’s the real deal. I started getting into it in earnest a couple weeks ago, and it’s a very weird experience because I’ve been a software engineer my whole career, even since I was ten years old, and I’m not really a software engineer anymore. I don’t really look at my code anymore because I would much rather just tell the AI.
It’s to the point where, just so people see how far it’s gotten, let’s say I wrote something a year ago, a piece of code. You would think that I have some context to get back into my own code and make some tweaks. But it’s now actually easier for me to tell a clean version of Claude Code or OpenAI Codex to just be, “Hey, this file of code, something in here, just go change this component to do this.”
It’s actually faster for me to use voice-to-text and just write a couple lines of instruction to the AI, and the AI will spend thirty seconds understanding what I was thinking a year ago. It’s actually faster to do that than for me to go open the file and read my own code, even though it’s my own code that I wrote a year ago. It’s that good.
Steve 00:07:06
Yeah. I don’t have any big software projects that I’ve wanted to do lately, but when I have little one-off shell scripts or this or that, definitely you just ask the LLM to do it, and it prints it out, and it often works the first try, and if not, you can say what the error message is, and it often fixes it.
Liron 00:07:06
It’s really crazy. As a software engineer, I know what’s possible. So I can be like, “Hey, can you just go to this server and download this and move it here and do that?” And the AI’s like, “Yeah, I can do all that. Here’s a little plan. Okay, done.” And I’m like, “Wow, I could have done it too, but it would have taken me an entire focused weekend, and I would have been coming out of the weekend being, ‘All right, I got something done this weekend.’” And the AI is thirty minutes later, “Here you go.” And I’m like, “Oh my god.”
That’s not something that could have been done a few months ago. This is really new. It’s really crazy. When I leave my house and walk around the world, I want to tell everybody. “You guys know we have AIs that can replace a bunch of jobs right now?” People are totally clueless.
Steve 00:07:42
Yeah. I mean, I don’t know whether they’re replacing jobs or not. I’m not cued in enough to the software engineering world to know whether they’re—
The Rise of AI Agents
Liron 00:07:49
So that’s my experience. I will say this, though. I think you and I — I’m having an easier time playing devil’s advocate after using Claude Code. I’m starting to empathize more with why the non-doomers are imagining that AI is always going to be subservient to humanity, because now I’ve seen AIs get really far in terms of their capabilities, and they’re still so friendly and submissive.
The intuition — it’s hard to fight the intuition that forever AI will just be, you give it a task and it runs off, does the task really well, and then comes back to you and says, “Here you go, master. You’re the boss, not me. I’m gonna shut down now.” That intuition is growing with me even though I don’t think it’s gonna last forever.
Steve 00:08:27
Yeah. I mean, we can talk about a lot of different stories of how things start going wrong from there. People — you can talk about competitive dynamics and races to the bottom. You can talk about bad actors or careless actors, and so on and so forth. The scenario that is first and foremost in my mind is that the future more powerful AIs are really just quite different in their disposition than the AIs that we’re used to today because it’s a different AI paradigm than it is now.
Liron 00:08:57
I think a different paradigm is key. There is a little bit of an update happening for me, and don’t get me wrong, I’m not saying I’m a non-doomer now. It’s a slight update, but it is significant. I just want to make sure that I’m milking the most information out of this update.
I think the update is that it’s easier than I thought for humans to build agents that just do what they want the way Claude Code does, meaning work for a while then stop for the next command. The worldview that I’m losing is this whole idea that it’s gonna even be hard to make the AI make you a cup of coffee — that any task you want to do, the process of doing the task, it’s gonna want to go hardcore optimize the universe. Well, I think we found a regime where they’re super useful and they do a bunch of tasks, but they’re corrigible, they’re aligned. We’re in a really good place.
Steve 00:10:07
Yeah. I mean, you should be exactly as doomer as the objective situation warrants. We don’t need to all be really pessimistic to fit in with the cool kids. I happen to be more pessimistic than you, again, because of this paradigm shift that I’m expecting. I think there are other people who would sort of agree that LLMs are the paradigm that we should be worried about, and they’re more worried about it because of things like competitive races to the bottom and other things.
Liron 00:10:51
You are doing an alignment program, right? That’s part one of our conversation. I called it — it wasn’t your decision, but I called it in order to maximize views — “the man who could actually save humanity,” in the sense that you’re actually taking a shot at the goal. I can probably count on one hand the number of people alive who I think are taking a straight shot on the goal of being, “Okay, this is how AI is likely to play out, and it’s not dumb. It’s actually a plausible guess. And given that this is the plausible guess, what is some direction that might actually help do something that’s friendly to humanity?”
So you do have an alignment program, and as I recall, it’s something like you’re saying, “Well, here are some programs that could specify good goals for the reinforcement learning.” That’s kind of your alignment direction?
Steve 00:11:21
Yeah. So I work on the technical alignment problem, and basically, how do we get these consequentialist frameworks like RL and model-based planning to create something that’s not a ruthless sociopath, without just leaning on imitation learning? I think there’s an answer, because the human brain is built on these frameworks but nevertheless manages to avoid that outcome, and I’m trying to understand how it works in human brains and how it might work in future AIs.
Liron 00:11:41
So viewers, check the show notes. There’s gonna be a link to Steven Byrnes part one, one of the best discussions you’re ever gonna see on the show. Most of the other guests come on the show, and my job is just to expose how way off base they are. But in the case of Steve Byrnes, I think that he’s pretty much spot on. And remember, his P(Doom) is?
Steve 00:11:44
Oh, don’t make me say it. This is really annoying.
Liron 00:11:46
All right, let’s refresh viewers’ memories. They wanna see where you stand on the most important question.
Steve 00:11:51
P(Doom). P(Doom). What’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?
Liron 00:11:57
Dr. Steven Byrnes, what’s your P(Doom)?
Steve 00:12:00
I guess if I have to pick a number, I would pick ninety percent, but we can have all sorts of caveats about how to interpret that and what it means. And I do think we should all be energetically trying to make things better.
Liron 00:12:14
Wow. Well, yeah. You’re at the high end of the sane zone. I think that’s still a sane estimate. I think if you start going higher than ninety percent, you start getting a little overconfident.
Steve 00:12:22
Yeah. I mean, it’s hard to... Prediction is hard, especially about the future, as they say.
Liron 00:12:28
Exactly.
Steve 00:12:28
I like to talk about the movies where you’re three-quarters of the way through and the hero is trapped in an alternate dimension with no way back home, and the dastardly plan is about to come to fruition, and you’re just sitting in the audience eating your popcorn being, “Oh man, how are the screenwriters gonna get us out of that?”
Liron 00:12:53
Right.
Steve 00:12:53
So that’s different. If you’re in the movie theater, you can make a prediction that they’re gonna come up with something. Here in the real world, we don’t know what’s gonna happen, and it’s hard to predict. I have a hard time imagining things going well, but that’s partly about me and not just about the world. And obviously, we should all continue fighting for getting things to go well.
There’s this term “optimist” that simultaneously means “I expect things to go well,” and it also means “I have a can-do attitude, and I’m trying to make things better.” And I think we should all take the second definition of optimist as part of our identity and have a can-do attitude and try to make things go well regardless of our expectations about whether we’re likely to overcome the challenges.
What’s Your P(Doom)?™
Liron 00:13:42
Well said. Yeah, I agree that it’s a tangled web of vines that all seem like they’re pulling us down, and you would have to cut many of the vines if you want to rescue yourself.
So last time you were here, we talked about the before and after, the two paradigms. I know you called the second paradigm brain-like AGI, which is AGI that does this amazing power that the human brain does to learn really quickly and stack up learnings over a lifetime and see through — learning to drive, you don’t need a million data points. You just need a few hours of lessons, and then you’re good. And you called that brain-like AGI, and you think that’s the next paradigm that’s coming. And then what did you call this paradigm that we’re in now, just LLMs?
Steve 00:14:17
Yeah, sure. Foundation models, whatever.
Liron 00:14:20
The terminology is starting to be tough for me to keep in my head because I actually see — my own sense right now is that, and it’s kinda surprising — I see AIs actually on pace to steadily surpass the human brain, and yet they don’t feel fundamentally different from ChatGPT-3. I almost feel like the LLM paradigm is just swallowing the entire power of the human brain. What do you think about that?
Steve 00:14:42
I think that’s a place where I disagree. I think that LLMs have shortcomings that are not ever gonna go away. I used to say that LLMs would plateau, but I realized that was never really the right term because plateau implies that they don’t improve along any axis. Chess engines haven’t plateaued. They continue to get better at chess.
So I shouldn’t have said plateau. I should have said that there are axes along which I think LLMs are not ever gonna be as powerful as a human brain or as powerful as this next paradigm that I keep saying is gonna happen someday.
Liron 00:15:20
It just — when I talked to you before and when I was doing my own thinking, I felt like the wall of subhuman abilities would be stronger. So yeah, they’d keep getting better and better, but even this idea that they can just blast through a half-hour time horizon — the METR charts are very strongly indicating that their time horizon is multiple hours now, even fourteen hours, although the benchmark is starting to saturate. So we don’t know if we can take fourteen hours literally.
But it’s very clear that an AI doing something for half an hour right now is highly productive and can potentially replace a human working all day long. It’s very clear that we’re just about at that point, and we seem to not be slowing down. Don’t you find that kind of surprising that it’s already plowing through these all-day or multi-hour types of tasks? Isn’t that farther than you thought the LLM paradigm could go this fast?
Steve 00:16:44
Yeah. Well, I think I never really had a strong take one way or the other. I guess if somebody had put a gun against my head and told me to guess, yeah, I suppose I would be surprised about how powerful the things that LLMs are able to do. But I think that the further you get into novel territory that no human has ever done or that’s not on the internet, that’s not in any books that the companies didn’t commission specialized data for, the more that LLMs struggle with novel tasks in those domains.
“Brain-Like AGI”: The Next Generation of AI
Liron 00:17:01
But it just seems like novelty is not a particularly useful type of firewall anymore because they can program anything, and you can look back and say, “Oh, well, I guess that wasn’t novel to just program anything or program weapons and destroy the world. I guess none of that was novel.” I feel like that’s where we’re at.
Steve 00:17:15
Yeah. I mean, the — yeah. So there’s a lot of code on GitHub and on the internet and yeah, anything you’re doing, it can be pretty similar to the training data because there’s so freaking much training data.
Liron 00:17:23
So the cat is out of the bag on how to uncreatively destroy the world. How to not novelly destroy the world.
Steve 00:17:23
Right. Yeah. Okay. So there’s sort of a separate issue, which is maybe this is a wall that LLMs can’t get past, but maybe LLMs are still able to wipe out humanity. To people who say that, I’m like, “Great. Go work on LLM safety.” I never want to discourage people from planning for a world with more powerful LLMs or even today’s LLMs. I think that’s a perfectly good activity for people to be engaged in.
But then there’s this other issue, which is if it is a wall, and I think it is, then how do we think about exactly what is the nature of that wall and what will LLMs not be able to do that humans can do, and what are the implications of that?
Liron 00:18:23
And when you say other people can go work on it, maybe in your mind, you’re just not super worried because you’re thinking, “Yeah, LLMs can try to destroy the world, but then humanity can always fight it by pulling a novel defense out of our bag of tricks, and the LLM is probably gonna get stumped by that.” Is that kinda what you’re thinking?
Steve 00:18:23
I think LLMs would be unable to wipe out humans and run the world by themselves, for example. So that’s reassuring a little bit. I guess they could keep humans as slaves. I don’t know.
I think it’s more that I only have so much time in a day, and I need to focus on things, and I’m focused on this kind of superintelligence that I’m expecting to happen sooner or later. And God willing, we’ll survive enough to die in this more exotic way that I’m planning for. If people want to work on pandemic prevention, then that’s great too. If people want to stop nuclear war — yeah, maybe we won’t die from superintelligence because nuclear war takes us first. If somebody else is working on nuclear war prevention, then more power to you. I think that’s great. I think we need people to be on all these different problems.
Liron 00:19:59
Gotcha. So the only thing is, for the rest of this discussion, I tend to question the terminology of brain-like AGI versus LLMs or versus imitative. LLMs are just imitating tokens that they’ve already seen.
Maybe I can propose new terminology where we call it human-like AGI versus goal engines. I think you may be assuming too much when you say that the future powerful AGI is the brain-like AGI. I think you may be surprised to learn that even today’s AGI is already surprisingly brain-like and powerful.
Steve 00:20:24
Yeah. We should be clear on these two different definitions, and I’m happy to use whatever terminology you like as long as you define it. If we talk about what an AI is capable of doing, that’s an interesting question. And then there’s a different question, which is what are the algorithms that were used to create it and how do those algorithms work? Where do they draw their capabilities from?
Liron 00:20:35
Okay. Well, let’s try to use neutral terminology of current type AGI and next generation AGI. Because you think there’s a fundamental qualitative generational leap coming that hasn’t come yet.
Steve 00:20:35
Yeah.
Liron 00:20:41
And you’re saying it just hasn’t even come since the first GPTs. We’ve been in the same generation in your view.
Steve 00:20:41
Yeah. I think there’s an important difference, which is related to this ruthlessness post that you were gonna talk about. I think a good way to think about LLMs is that they’re primarily powered by imitative learning.
The way pre-training works, as many people know, is you show the LLM lots of data — tens of thousands of lifetimes worth of data — and it’s trained, at least in the language case, to predict the next token based on what it’s already seen. So then you set it up in a situation where it sees some context, and whatever it predicts would come next, that’s what it outputs. And then there’s post-training and there are all these caveats and nuances.
But I do think that’s a good way to think about how LLMs are able to solve problems more or less. They see situations that are similar to what’s come in the training data, and the humans often in the training data have some useful thing to do afterwards, and the LLMs do that useful thing. They’re drawing their power from imitative learning, is the way I would put it. And then that’s supposed to draw a contrast with how I think human brains work, which I think draw their power from reinforcement learning, which is kind of different, and model-based planning.
Liron 00:22:22
No, I think it’s a very reasonable hypothesis. I think you’re probably right. More likely than not, you’re probably right. I just do personally see how the power of agents today that are just based on what you’re calling the weaker architecture, the token-based or imitative prediction-based — this architecture that you’re saying is kinda hopeless to make the leap to this novelty or this true threat. I’m just getting weaker on that distinction just by empirically observing that even the weak paradigm potentially can make everybody unemployed.
Steve 00:22:39
I mean, I don’t know about everyone. We could talk about what is sufficiently similar to the training data that LLMs are actually good at it, and it turns out it’s a lot of things — almost all of the software that people want to make on a day-to-day basis. Is it all software? I don’t know. I’d say probably not.
If I invent my own complicated programming language over the course of a year and don’t put any of it on the internet, and it’s some weird paradigm that I kinda made up, I think Claude Code would be worse than me at using that completely novel programming language. And that’s a guess. I’m not a hundred percent sure. Or, take Olympiad math problems.
Liron 00:23:02
Right.
Steve 00:23:02
Famously, these are all supposed to be original problems, but actually there’s many tens of thousands of Olympiad math problems that people have published, and there’s only so many techniques that people can use to solve them. And LLMs have seen all of the solutions and all of the problems, and it turns out that they’re sufficiently good at generalizing that if they see practically any new Olympiad math problem, they are in fact able to churn their way through it, and maybe they try twenty different things but eventually find a solution.
So yeah, we need to have a broad view of similarity. But then I can give you a different idea. There are millions of humans over thousands of years starting, let’s say, at 3000 BC. We are on a planet, and no angels drop new training data from heaven. But nevertheless, we invented language and science and technology and the hundred trillion dollar global economy and everything in it all by ourselves.
I think that if you put millions of LLMs over thousands of years in a sealed box and didn’t give them any new training data — you gave them some VR environment to play around in — I don’t think the LLMs would be inventing new knowledge and building on that new knowledge in an open-ended way, the way that humans can. And of course, this is an insane example. Maybe LLMs can kill everybody without being able to solve that problem. But it’s a toy example to illustrate a deficiency, I think.
Liron 00:24:26
There’s kinda this horseshoe theory where I consider you one of the most insightful people in this domain. But a lot of the language you’re using right now dovetails with some of the least insightful people in the domain, in my view. If you look at the David Deutsch school, who I think is insightful in general but not on the topic of AI, he’s saying, “AIs can’t create knowledge.” And I’m like, “Are you sure about that?” Or there are some people who say, “AIs can’t truly reason.” And there are many people who, as you’re saying now, “AIs can’t do anything novel. They’re always finding similarity.”
I personally have always felt like that is super misleading to appeal to similarity. Yes, it was true about early GPTs, but I feel like we’re at this point where appealing to similarity — I don’t think there’s much value to be mined from it. Because one thing we’ve learned, one of the deepest things that humanity has ever learned, is this fundamental concept of reductionism and formal systems.
One thing we know about physics is that at the end of the day, everything is actually made out of just a few building blocks. So there’s a sense in which everything is similar to everything else. Everything is just part of the same building blocks or part of the same configuration space. And similarly, in math — you look at every domain of math, all these different textbooks, all these different theories, subfields, and yet they’re all constructed out of the same parts. They can all be formalized. If you look at proofs, every proof is formalizable. When you get from a claim to a proof of the claim, you’re always just cranking the same narrow set of possible rules. So it’s like, oh, you’ve already seen how to reason before? Yeah, everything is the same. So how much are you gonna be able to appeal to similarity?
Steve 00:26:04
Yeah. I guess I like to talk about putting a lot of complexity, building on complexity, building on complexity, and novel ideas that are only in the context window and not in the weights. I think that’s where you can stretch an LLM to its breaking point.
So this silly example that I talked about before, where you have thirty thousand years of human knowledge and science and technology — if you tried to put all of that in the context window and not the weights, I really don’t think LLMs would be able to do that. They can do some things in the context window, but I think the weights is where the bulk of their knowledge is. That’s the part that can scale arbitrarily up, and the context window can’t.
Liron 00:27:15
I see what you’re saying. So the context window — the weights are the combination of long-term and short-term memory. Or the point you’re making is that when they’re trying to reason, if there are a bunch of steps and they all have to happen in context, you might get to a situation where the context is overwhelmed. And that’s where humans shine, because humans can keep writing to longer term memory, and they’ll have a bigger buffer so they can do more complex things.
Steve 00:27:15
I tend to think of it more in terms of knowledge than memory. If you have all the information you need to come up with a new math concept — schemes or whatever — and you’re right on the verge of coming up with schemes, I think LLMs can likewise take existing human knowledge and perhaps they can also come up with schemes.
But what I don’t think they could do is then lock in that concept of schemes and play around with it for months and then start building on it, with it all being only in the context window and not the weights.
Liron 00:27:48
They can’t develop a field for a year and then use their new insights to do their next task.
Steve 00:27:54
Yeah, if you’re just keeping the knowledge of the field in the context window.
Liron 00:28:03
And then, as you’ve already conceded, that particular thing that they can’t do might just not be on the critical path to taking over the world.
Steve 00:28:03
I guess I don’t have a strong opinion about that one way or the other. I lean no. But yeah, time will tell. Certainly I hope not.
Liron 00:28:14
Which reminds me of — when I ask some people, notably the Deutschians who are always saying, “AI can’t be truly creative, can’t be truly novel,” then I’m just, “Okay, tell me something that a human would do in a typical year that you don’t think the AI can do. Be specific. Don’t just use this abstract word ‘novel.’”
And then they’re, “Okay, invent general relativity.” And I’m, “Okay, but you’re also a human, right? You and I are also humans, and we don’t really invent general relativity ourselves. So are you saying that the AI is now, if you rank it, it goes above almost every human and just below Einstein? Because I feel like that’s already good enough to take over the world.”
Steve 00:29:23
I mean, we use these examples of science because they’re hard for normal people and very impressive. But I think there are sort of more everyday conceptual spaces that people come up with that are equally in the area of concepts building on concepts. Packing a bag full of groceries — if you do it enough, you sort of get this gestalt sense of what’s a good way to pack different things. And you’re learning on the job, that’s legit continual learning, but it’s so mundane that we don’t think about it as the pinnacle of human knowledge.
Liron 00:29:24
Right.
Steve 00:29:31
And so, yeah. What does it take to take over the world? Yeah, I don’t know. Maybe I should just say I don’t know.
Liron 00:29:31
So based on this conversation so far, you’ve got perfectly plausible hypotheses about the thing that the AI is not doing that the human would otherwise do. It’s a very interesting topic of conversation, what’s left for the AI to do. And I think you’re probably onto something, this idea that the AI has done something every day for a year, but it didn’t take down that gestalt impression and update its weights. It didn’t do that, and so it’s falling behind, and a human would have done that.
But at this point, it just seems like the core of the AI is so powerful that we’re now potentially just talking about a harness. If you just have a meta process that watches AI and writes down some notes and then also kicks off another training run every few days, doesn’t that seem like it could just patch everything?
Steve 00:30:12
Let’s see. Another point that I would make is that part of our mental picture has to be the amount of specific effort that AI companies have put into coding. I believe they commission — AI companies spend billions of dollars a year commissioning specialized data that they can train on to make their AIs better, and RL environments that they can train on to make their AIs better. And they’re working really hard towards coding.
We notice how good they’ve gotten at coding, and we shouldn’t overgeneralize from that, I think. AI companies are not specifically targeting the set of capabilities that it takes to, I don’t know, take over the world, I guess. And hopefully, that means that they won’t be as good at that.
Can LLMs Ever Match the Human Brain?
Liron 00:31:51
All right. So let’s talk about qualitative jumps, because I feel like there have been a few insane qualitative jumps in the last few years, and there are another couple qualitative jumps coming, and then the world ends. That’s what I’m imagining.
Previous qualitative jumps — if you remember the 2010s and before that, we basically had narrow AI, like Google Translate. It would give you decent translations but didn’t truly understand the text that it was translating. You couldn’t correlate the knowledge and reason with it, so it was narrow AI.
And then we had general chatbots, the GPT revolution. And today we very much have general agents that are getting more and more powerful by the day and can run longer and longer and can really make stuff happen — really engineer whole systems.
Then pretty soon, it feels like we’re likely on track to AGI, meaning superior replacements for humans in most of the economy. Would you agree we’re probably on track for AGI in a few years?
Steve 00:31:55
That depends on how you define AGI.
Liron 00:32:13
What if I defined it as, let’s say eighty percent of human jobs that are currently on a salaried payroll making a livable salary right now — eighty percent of those jobs can just not be jobs at all and have the AI take them instead.
Steve 00:32:13
Hmm. I think I would take the under on that. That that’s not gonna happen. But I’m not extraordinarily confident one way or the other. It’s not a thing that I’ve thought about enough to have an opinion.
Liron 00:32:48
Well, maybe what you’re thinking is that if we really could do the full drop-in replacement — if the AI really could show up to work every day for a year — at that point, it must be doing enough learning over time, or it must have that extra secret sauce that today’s LLMs don’t have, that it might as well catapult us all the way into the next paradigm. So it wouldn’t just still be chilling in the current paradigm.
Steve 00:32:54
I definitely think the next paradigm would be able to do all human jobs.
Liron 00:33:02
Right. But you don’t think there would be a separate window where we’re all still living and chilling, except we don’t have jobs. You’re saying, “No, there’s no separation like that.”
Steve 00:33:02
I do expect that the order of things would involve catastrophe or utopia sooner than job loss.
Liron 00:33:09
Wow. That’s an interesting order of operations, yeah.
Steve 00:33:11
I like to talk about — there’s this quote from Eliezer where he says, “If you’re asking about the impact of machine superintelligence on the job market, it’s like asking about the impact on China-US trade patterns of the moon crashing into the Earth.”
Liron 00:33:27
Yeah, yeah, exactly. My mainline scenario is probably more — at this rate, I feel like I’m expecting a huge unemployment wave, but we’re still living and chilling because the current paradigm is gonna cause all the chaos, and it’s not a twenty-four-hour takeoff or anything. It’s just gonna be yep, a bunch of people don’t have jobs, and yet somehow we’re surviving and things are pretty good and productivity is high.
But then a couple years later, I basically think FOOM is the end game. So if I had to think about qualitative shifts — one happened with chatbots, where you don’t have to write anything anymore, they’re incredibly good at writing. Then one just happened with agents. It’s still in the process of happening, but it’s happened to an incredible degree. Yeah, you don’t really have to build stuff anymore. You don’t have to build software anymore. You just manage agents that build your software. And it’s not just building software, it’s making spreadsheets, making presentations. You really just have to give them a little bit of input here and there.
And then the next shift is, well, your whole job can be replaced. You don’t even have to sit there managing the AI. The AI can just take whatever your boss sent you, and you’re now out of the loop, and then your boss is out of the loop. They’re climbing their way — they’re taking tickets from the ticketing system. They’re grabbing the ticket and doing the whole task. They don’t need you to even do that part for them.
So I think the unemployment-style AGI is coming next. And then the shift after the unemployment AGI is the FOOM, where the AGI is like, “Hey, there are these things that the brain does that we haven’t unlocked yet. I didn’t even need this to do AGI, but I can do this, and I can kick off a recursive feedback loop, and now humans are really dead meat. Now they can’t even hold on to control of the world.” That’s my order of events.
Steve 00:35:00
Yeah, I mean, I think compared to you, I’m a little more inclined to emphasize that LLMs aren’t that great at writing. It’ll be interesting when it stops being easy to distinguish LLM writing from human writing, and then we won’t know if it’s an LLM or just someone who’s bad at writing. It’s nice that we still have Pangram Labs and stuff like that.
Liron 00:35:21
Right.
Steve 00:35:21
And let’s enjoy it while it lasts. The last time I talked to a lawyer about how LLMs were, he was not very impressed with them, but that was whole months ago, so—
Liron 00:35:34
Oh yeah, no, whole months ago. I mean, you can’t — because the coding agents were a lot worse a few months ago.
Steve 00:35:36
Yeah. But I guess I’m just saying I don’t know either way whether the remaining gaps are things that will fall shortly or are indicative of things where LLMs are just not fit for purpose.
Liron 00:35:55
It’s just that there’s now been a drumbeat where every couple weeks something major keeps falling. So they’re always finding something major to attack. I just feel like they’re gonna keep successfully attacking major things.
Steve 00:36:08
I mean, that’s a possibility. We’ll find out one way or the other.
Will AI Kill Us Before It Takes Our Jobs?
Liron 00:36:12
All right. Let’s talk about the country of geniuses in the data center, because now that’s Dario’s trademark quote. I think he’s the one who’s always saying that. Think about a country of geniuses in the data center. He thinks that’s what’s coming maybe two years, probably less than five years.
He’s talking about superior replacements for the biggest companies and nations. So even if we get an AI that’s as smart as a person, which I think is very plausible — as a software engineer, I think it’s very possible you’re gonna hire the AI instead of hiring the human software engineer. But okay, the AI will just be a really good software engineer, but it still can’t take on the entire US government.
But when you think about a country of geniuses in a data center, that becomes a drop-in replacement for a government or for the management of a corporation. Isn’t that where we’re headed?
Steve 00:36:53
Yeah. I mean, if you take “country of geniuses in a data center” literally, which I think Dario does — I think that I disagree with the prediction that LLMs can get there, and it’s for the reason that I was saying before.
If you take humans between thirty thousand BC and today, we invented all these new concepts, and the concepts are building on concepts building on concepts. I don’t think that’s a thing that you can do with the types of context windows and scratch pads and continual learning that we have in LLMs today. I think that requires a different paradigm that allows the weights to be continually updated and updated and updated with new, actually good knowledge in an open-ended way.
And I don’t think LLMs can do that at all. And I do think that if somebody figured out a way to do that with LLMs, they would find that they’re turning the LLMs into sort of less friendly and more ruthless kind of alien things than they expected from what you would expect from LLMs today.
Liron 00:38:40
Okay. So just to recap the Steve Byrnes worldview, you’re basically saying right now we’re still in a good place where unemployment is still very low — three or four percent in the US — and AI isn’t taking over the world and not causing harm, and it’s fundamentally lacking some secret sauce that the brain has. One day it’ll get the secret sauce, but we’ll never have a point where unemployment is hovering at twenty percent plus.
I think you’re saying that if unemployment is ever ten to twenty percent, that must imply that AIs have unlocked the secret sauce and we’re so close to FOOM and takeover and disempowerment. So basically, unemployment going past ten or twenty percent means we’re done. There’s no hovering there.
Steve 00:39:03
I don’t want to be pinned down on something that specific. It could also mean that there are jobs that involve less of the kind of building new concepts that I think humans are better at than LLMs, and that those jobs are not quickly replaced due to macroeconomic or regulatory or other issues. Yeah, I don’t have any specific number that I feel strongly about.
Liron 00:39:03
Gotcha. And then I think you and I agree that one way or the other, however it sequences with unemployment, there is going to be some next regime of vastly superhuman outcome-steering AI. And even though today’s agents are powerful, this next generation AI is just going to be so crazy powerful, like the human brain.
It won’t be like today’s agents where we look at them and we’re, “Oh wow, that was smart. What an interesting transcript.” The transcript will kind of be there, but it’ll be a thousand parallel transcripts written in a weird language that’s super efficient, and there are all these other considerations all factored in, and a human would take a lifetime to study one decision. That’s the AI that’s coming.
Steve 00:39:57
Yeah, I do think we’ll get there eventually, and the idea that there would be human oversight seems far-fetched just because there’d be too many AIs thinking too fast and perhaps also scheming against the humans to hide their true intentions.
Liron 00:40:43
Yeah. I have a very strong intuition about the fundamental factors involved here. I see a human brain and I’m, “Oh yeah. This is a cognitive engine. This is a goal engine. It’s a way to instantiate this type of system in the physical universe that has this kind of effect.” The same way a heat engine makes thermodynamic work happen out of heat — it moves energy coherently — that’s what the engine’s doing. That’s what the human brain’s doing.
But then I look at it and it’s, “Okay, but it’s just a nature design.” Nature designs tend to be pretty weak. You look at a bird’s wing. Okay, it lifts the bird up, it transports the bird, it lets the bird hunt or whatever. But if you look at a jet engine, we mastered the principles of flight, and now we’re in a whole other realm. The birds just don’t understand what flight could be when you strap a real lifting engine to it. So I qualitatively think that’s going to happen with human brains.
Steve 00:40:59
Yeah. I think that if human-level AI is possible, and I strongly believe it is, then human-level AI that runs a hundred times faster is possible too, and a million of them that are telepathically communicating is possible too. And that’s already pretty crazy superintelligence.
Country of Geniuses in a Data Center
Liron 00:41:34
So we both share this intuition of there will be another regime one way or the other, and that’s the end game. Let’s go to this. You recently posted — and I don’t think this was a super new post. It was more you doubling down, reiterating your position of why we should expect ruthless, sociopathic ASI. Because other people were coming to you with different devil’s advocate claims just saying, “Come on, Steve. Look how good alignment by default is. Everything is going great and Claude is so nice and Anthropic has Amanda Askell and the Constitution and the personality of Claude is so good.” Why do you still think that we should expect ruthless, sociopathic ASI?
Steve 00:41:42
Yeah. So this is definitely just another take on things that I’ve been writing about for a long time. I’m in the group of even more pessimistic people who think that technical alignment is a really hard problem that we don’t have a plan for yet, and that the fact that Claude seems generally nice is not evidence one way or the other about what we should expect for future more powerful AI.
We need to be thinking about what these future AI algorithms are, and in particular, if they’re making good decisions in novel domains, which by assumption they are, we need to ask the question of how is it that they’re making these good decisions. And in the post I offer two broad frameworks that I think are the only two that work in practice up to pretty powerful scales.
One is imitative learning. This is LLM pre-training, where it sees that in the pre-training data, in situation X, it tends to be followed by action Y when the human is doing it, and the AI says, “Okay, if I’m in situation X, then I will do action Y.” Or it doesn’t even have to think to itself. It’s just mechanically that’s what it does. Its expectations just directly turn into actions.
This is true imitative learning, a type of thing that does not exist in the biological world. If I’m imitating you, then I have to transform things that I hear and things that I see — which is my impression of you — into moving my larynx to make sounds and moving my arms for motor commands, and those are totally different things. We don’t have this kind of mechanical translation of expectations into actions the way that LLM pre-training mechanically turns a prediction of the next token into an output of that very same next token.
So that’s true imitative learning. I think that’s how LLMs get the lion’s share of their powers, and we can talk about post-training and other caveats, but I think that’s still the right basic picture.
Meanwhile, there’s this other way to make good decisions and get strong capabilities, which is basically consequentialism. There’s some end result that you’re hoping for, that you’re going for. The algorithm systematically chooses actions that lead to that result. Model-based planning is in that framework, search algorithms, and also reinforcement learning over time has these same effects.
I think the human brain has both. It has reinforcement learning — if I burn my hand on the stove, then I don’t want to touch the stove again. And it also has model-based planning. If I want to pick up the pot and not burn my hand, then I realize that I should put on oven mitts.
So humans — I think human competence comes from one of these kinds of consequentialist frameworks. And the issue, the way I see things, is that the imitative learning paradigm makes these more or less — sometimes they’re a little weird, but more or less you can get nice AIs that are not crazy sociopathic ruthless consequentialists. Whereas these consequentialist frameworks like RL and model-based planning, you get these ruthless consequentialist sociopathic results by default, unless you can invent some technique to get kindness in a consequentialist framework. And that’s of course what I work on for my research program.
Liron 00:45:01
Yeah. I’ve had an episode a long time ago on this show where I went over this slide deck by Jaan Tallinn where he had a toy model for AGI. It’s an agent that doesn’t even do reinforcement learning. It’s just doing a very simple search, and there’s a grid world, and it’s this agent, a robot that just has to roll to the finish and move some boxes while it rolls along and move the boxes to the finish. It just has to search for the best actions.
This simple presentation showed that if the human has the remote control with the stop button and the agent has this ability to roll over the human and get to the finish, the score does increase. Naturally, because then it can predict that it won’t get prevented from getting to the finish.
This is simple stuff, and it’s not implied by the particular personality of the AI. It’s just implied by the different search results, the search paths. You can find a search path where the human dies and you get more points. That’s just unfortunately true about problems. The nature of search spaces is that they don’t have any sort of nice personality to them.
Steve 00:46:02
Yeah, exactly. And this is why I think there was a lot more pessimism in the AI safety discourse in the 2010s — before LLMs came along — than there is today. Because back then everybody was thinking about model-based planning and everybody was thinking about RL. AlphaZero was in the news and the Atari-playing agents, and you could just look at them, and everybody who had worked with them, their lived experience was, these are kind of ruthless.
I keep using the word sociopath, and it’s not a technical term. It’s just an intuition that I think is useful here. So yeah, everybody had experience with these kinds of algorithms and they knew that they made ruthless agents, and that’s kind of what they expected for AI.
And then they stopped expecting that when everybody switched to these imitative learning frameworks. But I still think that if we want AI that goes way beyond imitative learning into sort of new fields of knowledge that don’t exist yet — AI that could run the world for a thousand years by itself and build a Dyson sphere — imitative learning doesn’t get there.
So people aren’t gonna be satisfied forever with imitative learning. They’re gonna keep working until they invent these more consequentialist ways of making powerful AI, and nobody’s really done that yet. RL agents of today kinda suck in many ways, but a solution exists and the human brain is an existence proof. And I think that sooner or later people are gonna get there.
Liron 00:47:22
I think a lot of people follow you so far. This is common knowledge. People have been thinking about this for a while. They accept, yes, instrumental convergence and all of this sociopathic stuff does happen when you’re doing reinforcement learning. It’s hard to deny if you really think about it. So I think people are on the same page there.
I think the popular thing now — a common way that I debate people — I think Bentham’s Bulldog, the debate went kinda like that. Debates with a lot of people are now going down this direction. They’re, “Yeah, yeah, reinforcement learning can be sociopathic, but see how we’ve managed to make these imitative AIs so nice? We’re just always going to have them along for the ride. We’re always going to have the nice AI, the friendly Claude, along for the ride. So even when we do reinforcement learning and Claude has the power to intuitively know what are these paths to get to an outcome, it’s just going to prune down the paths that are harmful.”
“The friendly part, the part that uses imitative tokens or whatever, it’s still going to be enough in control so that it’s going to bake in the nice personality into the combined system enough that it’s still going to be the driver. We’re still going to be nice for that reason.”
And then I turn around and say, “Wait a minute. It sounds to me like there really are two parts here. There’s the underlying engine, which I call a goal engine, analogous to the engine in a car, and then there’s the steering wheel. And you’re claiming that there’s always going to be some friendly token imitative system like Claude that’s holding the steering wheel, even though the car has an increasingly powerful engine.”
And what I usually say in that discussion is, “Well, once you build the engine, you’ve now built something incredibly dangerous. Even if you also attach this nice driver, best case scenario, the driver is really nice, the engine is going to be something that other people can copy without the driver or other people can exfiltrate whatever programming is just accounting for the engine.” What do you think?
Steve 00:49:04
I feel like you’re pointing towards a slightly different argument, which is if we know how to make friendly AI, then we also know how to make sociopathic ruthless AI, and the ruthless AI would outcompete the friendly AI.
Liron 00:49:18
Well, that’s actually not my point right now. I’m not even talking about competition between good drivers and bad drivers. I’m just trying to say that even in this ideal world where people are saying, “It’s gonna be fine because you’re going to have this friendly car. Yes, the car is going to have a really powerful engine, a really fast car, really powerful intelligence, but it’s going to be steered really well, and it’s just going to be one system, a well-steered powerful car.”
And I’m pointing out that actually, if you look at a car’s wheels, a car’s seats, all these different parts of the car are just designed to drive fast and drive anywhere, and only whatever is sitting at the steering wheel is designed to think about where the car should go. So I’m saying even in the ideal world, people’s ideal world is you have a good driver, but you’ve created something incredibly dangerous, which is the engine of the car. And I’m saying in that world, you’re going to have, one way or the other, copies of the engine that aren’t attached to a good steering wheel.
Steve 00:50:13
Yeah. I think the optimist response to that would be that the engine attached to the good steering wheel will help make this world safe from the engine not attached to a good steering wheel or something. And then that relates to offense-defense balance and all these other things. Maybe I’m misunderstanding you on that.
Liron 00:50:32
No, I think that’s right. But I think — so I’m just pointing out, okay, that’s the best case scenario. Giants that — these cars are more powerful than the human mind. So you have these giants more powerful than the human mind that have to battle each other, and you’re telling me that the result of that battle is going to emerge victorious with good situations for humanity, not too much terrorism.
Steve 00:50:50
Yeah. I agree that the story where the good AIs protect us from the bad AIs is kind of a dubious and scary story. But I also believe something stronger, which is that there won’t be any good AIs in the first place, at least not once the AIs become sufficiently powerful. So that’s an extra heap of pessimism on top of the normal level of pessimism.
Liron 00:51:08
Well, I actually agree with that too. So to me, the simpler argument is, okay, there’s gonna be a lot of cars. Maybe some cars will be good, but you’re not going to stop bad cars because the bad car design looks ninety-nine percent the same as the good car design. And even somebody making the good design, if they put a negative sign or make a small mistake, now they’ve got a bad car, and it’s as powerful as the good car would have been. So that’s my first argument — even your best case scenario is very precarious.
And then I go back to your argument, which is, how do you even train the good driver? Because even when you have this token imitator or whatever that wants to do its best — Claude is genuinely trying to follow its constitution and do its best — but the problem is that it’s, okay, all this data is coming in and it doesn’t have the architecture to synthesize the kind of huge arguments. It can’t actually steer as well as the engine. The car is overwhelming. The driver just can’t supervise the car. The control panel of the car is just too much for the driver. The driver can’t really anticipate where the car is gonna go.
Steve 00:52:12
Yeah. I mean, I think if you take — the way LLMs kind of exist today is you have mostly imitative learning. Pre-training and supervised fine-tuning is all imitative learning, and then you’re sort of sprinkling a little bit of consequentialism on top through this RLVR post-training process.
And I think if you’re very cautious about how much RL you sprinkle on top, then you can get some distance towards extra capabilities that way. But I think a country of geniuses in a data center over a hundred years — they’re not just sprinkling a little bit of new knowledge on top of their existing knowledge. Instead, they’re really rethinking things that they already knew. They’re really rewriting a lot of their preexisting knowledge and replacing a lot of their preexisting reflexes and intuitions, so to speak.
And the more that you do that, the more you have to say, what is the selection mechanism by which they’re overriding these sort of more mild pre-training inclinations with new data? And if the selection mechanism is ultimately rooted in consequentialism, then they’re just gonna be getting more and more ruthless as the country of geniuses continues to think and work over the years. Or if it’s not rooted in consequentialism, then I think that it just won’t work and they’ll go off the rails. They’ll get dumber and dumber instead of smarter and smarter.
Why We Should Expect “Ruthless Sociopathic” ASI
Liron 00:54:15
That’s so fascinating the way you describe it. You’re saying it’s a thin layer of consequentialism on top of these imitative agents. Because when they were trained, the LLMs, they’re, “Oh man, what’s the next word? I gotta get the best probability of the next word given my model of the text up to this point.” That’s how they emerge.
But then in post-training, it’s, “Okay, this outcome is happening in the real world. Do an action, do another action. Here’s the result of your action. Put that in the context.” So it’s this thin layer of finally lifting their head up and being, “Oh, consequentialism.” Driving outcomes in the real world. But most of the training hasn’t connected to that consequentialism, correct?
Steve 00:55:29
Yeah. Especially — yeah, so keep in mind, number one, that supervised fine-tuning can also help make LLMs more sort of goal-directed. But that’s still okay because the human is clearing that data and creating that data.
And then the second thing is RLHF and its cousins tend to make AI, the LLMs, friendlier, but actually — or at least the lore that I heard was that RLHF makes them kind of stupider. So you’re making them more helpful at the expense of being really sharp and understanding things.
And then the third part is this RLVR, and the important thing to note there is that even if you’re spending similar amounts of compute on RLVR versus pre-training, the amount of actual weight changes from the RLVR, I think, is gonna be much less because you have to do all these rollouts for one bit of data. Whereas pre-training, you get this giant river of data. There are sort of arguments about, if you have the right answer — the next token was supposed to be seven — that’s a lot more information than just saying the next token is not the thing that you said. Or the things that you did for the last five minute rollout is bad versus good. You get a lot less. It’s a trickle of data compared to pre-training, which is this flood of training data.
So if you put all those things together — yeah, people are — it is still true that RLVR can bring in a little bit of ruthlessness, and people have worked, especially in the early versions, like o3. The labs have kind of figured out how to minimize the damage from that little sprinkling of RLVR. But that’s still not gonna get us to the country of geniuses in a data center.
Liron 00:56:08
Let’s just explain what’s happening a little bit here. RLVR stands for reinforcement learning from verifiable rewards. That’s where you’re training an agent that under the hood is mostly an LLM, and you’re asking it, “Hey, what’s the next chess move, or move in this video game, or thing to write into my Excel spreadsheet?” or whatever. You’re trying to make a financial model in Excel or whatever.
And then at the end, we compile the model and we’re, “Oh, you got the model to compile. That’s worth a few points.” That’s the verifiable reward — you don’t even need a human to upvote it. You can have these objective criteria. Did your model compile? Stuff like that.
Steve 00:56:35
Yeah. Or did it get the right answer to the math problem and so on.
Liron 00:56:38
But now you’re saying this is also a difficult training loop that you can’t crank that much because it has to output so many tokens before you can ask whether they got the reward or not.
Steve 00:56:48
Yeah. So you can’t just compare the amount of flops used for RLVR to the amount of flops used for pre-training to get an answer to the question of whether the LLM is mostly getting its powers from pre-training or mostly getting its powers from RLVR.
There are also a few recent papers that seem to — I mean, I’m not sure how much to trust papers that come from outside the labs because maybe they don’t know the state-of-the-art stuff. But there is one paper that found that these RLVR models usually don’t come up with anything that is outside the top thousand. If you just try the rollout a thousand times, you tend to get the right answer that the RL models get on the first try, which means that they got about ten bits of optimization.
As opposed to pre-training, which is hundreds or thousands of bits of optimization in the sense that coming up with the right answer from a randomly initialized model is astronomically unlikely, one in a googol. And then there are papers that use different sampling techniques and other things and find that they can reproduce RLVR-type results without actually doing RL at all.
Based on that, I think there are really strong reasons to believe — or at least I don’t work in the lab and I don’t know the secrets — but as far as I can tell, we should really think of even post-trained LLMs as mostly getting their powers from imitative learning.
Liron 00:58:25
Now help me understand some terminology here. RLVR and just RLHF, reinforcement learning from human feedback — the effect of both of those things is to modify the weights that you got from the pre-training, correct?
Steve 00:58:26
Yeah.
Liron 00:58:26
Okay, so you’re just modifying weights and you’re pointing out the pre-training puts most of the substance into the weights and you’re not gonna change the weights that much overall, you’re only gonna tweak them. When we talk about RLHF and RLVR, do those both count as fine-tuning?
Steve 00:58:44
Oh, I don’t know. I would call them post-training.
Liron 00:58:44
Okay, they both count as post-training. And then fine-tuning, I guess, is post-post-training where you hand it off to a human user. Maybe that’s the distinction? Is it inside the AI company you do the post-training and then the human can further post-train, but at that point it’s called fine-tuning?
Steve 00:58:57
Maybe. There’s a different thing which is called supervised fine-tuning, which is just another variety of post-training where, instead of just picking out random books from the internet, you create text that is the way that you wish your LLM would respond and say, “Here, this is the right answer, LLM. If you’re in this situation, you should produce an answer that looks like this.”
Liron 00:59:49
So even when we do RLVR, we’re still kinda trying to import reinforcement learning back into that imitative, find-the-next-token paradigm. Because we’re basically trying to teach the AIs — and this goes all the way back to the first thinking model, right? What was it — GPT o1? The first time that we were doing thinking and we were saying, “Look, output a stream of thinking tokens, but make sure that those tokens really sound like how a person would think when given a consequentialist optimization problem.” That’s the approach.
Steve 00:59:57
Yeah, something like that.
Liron 01:00:38
Yeah, no worries. So I’m saying when you’re doing RLVR, you’re taking this larger idea of consequentialist reinforcement learning, which is this powerful force that has trained narrow AIs before, and it’ll probably train AGIs one day. But we’re just trying to import the essence of reinforcement learning into the imitative paradigm because we’re just telling AIs — this goes back all the way to o1, the first thinking model, OpenAI’s o1, a couple of years ago now.
When we’re saying, “Hey, here’s a thinking trace,” we want you to think about this hard math problem or whatever, and we want you to do it by outputting a series of tokens that really pattern-matches for how a human would think about this optimization problem.
Steve 01:01:09
It’s like, you tell GPT o1, “Try anything, do anything,” and we’re just — just try a bunch of things, and whatever ends up giving you the right answer to the math problem, we’re gonna tell you that that’s what you should have done, and you should do more of that in the future.
So that leads to things like noticing that something is on the wrong track and then changing tack, and it includes things like systematically trying strategies one after the other and all the other things that these thinking models can do.
Liron 01:01:32
But when you start imitating how people think, remember what I was saying about similarity? It’s like, okay, yeah, we think by chaining together all the different rules we know. There’s only a finite set of inferential rules. So if you can just think similarly to how humans think, don’t you then get the ability to do anything? And then maybe you’ll say, “Okay,” but then the context window stalls out. But isn’t that fundamentally a very powerful approach?
Steve 01:01:48
Yeah, I mean, we’ll find out just how far it goes as time goes on. And yeah, I was gonna say — you anticipated what I was gonna say — which is that I think LLMs are much better at imitating what humans can do over minutes, maybe hours, than what humans can do over days and weeks and months.
Post-Training & RLVR — A “Thin Layer” of Real Intelligence
Liron 01:02:32
Well, it sounds like you have two different objections here. When we started the conversation, your objection was more, “Okay, yeah, these LLMs can do a lot of the jobs of humans, but they’re not amassing knowledge. They’re not getting better over a few months, and that translates to their inability to be novel, because if you want to be novel, you have to collect all this learning from your experience and then synthesize it, and they can’t quite do that.” That was your first objection.
But then you came out with a second objection and you’re saying, “The paradigm of how they get trained is still mostly just done in pre-training, and there’s gonna be this future paradigm where they get trained differently.” But what’s your real objection here? Maybe the current training paradigm is good enough.
Steve 01:02:32
So from an alignment perspective, the important question is: if we continue to modify the weights of an LLM in a sort of open-ended way that can get really, really far from where it started — the way that humans are really, really far from a human of thirty thousand BC, the way that a country of geniuses in a data center would be inventing new fields of knowledge and building on them over the course of a hundred years if you seal the box, and coming out really, really different from how they started.
So if you keep modifying the weights, then the question is how do you figure out which weight modifications to go with? And the problem is there’s no training data because this new field of knowledge that your country of geniuses is creating, that new field of knowledge doesn’t exist. There’s no training data for it. They have to produce their own data. So imitative learning is out, and it seems that the only other option is something based one way or another, sooner or later, on consequentialism, where you keep the ideas that lead to good results.
And then my argument would be that the longer you crank that process, your country of geniuses gets more and more ruthless over time. And all of their inclinations to be nice get diluted away by just being ever more effective at whatever this objective is that chooses the new training data. I mean, it’s a little annoying to talk about this because I’m trying to make up how this continual learning paradigm would work for LLMs, but I don’t actually think it’s possible. And—
Liron 01:04:01
Hmm.
Steve 01:04:01
The people — if it doesn’t exist yet and people disagree about how it would work or if it even does work. But then this comes back to the capability side, where I really just don’t think you can get a country of geniuses in a data center in any real sense just from context windows. I do think you have to keep modifying the weights, and I think the capabilities and the alignment question sort of overlap on this question of where the new weights come from or where the updates come from for the new weights. I think you can choose alignment or capabilities, but not both.
Liron 01:04:48
Let me clarify this. You’re talking about the limits of why current AIs are gonna struggle, but are you foreseeing this new reinforcement learning paradigm where somehow the feedback loop does get closed and they can get feedback from the real world fast enough?
Steve 01:05:23
So humans are able to do open-ended learning. Over the past ten thousand years, we invented all this stuff — science and technology and the economy. And yeah, I think that comes from our brain architecture that’s firmly rooted in consequentialism, including model-based planning and reinforcement learning.
And so yeah, we are able to lean on that to get the kind of continual learning that I think LLMs don’t have and can’t have. But if LLMs could have it, I think they would become ruthless too.
Liron 01:05:50
So RLVR is just kinda weak because they have to generate all these tokens and eventually they get an outcome and they slightly update their weights, but it’s just not that many turns of the crank. It’s not adding that much information to the weights. But something that the future AIs are going to do — in the final generation of AIs where they go truly superhuman — they’re just always going to be updating models of the world. Anytime they observe anything in the world, that’s going to tweak their model. So they’re going to be sucking down so much information from the world.
Steve 01:06:29
Right. So my central guess is that LLMs cannot actually solve this open-ended continual learning problem where they invent whole new fields of knowledge when they’re sealed in a data center for a hundred years. But if I’m wrong, if LLMs are able to somehow solve that problem, then I would next say that the way that they would solve that problem is by somehow leaning on these kinds of consequentialist approaches like RL and model-based planning, and they would, as a result, gradually turn more and more ruthless because that’s just what these types of consequentialist approaches to AI naturally do to any model that they’re updating.
Liron 01:06:46
We should probably unpack this word “consequentialist.” I think it just means that the learning of a system, or how it gets shaped, is causally connected to the outcomes that it causes. That’s why we call it consequentialist. It feeds back the consequences into the structure of the thing.
Steve 01:07:02
Yeah. There’s something that I want to happen in the future, either directly or indirectly. And the reason that I’m making this decision right now is ultimately because of its impact on the future. It’s taking actions because of the expected consequences, just like you said.
Liron 01:07:21
And viewers have heard me say many times, this is a central point, that consequentialism is the central thing. It’s the magic power, and it’s really synonymous with goal steering. If you’re good at goal steering, it’s because you are a strong consequentialist. Is that basically synonymous?
Steve 01:07:23
Yeah. Yeah, that’s basically synonymous.
Liron 01:07:23
And the reason why biological organisms, especially humans, but even animals to some degree — the reason why they can function and have all these useful adaptations is because evolution hit on this consequentialist feedback loop. The survival of the fittest. Oh, you survived? Okay, I’m gonna make more copies of you. So that’s one early consequentialist feedback loop.
Steve 01:08:21
Yeah. Evolution itself gets its power from consequentialism. It makes changes, and it keeps the changes if the end results are effective according to the metric of genetic fitness. And then at the same time, evolution also eventually created brains that themselves have these within-lifetime learning algorithms, and those within-lifetime learning algorithms are partly based on model-based planning and reinforcement learning, these consequentialist approaches.
And that allows us to find good foods and find good mates and found companies and go to the Moon and all these other things that don’t happen by random chance.
Liron 01:09:04
Exactly. And now you’re contrasting consequentialism to the way LLMs work today, which is they’ve been trained to model what the next token is after a really big sequence of tokens. And yes, they have all these structured understandings of what they’re reading, so they’re not dumb, they’re not just doing statistics. We know that much. They really have a rich model.
But even though they have a rich model, when they’re thinking about what next word is going to come, they haven’t fully closed the consequentialist loop because they’re not thinking about what environment am I in, what is my user trying to do, what’s going to be the causally downstream effect of me outputting the next thing I’m going to output. The original models don’t think that, but they do then have a layer above them that starts to become consequentialist.
Steve 01:09:32
Yeah. You could argue that a modern frontier LLM has some sort of emergent consequentialism in the sense that within its chain of thought, it’s outputting tokens that will ultimately lead to the problem getting solved, at least better than chance. But the foundational architecture doesn’t really involve as much of the same kind of consequentialism. I think we should mostly think of them as being effective because of imitative learning.
Liron 01:09:54
So you just think that the future AI is going to somehow suck up learnings, self-improve really quickly from running a bunch of experiments or designing its own learning loop. You’re just expecting faster learning loops that are different than just reading the token, different from the pre-training loop.
Steve 01:10:38
Yeah. I mean, when humans figure things out, they certainly use real-world feedback, but also they’re pretty good at figuring things out even without that. Mathematicians invented the entire edifice of modern mathematics and pre-modern mathematics starting from nothing whatsoever and more or less in the absence of feedback. I mean, we could argue about the ancient Greeks measuring land or something, but basically they just had to figure it out based on what makes sense to them. Without getting into details, I would just say that this is intimately tied to the kinds of consequentialist AI architectures that are built into the human brain.
Liron 01:11:40
I don’t know. I’m just trying to think. I’m trying to get a mental picture for when is a time when I personally use my human secret sauce that a modern agent is so far away from being able to do.
First of all, there’s the immediate caveat that whenever I’m trying to do anything, it usually just means I’m trying to get up to speed on something that there’s already a textbook for. I moved into a new house, so I’m doing so much home improvement. I’m working with contractors to fix a million little things. Things are breaking from ten years ago. And it’s, okay, yeah, I’m just learning home improvement. That’s a well-known thing. I’m not doing anything novel. So we can throw that out right away because the AI is gonna come preloaded or it can easily reference all this knowledge.
So then I’m trying to think — sometimes I’m learning a skill, I’m learning to play the piano, and then it’s, okay, but that’s kinda just training my cerebellum, how to read music. I just feel like the AIs can gobble that up so easily. So I’m just trying to think — you mentioned this idea of inventing all of math, but that just seems pretty esoteric. Humans who are pushing the frontier of math. I’m just trying to think what is something that an ordinary human is doing that’s so special.
Steve 01:11:56
Motor control would be an example. A sufficiently fast LLM with access to a joystick and a weird-shaped robot body, and a month to practice, no RL — I think it would be pretty bad at that.
Liron 01:12:18
You’re saying no RL, but if you look at the companies right now, they’re making big strides in robotics. We keep seeing more and more impressive robots in different situations, and the way that they’re achieving that still seems relatively simple. Yes, they’re pushing the frontier, but at the end of the day, the playbook of how you do it seems pretty small and compact. So can’t the AI just use the playbook?
Steve 01:12:32
I don’t really know how — I guess I had sort of assumed that there was some hybrid of more traditional robotics algorithms with LLMs that these companies are using, but I haven’t been following the details.
Liron 01:12:47
This seems like it could potentially be a good experiment to distinguish whether your mental models are on the right track, because you’re kind of framing it now like, “Oh man, it would be so hard for robots to learn to do smooth actions.” But I feel like there’s a lot of progress in that.
Steve 01:13:23
Yeah. I mean, the issue is that it’s possible that humans and LLMs are both able to do task X, but humans do it in a different way than the LLMs. And the common example is that the LLMs — it’s sort of a question of already knowing how to do something versus figuring it out. When humans see a problem, they figure it out. And when LLMs see the problem, they’re more likely to more or less already know how to do it. Either way, you are able to solve the problem, so that’s great if all you care about is the problem getting solved. But it still points to some difference underneath the hood.
Liron 01:13:38
So it sounds like maybe the ultimate test of LLMs, the only way to not have to talk about super esoteric things like discovering the theory of everything, is to purposely delete a bunch of human knowledge and then really just test them on, okay, learn this new knowledge that we haven’t told you about.
Steve 01:13:54
Certainly we haven’t seen any evidence that the necessity of pre-training has gone away because LLMs have gotten more sample efficient. If anything, we’ve seen the opposite. The LLMs are getting better and better partly because their training data is getting better and better.
Liron 01:14:08
Yeah. Well, we’re seeing that they’re making more use of their context. It’s very common now where I ask the AI something and it’s, “Okay, hold on, let me get some context,” and then it searches the web, loads a bunch of context, and then it answers. So that does mean that it has to memorize less.
Steve 01:14:08
Yeah. I mean, that helps. I guess I’m not really sure — I think I’ve lost the plot of what we’re arguing about. Because, again, just because humans and LLMs do something in a different way — that’s an interesting point if we’re trying to understand LLMs, but it’s not an interesting point if we’re trying to understand what LLMs can do in the economy or something.
Liron 01:15:10
Yeah. Well, the discussion I was interested to have here is just that I want to know what it’s going to look like when the next generation of AI is here, and they’re doing the secret thing that the human brain can do, and they’re doing it even better. And I think you pointed out, well, there is going to be something like RLVR but with many more turns of the crank. More bits of information are going to turn that crank of some sort of feedback and self-modification or weight modification learning, and it’s something that humans do and AIs are going to do it. So I’m just trying to build a concrete picture of what that will look like. And you kinda said, “Well, it’ll look like a robot that thinks for a little while, experiments a few times, and then suddenly gets really good at a new dance or whatever.” That’s kinda what you’re imagining.
Steve 01:15:42
I mean, I do think that LLMs are not very impressive in their sample efficiency in the sense that they have ten thousand lifetimes of training data. And I think that the future AI will be able to do things with higher sample efficiency.
But the really special thing is doing things when you have zero samples — for example, mathematicians inventing math. That’s — so it’s not just about sample efficiency, but I do think sample efficiency is part of the picture.
Liron 01:15:56
Okay. Yeah, so it’s gonna be more sample efficient, and it’s going to have some kind of loop. RLVR is temporary in your view. There’s gonna be some kind — and are you purposely not telling me some ideas on this front that you have in mind?
Steve 01:16:05
Yeah, I don’t want to get into too many details because I don’t think we’re ready for next-paradigm AI yet.
Liron 01:16:06
Okay, so this question of what it would look like to have something like RLVR but that was actually able to give you a lot more useful information and update your weights — you feel like that’s the forbidden question.
Steve 01:16:19
I mean, the difference doesn’t have to be in the RL part. It can be in — it’s a whole system, and I think RL is very important from an alignment perspective. I don’t think the RL has to be unusually fancy in order to — and I think the RL is the secret of capabilities in the sense of — I guess this is confusing.
I don’t think that there’s a different kind of RL that’s really fancy and powerful. Instead, I think it’s mundane RL built on a better learning algorithm and world model or something like that.
Liron 01:17:00
Right. And I know you have used the term before — it’s an actor-critic reinforcement learning.
Steve 01:17:02
Yeah. Yeah. Something.
Consequentialism and the Path to Superintelligence
Liron 01:17:02
Okay. This is how I see the current age, right? Just getting back to the current age, where Claude Code is so useful and it stops — I’m not telling Claude to fetch the coffee and then it’s destroying the world. It’s just fetching the coffee, and it’s doing super valuable work.
I keep banging on this drum, but right now, Claude Code — I’m using the Max plan. It’s costing me two hundred dollars a month. If you told me that it was costing twenty thousand dollars a month, I’d be, “Okay.” Because quite frankly, it really is twenty thousand — that’s barely roughly the cost of one engineer in terms of engineering salaries. Would I rather hire one human engineer or just use Claude Code? I’d rather use Claude Code because Claude Code works faster and at similar quality. So I’m getting twenty thousand dollars a month of value for two hundred dollars a month. This is a good time right now. This is a golden stage.
So I have an analogy for the time right now. Tell me if you resonate with this analogy. We’re in the golden age of imitative AI. It’s almost like, humans have this engine, this engine has a kind of propulsion. It’s not really propulsion, but it’s making outcomes happen. But let’s make an analogy between that and rocket propulsion.
If you know rocket propulsion, you can just steer anywhere. You can go anywhere. You can drive around the galaxy. You can even go to other galaxies. So that’s the ultimate way to move yourself around — if you figure out rocket propulsion, or propulsion by equal and opposite reaction. One way or another, that’s the ultimate way to get around.
But we didn’t quite figure that out. Instead, what we have now is atmospheric flight. So it’s, “Look, I’m lifting. I’m lifting.” But you’re not doing it the ultimate way with rocket propulsion. You’re just doing it because you banged yourself against the air. The wings, the airfoil, are banging against the air particles and you’re getting lift that way.
And it’s, okay, yeah, that works great in the atmosphere and now you’re up there with the birds. You shouldn’t get too cocky because you don’t even have rocket propulsion yet. But it’s awesome to be flying around the earth’s atmosphere, and there’s a whole field of aerodynamics. It doesn’t really apply in space, but it’s very critical to the field of flight. And it feels like that’s where we are — this crazy middle atmospheric flight stage. What do you think?
Steve 01:19:07
Yeah, I think that’s a good analogy. I also like to talk about how becoming an expert in airplane design doesn’t necessarily help you build better rockets, and conversely, progress towards the rocket won’t necessarily — if you invent the heat tile, that doesn’t help you build better airplanes. In fact, it produces really nothing economically useful at all.
And that’s why I’m very concerned about the sort of AI research that’s happening on arXiv and in GitHub that nobody’s paying attention to. Everybody’s really dismissive of it because, hey, if it’s so great, why isn’t it making LLMs work any better?
Liron 01:19:47
Right.
Steve 01:19:56
And yeah, it’s for the same reason that building a component of a rocket engine doesn’t make airplanes work any better. It’s just because it’s a part of a different engine.
Liron 01:20:17
Exactly. And then people are talking about things like, ultimately, the airplane’s engine, yeah, it’s really powerful, but ultimately all it does is push the air against the wings, but it’s really the angle of the wings and the airflow over the wings that makes the plane lift. That’s how you get lift. And I’m, “Well, you know, you could also just have an engine that you point down.” It’s kinda the analogy that—
Steve 01:20:19
Yeah, it’s a different thing.
Liron 01:20:19
So there are these little analogies where people are seeing sparks of true flight or true lift because they’re seeing it in the atmosphere. And I’m, “Yes, you’re right, but the spark is going to be in a different configuration.” The pieces are going to be in a different configuration, and you’re going to get a whole other qualitatively different regime of lift.
Which is what you’re saying about outcome steering. You’re saying the post-training is a thin layer of consequentialism above this other pre-training thing that kind of works — it works to push a plane horizontally really fast when you’re in the atmosphere. That actually gets you flight.
Steve 01:21:02
Yeah. And it’s nice, in your analogy, that some skeptics who thought there’s no such thing as flight — you can say, “Well, look at this. It exists.”
Liron 01:21:05
Right, exactly. Heavier-than-air flight has now been cracked, yeah.
Steve 01:21:05
Yeah. I mean, there are some bad takes that have been obsoleted by LLMs. “Nobody will give powerful AI access to the internet.” “Nobody would — we’ll just use AI as a tool, not an agent.” I think people are saying that somewhat less. Of course, they’ve gotten replaced by a different set of bad takes that are overgeneralized from what we see right this second, in my opinion.
Liron 01:21:28
Right. Yes, exactly. Obviously seeing flight — some of the skeptics — if the doomers are saying, “We’re just gonna fly around the whole universe, that’s coming soon.” And they’re, “No, we’re not. We’re gonna stay on the ground. Nobody can fly higher than the birds.” And we’re, “Well, here we are flying right next to the birds.” And they’re, “Okay, fine, you’ll fly next to the birds. Maybe you’ll even fly just above the birds’ heads.” And I’m, “You see those stars? We’re actually gonna fly there. And the birds aren’t even close to there.”
Steve 01:21:50
Yeah.
Liron 01:21:50
And then some people are, “Well, seeing you get to the birds, I’m now not a skeptic.” That’s basically what you’re saying — some people are coming around, even though there’s a long way to go.
Steve 01:21:59
There you go.
Liron 01:21:59
And when Dario talks about the geniuses in a data center, I actually think — and I’m kind of shifting my default expectation here. I guess different from your expectation — I think we’re just getting close to Anthropic releasing — I mean, the same way that I would pay twenty thousand dollars a month to use Claude Code or OpenAI’s equivalent, the same way that we’re there, I think we’re probably going to get to, “Okay, here’s your genius. Your genius is your slave. Hopefully, it’s not conscious, so it can just do what you want happily.” And there’s a victimless experience. So just order your slave around, and it could be a robotic slave, a slave in the computer.
I think genius in a data center might accurately describe where we’re going — very supplicating geniuses, geniuses who are just happy to take orders in a data center. I feel like that is my mainline scenario. I think Dario might be correct. And to be fair, I feel like he has a hell of a track record of being correct so far. Is that fair to say?
Steve 01:22:56
Yeah. I think we should be impressed by Dario’s predictions about LLMs.
Liron 01:23:03
I know. Dario is probably one of the single best trajectory predictors to this date of anybody alive.
Steve 01:23:15
I guess we’ll find out what happens in the future. I already talked about what I think a country of geniuses in a data center could do that AIs can’t do. We don’t have to keep arguing about it.
Liron 01:23:15
I was just bringing it up again because to me, part of what I think is the golden age where we’re just chilling in the atmosphere and it feels so good to get off the ground — I think that genius in the data center may work out to just be part of this golden age where we get a regime of increasingly optimized human-like AI, analogous to increasingly optimized atmosphere-flying planes.
And yes, we get more and more unemployment, and yes, it’s gonna happen crazy fast, but it might only happen crazy fast in the sense of yeah, the economy grows exponentially, and then it didn’t grow three percent, it grew nine percent a year. But still somehow, it’s still okay. Yeah, there’s universal basic income. I think we might enjoy a few years where it’s just geniuses in the data center and it’s just tokens that don’t have instrumentally convergent runaway FOOM. Yeah, so we may have a golden age. It’s probably gonna be a jump from before unemployment to after doom. That’s where we disagree.
Steve 01:24:25
Yeah, yeah. No opinion on what our near-term LLM future will hold. I will find out one way or the other, and I’m happy for people to be paying very close attention to it and thinking about possible downside risks and trying to minimize them.
Liron 01:24:33
All right. So last question I have for you is where do we stand on FOOM, or recursive self-improvement? How do you think that plays out?
Steve 01:24:33
So I think I want to distinguish those two things. You can have a FOOM without recursive self-improvement. If you know how to make human-level AGI, then you can make it faster and you can make a million of them that are cooperating, and now you have superintelligence, but nothing about that necessarily involves any real amount of recursive self-improvement.
I think recursive self-improvement is a thing that could happen. I basically expect a rapid emergence of this next paradigm from almost nobody’s heard of it to radical superintelligence in a much shorter amount of time than most people seem to think — a year or two years or less than a year. But I don’t think that’s really much related to recursive self-improvement. If recursive self-improvement happened, it would be even faster, I guess.
Liron 01:25:23
Maybe your worldview is, look, I know there is a better learning algorithm out there, and it kinda has the flavor of what the brain does. I know you’ve used that term “actor-critic,” and we got into it last time exactly how that works in more detail. And maybe in your view it’s, look, once this algorithm gets unlocked as a better version of RLVR, just as a replacement — more consequentialism in the mix or whatever — once we do this gear shift, it’s just gonna be ridiculously powerful already. It is going to take it really far, and we’re going to have rockets flying around.
And then from there, to talk about self-improvement — I mean, the learning itself already is a powerful type of improvement. So that could just be the ultimate paradigm where you just throw data into it and you’re done. And the only type of improvement is pretty straightforwardly make some tweaks, throw more data into it. So maybe that’s why you don’t really think about things in terms of self-improvement.
Steve 01:26:08
Yeah. I like to talk about how the human brain — we have one human brain design just barely changed from the African savanna, and man, look at all the things that that one human brain design has been able to do. It built our whole civilization. And by the same token, you can have one AI design that can likewise do all those things, but more so and faster. And I don’t think we have that one AI design yet, but I do think we’re gonna invent it sooner or later.
Liron 01:26:38
I definitely agree that I don’t think there’s going to be a long series of designs. I can imagine there being a few more designs, but I don’t think it’s gonna be tinkering on designs a million years from now. I think it will have settled into, “Yep, this is the design, and here’s how you throw more data at it, and this is just damn near optimal. We’re good here.”
So there’s not necessarily that many iterations of the FOOM. The iterations probably just look like vacuuming up a ton of data and vacuuming up knowledge, doing whatever experiments you need to do and whatever thinking you need to do to get the theory of everything, and to understand how humans tick, and it’s, “Okay, I pretty much solved it. This is a solved problem for me, navigating the universe.”
Steve 01:27:19
Yeah. Once we have brain-like superintelligence, that’s about where my planning horizon ends because at that point it’s — they’re gonna be in charge of everything, and the future is up to them, for better or worse.
Liron 01:27:43
All right, so that is your thought on FOOM. Thanks so much, Dr. Steven Byrnes, for joining us to give this update. A lot has happened over the last six months. I personally expect even more will happen over the next six months, so I would love to keep bringing you back regularly and benefit from your insight.
Steve 01:27:45
All right. Always nice to chat.
Liron 01:27:55
All right. Thanks so much.
Doom Debates’s Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate. Previous guests include Harlan Stewart, Max Harms & Jeremy Gillen, Noah Smith, Vitalik Buterin, Carl Feynman, Robin Hanson, Gary Marcus, Jim Babcock, and David Duvenaud.