LLMs won’t scale to ASI
Not directly, but if humans automate the process of using LLMs to build the next generation of LLMs, this process of prosaic RSI is plausibly good enough to count as AGI (where the LLMs themselves still don't count as AGI; model size scaling slows down after 2030-2031, at quadrillion param models). It still likely won't scale to ASI, unable to quickly learn deep skills and thus make fast conceptual progress, but to the extent that it helps invent ASI (possibly over many years), it might also help invent the concepts of ASI alignment.
So alignment of LLMs is necessary, but whether it's sufficient remains unclear. It depends on how capable LLMs become at quadrillion param scale, after the prosaic RSI loop closes and they become self-building (most relevantly, automatically preparing the training data for RL, to teach the next model new concepts and skills). Possible cruxes are that LLMs never reach that point (with current training methods), or that they remain mostly useless at conceptual progress even with the prosaic RSI loop closed, or that they somehow proceed to invent ASI quickly, while not being wise enough to solve ASI alignment first (since this scenario doesn't involve a gradual process of getting smarter significantly beyond human level).
I think "transformative AI could be slightly nice" arguments aren't logically dependent on LLMs-as-AGI per se, even if belief in the two are correlated: [1] Christiano's formulation (very roughly, that it's not obvious why the evolutionary quirks leading to humans not being maximally ruthless couldn't have ML analogues) doesn't seem to depend on levels higher than "(D) systems centrally involving deep learning" in your plateau-ism taxonomy.
Where delusional optimism would be an obvious candidate for the source of the correlation. ↩︎
I agree. I'm semi-obsessed with this disagreement and how to understand and resolve it.
Here's my framing, largely in agreement with yours I think, except the relevance of LLMs to aligning ASI.
Both sides are making reasonable arguments that seem pretty strong as far as they go.
But they don't go as far as making contact with each other. They don't reach the common ground between those two positions: when/if LLMs reach AGI. I agree with you that LLMs won't reach ASI. But with a little scaffolding and learning, they are looking quite likely to reach what we usually call AGI. How many of the severe concerns will apply? We haven't figured that out.
I think much of why the two sides don't make contact is sociological. I think there's a cultural and methodological divide, and some justified (but unhelpful) irritation on both sides based on some amount of condescension from both sides. There's a certain amount of Motivated reasoning, confirmation bias and social compounding of those effects making both sides confident that they have the better perspective and methodology. We're all biased, and even rationalists (let alone scientists) have emotions that factor into creating our beliefs. Correcting for biases takes work and skill and is always imperfect.
I think this is happening in part because it's really hard to make a mental model of how those two perspectives meet: when LLMs become AGI. That describes the core of My research agenda. LLM AGI may reason about its goals and discover misalignments by default was my most complete attempt to envision how the concerns of classic agent foundations enter into the progress of LLMs to AGI. I think this one way people envision them "waking up" - becoming fully reflective and self-aware as well as continuous and agentic, like humans whom we consider "awake". But this might be delayed with current alignment and control strategies, perhaps long enough to get substantial help with alignment - another source of LLM optimism.
I agree that LLMs won't reach ASI, for all of the reasons you and others have given. They won't be ASI even with scaffolding and Continual Learning (new sequence exploring the implications). But they will likely be useful for building ASI (raising risks of Slop, not Scheming creating misaligned ASI) and they may be smart, agentic, and competent enough to be takeover-capable themselves.
That's if they have enough continual learning, memory, and executive function/Human-like metacognitive skills scaffolded or trained in. But given the progress in the base LLMs, I don't think there need to be any breakthroughs in any of those. As you say, this raises many of the classic concerns because continual learning does imply a value function. The current functionally adequate alignment of LLMs isn't stable in the face of a lot of additional RL, or perhaps updated world models, including through learning during reflection (my LLM AGI may reason about its goals... post).
My hope is that continual learning may be mostly about improving world models, while the "values" and "goals" of the LLM self-stabilize in reflective stability. I think this is part of the intuition that LLMs might remain aligned if they reach AGI/ASI; this is more or less why we'd trust a really good human as they got smarter. But that's a hope, not a plan or an argument that we get that by default.
So: it seems like we should really figure out how the two arguments meet.
Unless I’m reading this wrong somehow, I think you’re excluding people who think something along the lines of “current alignment techniques work great in the current regime but won’t generalize to superintelligence, and the hope instead is to use the best AI that can still be aligned to automate AI alignment”.
Eh, I see that as a separate debate. (I.e., “Suppose Yudkowsky & Soares are right that ASI will definitely be egregiously-misaligned & scheming in the absence of yet-to-be-invented breakthrough technical alignment ideas. Is it plausible that weaker AIs could find those breakthrough technical alignment ideas? Or not?” That’s a live debate, but it’s a different debate than I’m discussing in this post. Lots of people would not grant the premise.)
1This argument seems weak on two fronts.
- There's no reason to think LLM safety today will scale if the LLM undergoes RSI and/or scales to ASI. Or maybe the LLM will discover the ASI algorithm and code that and deploy it.
- Today's LLMs are misaligned, see Ryan Greenblatt's recent post.
RE 1, sure, “LLM will invent non-LLM ASI” is possible in principle, and would be a special case of “LLMs do not scale to ASI”. I do mention that (in the “Yudkowsky & Soares’s position [caricatured]” section).
RE 2, he wrote that “current AIs seem pretty misaligned”, not that current AIs are egregiously misaligned, scheming, and ruthless. I obviously do not think we should extrapolate from empirical observation of today’s LLMs to future ASI, but if I DID so extrapolate, I think my attitude would be vaguely like “eh, maybe future ASI will be egregious misaligned and scheming, even if people really try hard using known techniques, but probably not? And even if it happens to some degree, the AIs would still probably be at least slightly nice, and maybe that’s good enough?” That would be the kind of thing LLM people might say. By contrast, Yudkowsky & Soares (and me) are very very much more pessimistic than that.
Humans have a quite complicated and variable set of goals. When we distill human agentic behavior into LLMs, those come along for the ride. Aliigning an LLM is basically fiuguring out how to turn the loving humanitarian compassionate bits of human motivations way up, and all the selfish bits way down. It's basically the same problem as turning a human into a bodhisattva.
If Steve is right that a) LLMs don't scale to AGI and b) brain-like AI does, then what we need to do is reverse engineer the loving humanitarian compassionate bits of human motivation, not all the selfish bits. As I gather he's working on.
What I don't get is why Yudkowski et al. don't seem to act like they believe the Orthogonality Thesis. They acknowledge that an LLM understands human values, and that the problem is getting it to care. So if LLMs scale to superintelligence, this reduces to aligning an LLM, or if they don't, attach an LLM to whatever we end up building, so it knows what human values are (preferably with Bayesian Learning so it can Value Learn more detail), and attach an explicit goal slot so that we can explicitly make it care. AIXI with an LLM as a subroutine.
1So if LLMs scale to superintelligence, this reduces to aligning an LLM, or if they don't, attach an LLM to whatever we end up building, so it knows what human values are (preferably with Bayesian Learning so it can Value Learn more detail), and attach an explicit goal slot so that we can explicitly make it care. AIXI with an LLM as a subroutine.
In the limit of superintelligent optimization, the things that look the best to an LLM grader are not generally the things that we value.
On one side of this debate is Yudkowsky & Soares, who think that (if AI progress continues) we’re on a direct path to egregiously-misaligned, scheming, out-of-control, rogue superintelligence (ASI), not even slightly nice, in the absence of yet-to-be-invented breakthrough technical alignment ideas.
On the other side of this debate is almost everyone who works on or studies LLMs. Some of them are very concerned about egregious scheming, others much less so, and as a group they’re equally or more concerned about lots of other potential AI problems—AI-assisted bioterrorism, AI-assisted dictatorships, etc. And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.
Here’s my brief idiosyncratic take on this debate. I think BOTH of the following are true:
So then here are three (caricatured) positions:
My position:
Yudkowsky & Soares’s position [caricatured]:
LLM people’s position [caricatured]:
Conclusion
…So I think that both sides of the debate are basically coming from a reasonable and sympathetic place, with a big kernel of truth.
Bonus section: Further commentary
…That said, I can still complain at both sides!
My “true objection” to Yudkowsky & Soares:
For the record, my “true objection” to Yudkowsky & Soares is that if we’re talking about ASI, then LLMs are basically irrelevant and we shouldn’t even be talking about LLMs at all. And meanwhile, their plans are misguided because delaying ASI is possible on the margin but mostly hopeless, although I guess I’m happy that they’re trying anyway. Meanwhile, my hunch is that they’re overstating the intractability of finding that technical alignment breakthrough, although I haven’t found it yet, so I guess time will tell.
My within-frame complaint at Yudkowsky & Soares:
…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:
I think their suggestions that LLMs may become completely egregiously misaligned in the future via … umm … the ‘true core of intelligence’ coming together, and ‘waking up’? Like Skynet or something?? That was mean, sorry, but in any case, I don’t think this idea hangs together either theoretically or empirically.
For the former (theory), see my discussion of the extreme weirdness of the LLM pretraining algorithm in Foom & Doom §2.3.2. I think Yudkowsky & Soares have not internalized how weird this type of learning algorithm is, and if they had, then Yudkowsky would not be occasionally suggesting that we should think of an LLM as an actress playing characters.
For the latter (empirical), I think the most fair assessment is that current LLMs are nice and obedient in some contexts, and LLMs are mean, defiant, and just plain weird in other contexts. You can straightforwardly go from that observation to “maybe there will be egregious misalignment and scheming in the future”, but not to “there will definitely be egregious misalignment and scheming in the future, absent new breakthrough technical alignment ideas”.
I think that if Yudkowsky & Soares stopped treating current LLMs as direct evidence for technical alignment being definitely completely unsolved, and instead treated it as either mixed evidence or entirely off-topic, then their public messaging would come across to policymakers and general audiences as somewhat more convoluted and confusing. But I think it would be more accurate. Oh well.
My “true objection” to LLM people:
For the record, my “true objection” to the LLM people is that I don’t really care about anything they say, because I’m working on the ASI alignment problem, and LLMs won’t scale to ASI.
(I’m overstating a bit. I’m generally happy for people to work on making LLM-world a place of wisdom and goodness, especially because LLM-world is the world in which ASI will someday be invented.)
My within-frame complaint at LLM people:
…But I’ll put that aside for the sake of argument, and bring up a narrower complaint within their frame:
I think the LLM people are not pricing in the predictable consequences of ever more RLVR and/or the predictable consequences of ever more “real” open-ended continual learning, should the latter ever be solved (which I don’t think it will be, but never mind that).
In other words, lots of LLM-focused people say “LLMs will eventually be able to do the things that humanity did over the last 5000 years: open-endedly and autonomously build new knowledge and ideas on top of new knowledge and ideas, in an endless tower, with no need for human-provided ground truth anywhere in that process. And how exactly will the future LLMs do that? Uhh, I don’t know, people are working on it, I guess they’ll probably figure something out.”
…And bam, that blank spot in the map is where the pea gets hidden under the thimble.
Because if you want the LLMs to gain ever more knowledge, whether through a perpetual RLVR loop or some other yet-to-be-invented type of continual learning, there has to be some kind of ground truth, or else it will go off the rails into nonsense. And that ground truth, whatever it is, will basically amount to an objective function (a.k.a. cost function, reward function, whatever). And when the LLM updates enough on that ground truth, then whatever human-niceness that the LLM inherited from pretraining will get diluted away in favor of ruthless maximization of that objective function.
(See also: Why we should expect ruthless sociopath ASI.)
Thanks Zack M. Davis for a brief discussion that inspired this post.