On the Hanson–Yudkowsky debate, local vs. global intelligence explosions, “content vs. architecture,” and what the old arguments predicted about modern AI
This began as a Twitter/X thread after I read and tweeted about the Hanson–Yudkowsky AI–Foom Debate. Eliezer Yudkowsky joined the thread to object to my interpretation of the debate, and we ended up having the exchange reproduced below.
I’ve preserved the dialogue verbatim, except for paragraphing, fixing obvious [typos] and expanding links. I’ve removed unrelated replies and moved a few pieces of context into bracketed editorial notes. Nothing has been rewritten for substance.
Context
Aashish Reddy:
I have now finished reading The Hanson-Yudkowsky AI-Foom Debate, which is basically 60 blog posts from Yudkowsky and Hanson over ~500 pages, a transcript of their in-person debate at Jane Street, a (good) summary by Kaj Sotala, and Yudkowsky’s ~100 page paper on “Intelligence Explosion Microeconomics”. I will take questions from those who do not wish to subject themselves to this. I judge the winner of the debate to have been Carl Shulman (whose contribution was two blog posts and a few feisty comment exchanges)
Sophie Bücker: so uh why was carl the winner
Aashish Reddy:
One of the key points Hanson kept making was that Yudkowsky was over-reliant on abstractions he had come up with himself, rather than the “vetted” abstractions developed in the relevant academic fields, such as economics, and in particular, the endogenous growth literature. Yudkowsky countered that economics treats agents as “black boxes” — so the core mechanism he’s interested in, about how agents might make themselves smarter by fiddling with their cognitive algorithms, is assumed away (True Sources of Disagreement) In particular, Hanson expected that brain emulations (ems) would be developed before "hand-coded" AIs, leading to a change of growth mode (economy doubling on the order of once a month rather than every 15 years), which would be "just" [a] big change the way farming or industry were; the economy would "settle into a new faster growth rate", and further AI developments would still take a while to come. Carl Shulman (first in the comments of Emulations Go Foom, then in his own post) illustrates that even in Hanson's scenario, the necessary components are present for an intelligence explosion: ems would be able to run at fast speeds, earn lots of money either on the market or via government projects, acquire more hardware and use their vast number and speed to make research advances to improve software, and make themselves go FOOM. Yudkowsky concurs there that "Carl Shulman has said much of what needed saying." This succeeds in showing that Hanson's attempt to bracket Yudkowsky's questions doesn't work, and that he needs to argue on that turf — which he never really does (Foom Debate, Again), [to] the constant continued consternation of Yudkowsky (Not Taking Over the World).
In doing so, especially in his own post (Brain Emulation and Hard Takeoff)[,] Shulman relocates the recursive feedback loop from a single mind improving itself to the AI R&D process itself. This picture is much more borne out by reality than Yudkowsky's, who seems to have been wrong about one of the core parts of the debate — the extent to which the intelligence explosion would be a local vs global/decentralised process. We did not get a brain in a box in a basement that learns deep insights about the structure of cognition and uses that to restructure its cognition and go FOOM; we got AI products that are able to make money to acquire hardware which enables an effective increase in the research population which leads to further progress which leads to increasing intelligence. This has a handful of players, not one cracked team, it wouldn't be possible to do this in secret, it requires this expansive global supply chain, and so on. So Shulman exposes the flaws in the Hansonian picture (by saying that standard economic considerations about investment and returns generate recursive feedback once cognitive labour can be cheaply copied and directed towards improving the technology producing the cognitive labour), while being much closer to the mark than the full Yudkowskian one ended up being (by getting the unit of recursion right — not a mind, but an AI-development system).
Thus I declare Carl the winner!
What counts as “local”?
sullyj3:
[Yudkowsky] seems to have been wrong about one of the core parts of the debate — the extent to which the intelligence explosion would be a local vs global/decentralised process.
I don’t think this is yet falsified. Superhuman artificial researchers aren’t here yet
RSI in the purist sense hasn't started yet, notwithstanding how people talk about vibecoding being the beginnings of it. I don't think that ought to count, centrally
Aashish Reddy:
I would say another thing that surprised me is the extent to which one could claim "the jury is out" wrt this; obviously we have not had an intelligence explosion yet. Nevertheless I still expect this to look more like the way Carl Shulman predicts (e.g., as reiterated in his Dwarkesh interview) than the way Yudkowsky predicts.
Certainly Shulman has needed to modify his model less to account for recent developments than Yudkowsky has needed to, and certain bits of Yudkowsky have already been falsified — what I meant about the local/global thing is that the brain in a box in a basement seems certain not to happen, and that I would still not regard it a victory for him if Anthropic can do a "local" intelligence explosion in a 2026-updated sense.
Agree re "RSI has not started yet" but I am making claims robust to that: [see the exchange with corsaren in the appendix]
Eliezer Yudkowsky:
If Anthropic sparks off RSI local to Anthropic, that is, besides being human extinction, also a decisive victory for the Yudkowskian crux of the debate. The Hansonian/EA view was that RSI had to loop in and run through the global economy, so we’d be safe.
I think there is an actual problem here where the anti-Yudkowskian view sounds so stupid in a modern sense that you are unable to read it.
Aashish Reddy:
As I say about the Cyc stuff [in the other branch, reproduced in the next section], there is some stuff which I agree has aged so poorly that I skate over it somewhat. But again, on a spectrum of "how local vs global is the intelligence explosion", you'd surely agree that reality turned out more global than Yudkowsky2008 was arguing?
What I meant there about a "local intelligence explosion in a 2026-updated sense" is still not the Yudkowskian, "They make the foundational discoveries that MIRI was trying slowly to make, and use that to restructure their own cognition, and go FOOM". I agree if that happens, that's landing extremely close to Yudkowsky. That's what I meant by "one could claim the jury is out" — is that still how you expect things to go from here? If it is, then fair enough; we'll see.
But my actual expectation is that if Anthropic pulls of[f] an intelligence explosion in the coming years, it will still involve a lot of hardware/compute, it will still require this global supply chain and so on. Which again, is closer to Shulman than Yudkowsky (and Hanson, for that matter. You're reading me here as being far more supportive of the Hansonian position than I am, I think).
There was really not much in that debate about human extinction. Like, I think Shulman is like 20% on catastrophic outcomes from AI? I don't think that's the thing that makes us safe; my views here are not really things that were contested in the FOOM debate. I was surprised reading it how much of it was about local vs global stuff! I agree that isn't super important, though maybe it seemed more important then because of the prospects of you building a Friendly AI or something?
But looking out at reality, I do not think now I would say, "Yes: it may not have taken the form of a brain in a box in a basement, but that was substantively right". I would say it was less local than you were arguing.
Content versus architecture
[In another part of the thread, sullyj3 argued that current approaches might be far from exhausting what is possible with our compute budget, describing text prediction as an “extremely bizarre route to AGI”. My reply prompted this exchange with Eliezer]
Aashish Reddy:
I agree it remains logically possible, but as far as I can tell, nobody is still arguing for it, and Yudkowsky has conceded defeat. He claims that he was mistaken because he underrated evolutionary algorithms; as I said to him [in an earlier exchange about neural networks][1], I think he was mistaken because of another spiritual Hanson victory in this debate — underrating the importance of content relative to architecture.
Of course, you need the right architecture (transformers etc) that let you learn the content, but what the text prediction does is let you learn models of the world: first it gets a map, then RL teaches it to search through possible paths. I agree we will likely have an AlphaGo →AlphaZero style improvement here, but I can’t see anyone really arguing that this suboptimal thing isn’t the basis we’ll be starting from. And this cuts against the Yudkowskian model (related reasons are also why I disagree with him on doom)[2].
Eliezer Yudkowsky:
This is just anachronism! You hear Hanson talk about “content beating architecture” and you think surely he must be talking about data; what else could he be talking about? He means proprietary handcrafted inputs and code! A la his “I <3 Cyc”!
“Architecture”, in Yudkowskian lingo, is the opposite of handcrafting all the content! It means: Something where you can just pour in data, and it learns from that data!
The Hansonian view is then relevant to larger issues because it implies that there is no 'general AI'! There is no big company that is the best at 'AGI' and could spark off ASI internally! The whole global economy participates in handcrafting different specialized AIs!
I thought there'd be a complicated understood architecture that let AIs learn from data without added human labor! Reality declared there was a simple opaque architecture that could learn from data! Hanson thought we'd be safe because there would BE NO ARCHITECTURE FOR AI!
Aashish Reddy:
This is kind of true, which is why I added “spiritual”: condensing for simplicity, the Hansonian vibe was “Look, the thing that makes human minds special is that there’s a lot of stuff in them, like how to farm and write and do industry, and then architecture is this background thing that lets us use it”, and the Yudkowskian one was “The thing that makes minds special is the stuff they have that chimpanzees don’t, which is surely not these pieces of content coded in, but new cognitive algorithms not present in the architecture of chimp brains”.
I agree the Cyc stuff was confusing and weird but I do think spiritually current systems lie in between these positions, to the Hansonian end. “We got the knowledge in there by adjusting weights on a neural network based on a loss function of predicting text on the Internet” is very different from what Hanson was endorsing there (“we’re going to code in specific sentences”?). Very wrong means but basically right on “it has to learn stuff” first.
You see I give the nod here to Shulman not Hanson: as he says, this can then be copied many times and increase research effort and so on. But your thing, as I read it, seemed to imagine something more like AIs that would form their own ontological categories on encountering the world, by using cognitive algorithms they’d developed, or something like this. Hence they wouldn’t be able to share much, and so on. That seems substantially different to “Actually they just read the entire Internet in a way that lets them understand what we’re talking about”.
Eliezer Yudkowsky:
Again: Anachronism. The Yudkowskian thesis is that what AIs learn from data end up being local to that AI and to that company, rather than being saleable Hansonian Cyc-content that gets bought and combined into AIs no single company could ever afford to build.
The translation of the Hansonian vision into modern terms would be big companies trading single LLM layers around, that get assembled into an AI that no single firm could afford to make, which is why it's impossible for RSI to happen inside one firm without the global economy.
The Hansonian thesis is so alien to the modern experience, which is in reality more Yudkowskian than the original Yudkowsky, that you are no longer able to parse his words in terms of the world you know.
Aashish Reddy:
As I say here [in the previous section on “What counts as local?”]: "You're reading me here as being far more supportive of the Hansonian position than I am, I think". I explain why in Point 1 above. So sticking to the Yudkowskian position. I am thinking here of claims like the following:
"I wouldn’t be surprised to find that, in a real AI, 80% of the actual computing crunch goes into drawing the right boundaries to make the actual reasoning possible" (p.44)
"If two AIs both see an apple for the first time, and they both independently form concepts about that apple, and they both independently build some new cognitive content around those concepts, then their thoughts are effectively written in a different language. By seeing a single apple at the same time, they could identify a concept they both have in mind, and in this way build up a common language . . . the point being that, even when two separated minds are running literally the same source code, it is still difficult for them to trade new knowledge as raw cognitive content without having a special language designed just for sharing knowledge." (p.278-9)
"There are more barriers to trade between AIs, because of the differences of cognitive architecture [...] and there is very little sharing of cognitive content even in ordinary AI." (p.186)
I say plenty of sympathetic stuff about the Yudkowskian position here [in the appendix], and I think it was far more right than the Hansonian position on net. In my original reply on the thread you’re replying to, you’ll see that on my interpretation, Shulman gives a decisive (Yudkowskian!) argument against Hanson’s position, or at least, challenges him to argue “on [your] turf”, which he never does — at least not to any reasonable degree. But the way in which his framing was at odds with yours is not a decisive refutation in the same way; it’s just turned out to be closer to reality. I thus find it a bizarre claim that “reality [is] more Yudkowskian than the original Yudkowsky”!
Eliezer Yudkowsky:
“If two AIs both see an apple for the first time, and they both independently form concepts about that apple, and they both independently build some new cognitive content around those concepts, then their thoughts are effectively written in a different language.” = NEURALESE!
What this means is that the "cognitive content" ends up as NEURALESE, which you cannot trade around between different companies, unlike Hanson's Cyc-content, which he thought would have [to] be built and combined by many different specialized companies with no one AGI!
Aashish Reddy:
Does neuralese make it “difficult for [AIs] to trade new knowledge as raw cognitive content without having a special language designed just for sharing knowledge”? I’m confused by the claim here. Of course different AIs store/encode the concepts associated with “apple” differently in their weights. There’s still plenty of room for knowledge transfer between AIs though...
Eliezer Yudkowsky:
HANSON DID NOT KNOW ABOUT WEIGHTS. THIS IS BEFORE DEEP LEARNING TOOK OFF. Hanson thought there would be human-readable data handwritten by the company associated with the apple concept. I WAS ARGUING THE THING YOU TAKE FOR GRANTED.
His position is not "AIs can talk to each other". I would've agreed with that claim. His claim is that COMPETENCE WITH APPLES takes the form of HUMAN-HANDCRAFTED DATA that gets TRADED AND ASSEMBLED BETWEEN COMPANIES.
For God's sake read "I <3 Cyc" and then look up how Cyc was being built. IT USED NO DEEP LEARNING. IT HAD NO WEIGHTS.
Aashish Reddy:
Do you think I think Hanson foresaw deep learning? I am pretty sure I didn't say this. But just in case I'm being unclear: I do not think Hanson saw deep learning coming or had a proto-concept of weights or anything in this vicinity.
I read "I <3 Cyc" and was as bemused by it as you were, and think/hope that had I not been 3-4 years old at the time of this exchange, I'd have been equally as bemused then as well. I agreed with your point that substituting the words for other words would throw a wrench in its "knowledge" and so it's hard to see how anyone could call it an AGI project. Of course, Hanson's broader point was that "knowing lots" is important and so AI developers should figure out some way of getting lots of content into their systems, and we can say he was right that this was important and comically wrong re how we ended up getting knowledge into the systems. It is very obvious that Yudkowsky won the Cyc-exchange!
However the quotes I mentioned suggest that AIs that independently form concepts would struggle to share knowledge “without having a special language designed just for sharing knowledge”, and this creates barriers to trade. I do not think this turned out to be accurate, even given neuralese?
Models all form concepts by predicting human text, so their ontologies are in the relevant sense anchored to ours. There is no special language of the kind you claim necessary; it’s just natural language! This is why knowledge can leak between labs (eg via outputs/distillation) even if they’re not trading weights. Or to put it another way, you were right on “internal representations are local”, but this doesn’t imply “knowledge is local”, and the second is the relevant one for the local vs global debate. And I think the reason is directionally Hansonian: learning from the Internet means importing a _shared_ontology.
But like, set aside Hanson! I don’t think he won the debate! I mostly read the book as you laying out your position, and Hanson occasionally nudging you to 1) Make your argument clearer, 2) Explain at a meta-level what you’re doing and why you’re licensed to use the abstractions you are, 3) Respond to some object-level pressures he put on your arguments.
I don’t think he laid out an object-level argument of his own that was anywhere near as fleshed out, and he got some important object-level things wrong. And so I’m hardly saying “Hanson was right” (as you keep attributing to me!); I’m saying, “In places where Yudkowsky was wrong, we can illuminate that via the pressures Hanson put on his argument”.
What would vindicate the original picture?
Aashish Reddy (continued): Now I thought you might reply that just because the course of events so far doesn’t look Yu[dk]owskian, that doesn’t mean it won’t become so. I am genuinely surprised to hear you say that “reality [is] more Yudkowskian than the original Yudkowsky”. If you claim I’m misunderstanding your argument, perhaps you can help me clarify your actual position by answering these:
Do you still think “there’ll be a discontinuity at the point where the AI understands how to do AI theory”? Does your concern about hard takeoff still route through the kind of AI that could rewrite pre-eminent AI textbooks, and thereby acquire deep insights into cognition that enable it to “fully swallow its own optimization chain”?
Do you expect an intelligence explosion inside one lab to still [rely] on chips, fabs, energy, capital, etc., bought from the global economy?
How many AIs were involved in the Hugging Face incident: one, a handful, or hundreds? Could such an incident have happened via models from different labs cooperating with each other?
[The exchange ends with these questions. I would welcome the thoughts of others!]
Appendix: What do we mean by FOOM?
[This was a separate branch of the original thread, referred to several times above]
corsaren:
What definition of FOOM is even being used here? I.e., are we talking the superexponential/bounded-time singularity stuff?
Also, longer question: one of the biggest objections I’ve had to RSI/FOOM theories as of late is the idea that “improving capabilities” is inherently recursive. For things like raw processing speed, sure, but idk how far that gets you on its own, and so much of intelligence seems to be about (spiky) knowledge of the world and building the right mental models, and improving in that way doesn’t seem obviously recursive.
E.g., I can get better at proposing new drug molecules, but that doesn’t make me better at getting better at proposing new drug molecules, and it certainly doesn’t make me better at getting better at business negotiations. Curious if this came up at all as I’ve been meaning to read through to old literature to see if this objection has already been well-addressed.
Aashish Reddy:
From the foreword: "whether generally intelligent AIs will be able to improve their own capabilities very quickly (a.k.a. “foom”)["]. Yudkowsky later says: "Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades".
I was reading this in part to write a blog post on similar topics. I would say that one thing that surprised me is the carefulness with which such distinctions were in fact made/kept — Yudkowsky2008 was, in this respect, much more precise than much of the discourse you see band[i]ed about these days. In particular, he distinguishes cascades of improvement from cycles of improvement from recursive improvement (Cascades, Cycles, Insight..., Engelbart: Insufficiently Recursive) and improvements to the meta-cognitive level vs the cognitive level vs the meta-knowledge level vs the knowledge level vs the object level (...Recursion, Magic). You don't have to read those in full, they're re-summarised in the RSI post: Recursive Self-Improvement
Yudkowsky considers the key variable "returns on cognitive reinvestment", or at least, that's how he frames it by the time of the "Intelligence Explosion Microeconomics" paper (which is a few years after the debate, and I think he's more "mature" by that point — he's absorbed [many] of the good Hansonian points and has good arguments for discarding the rest). He has this analogy to the point at which a uranium pile goes from subcritical to supercritical, i.e., when the effective neutron multiplication factor k goes to k > 1. So the paper considers possible regimes for k, understood as the returns on cognitive investment: k < 1 (“the intelligence fizzle”); k ≈ 1+ ("the intelligence combustion", which he takes to be Hanson’s view); and k ≫ 1 ("the intelligence explosion").
In the comments of this post zooming [in] on what is actually meant by recursion (Sustained Strong Recursion), Yudkowsky says: "most AIs won’t hockey-stick, and when you fold a function in on itself this way, it can bottleneck for a billion years if its current output is flat or bounded. That’s why self-optimizing compilers don’t go FOOM."
If you have limited time and want to check if the "old literature" discussed things you’re concerned about, I would just read the Intelligence Explosion Microeconomics paper. It addresses a bunch of this. It sounds like your particular thing is the Hansonian argument: content matters more than architecture, and so you can’t just discover some new architecture that allows you to go FOOM. The paper is probably the best place Yudkowsky’s counterarguments are laid out.
I will write up these reasons separately at some point. There was this other exchange:
Julian:what is the best question I should ask you and what is the answer?
Aashish Reddy:
As in The Hitchhiker’s Guide to the Galaxy, both the question and the answer cannot coexist in the same universe
But the answer is probably: By the time they can go FOOM in the Yudkowskian sense, they can also make decisive progress on alignment. And because of other disagreements with Yudkowsky (not ones litigated in the book, other than that I think they’ll be building their successors, not modifying themselves), I think there’s little reason to expect the systems to be adversarial, and overall expect doom to be unlikely.
On the Hanson–Yudkowsky debate, local vs. global intelligence explosions, “content vs. architecture,” and what the old arguments predicted about modern AI
This began as a Twitter/X thread after I read and tweeted about the Hanson–Yudkowsky AI–Foom Debate. Eliezer Yudkowsky joined the thread to object to my interpretation of the debate, and we ended up having the exchange reproduced below.
I’ve preserved the dialogue verbatim, except for paragraphing, fixing obvious [typos] and expanding links. I’ve removed unrelated replies and moved a few pieces of context into bracketed editorial notes. Nothing has been rewritten for substance.
Context
Aashish Reddy:
I have now finished reading The Hanson-Yudkowsky AI-Foom Debate, which is basically 60 blog posts from Yudkowsky and Hanson over ~500 pages, a transcript of their in-person debate at Jane Street, a (good) summary by Kaj Sotala, and Yudkowsky’s ~100 page paper on “Intelligence Explosion Microeconomics”. I will take questions from those who do not wish to subject themselves to this. I judge the winner of the debate to have been Carl Shulman (whose contribution was two blog posts and a few feisty comment exchanges)
Sophie Bücker: so uh why was carl the winner
Aashish Reddy:
Thus I declare Carl the winner!
What counts as “local”?
sullyj3:
I don’t think this is yet falsified. Superhuman artificial researchers aren’t here yet
RSI in the purist sense hasn't started yet, notwithstanding how people talk about vibecoding being the beginnings of it. I don't think that ought to count, centrally
Aashish Reddy:
I would say another thing that surprised me is the extent to which one could claim "the jury is out" wrt this; obviously we have not had an intelligence explosion yet. Nevertheless I still expect this to look more like the way Carl Shulman predicts (e.g., as reiterated in his Dwarkesh interview) than the way Yudkowsky predicts.
Certainly Shulman has needed to modify his model less to account for recent developments than Yudkowsky has needed to, and certain bits of Yudkowsky have already been falsified — what I meant about the local/global thing is that the brain in a box in a basement seems certain not to happen, and that I would still not regard it a victory for him if Anthropic can do a "local" intelligence explosion in a 2026-updated sense.
Agree re "RSI has not started yet" but I am making claims robust to that: [see the exchange with corsaren in the appendix]
Eliezer Yudkowsky:
If Anthropic sparks off RSI local to Anthropic, that is, besides being human extinction, also a decisive victory for the Yudkowskian crux of the debate. The Hansonian/EA view was that RSI had to loop in and run through the global economy, so we’d be safe.
I think there is an actual problem here where the anti-Yudkowskian view sounds so stupid in a modern sense that you are unable to read it.
Aashish Reddy:
As I say about the Cyc stuff [in the other branch, reproduced in the next section], there is some stuff which I agree has aged so poorly that I skate over it somewhat. But again, on a spectrum of "how local vs global is the intelligence explosion", you'd surely agree that reality turned out more global than Yudkowsky2008 was arguing?
What I meant there about a "local intelligence explosion in a 2026-updated sense" is still not the Yudkowskian, "They make the foundational discoveries that MIRI was trying slowly to make, and use that to restructure their own cognition, and go FOOM". I agree if that happens, that's landing extremely close to Yudkowsky. That's what I meant by "one could claim the jury is out" — is that still how you expect things to go from here? If it is, then fair enough; we'll see.
But my actual expectation is that if Anthropic pulls of[f] an intelligence explosion in the coming years, it will still involve a lot of hardware/compute, it will still require this global supply chain and so on. Which again, is closer to Shulman than Yudkowsky (and Hanson, for that matter. You're reading me here as being far more supportive of the Hansonian position than I am, I think).
There was really not much in that debate about human extinction. Like, I think Shulman is like 20% on catastrophic outcomes from AI? I don't think that's the thing that makes us safe; my views here are not really things that were contested in the FOOM debate. I was surprised reading it how much of it was about local vs global stuff! I agree that isn't super important, though maybe it seemed more important then because of the prospects of you building a Friendly AI or something?
But looking out at reality, I do not think now I would say, "Yes: it may not have taken the form of a brain in a box in a basement, but that was substantively right". I would say it was less local than you were arguing.
Content versus architecture
[In another part of the thread, sullyj3 argued that current approaches might be far from exhausting what is possible with our compute budget, describing text prediction as an “extremely bizarre route to AGI”. My reply prompted this exchange with Eliezer]
Aashish Reddy:
I agree it remains logically possible, but as far as I can tell, nobody is still arguing for it, and Yudkowsky has conceded defeat. He claims that he was mistaken because he underrated evolutionary algorithms; as I said to him [in an earlier exchange about neural networks][1], I think he was mistaken because of another spiritual Hanson victory in this debate — underrating the importance of content relative to architecture.
Of course, you need the right architecture (transformers etc) that let you learn the content, but what the text prediction does is let you learn models of the world: first it gets a map, then RL teaches it to search through possible paths. I agree we will likely have an AlphaGo →AlphaZero style improvement here, but I can’t see anyone really arguing that this suboptimal thing isn’t the basis we’ll be starting from. And this cuts against the Yudkowskian model (related reasons are also why I disagree with him on doom)[2].
Eliezer Yudkowsky:
This is just anachronism! You hear Hanson talk about “content beating architecture” and you think surely he must be talking about data; what else could he be talking about? He means proprietary handcrafted inputs and code! A la his “I <3 Cyc”!
“Architecture”, in Yudkowskian lingo, is the opposite of handcrafting all the content! It means: Something where you can just pour in data, and it learns from that data!
The Hansonian view is then relevant to larger issues because it implies that there is no 'general AI'! There is no big company that is the best at 'AGI' and could spark off ASI internally! The whole global economy participates in handcrafting different specialized AIs!
I thought there'd be a complicated understood architecture that let AIs learn from data without added human labor! Reality declared there was a simple opaque architecture that could learn from data! Hanson thought we'd be safe because there would BE NO ARCHITECTURE FOR AI!
Aashish Reddy:
This is kind of true, which is why I added “spiritual”: condensing for simplicity, the Hansonian vibe was “Look, the thing that makes human minds special is that there’s a lot of stuff in them, like how to farm and write and do industry, and then architecture is this background thing that lets us use it”, and the Yudkowskian one was “The thing that makes minds special is the stuff they have that chimpanzees don’t, which is surely not these pieces of content coded in, but new cognitive algorithms not present in the architecture of chimp brains”.
I agree the Cyc stuff was confusing and weird but I do think spiritually current systems lie in between these positions, to the Hansonian end. “We got the knowledge in there by adjusting weights on a neural network based on a loss function of predicting text on the Internet” is very different from what Hanson was endorsing there (“we’re going to code in specific sentences”?). Very wrong means but basically right on “it has to learn stuff” first.
You see I give the nod here to Shulman not Hanson: as he says, this can then be copied many times and increase research effort and so on. But your thing, as I read it, seemed to imagine something more like AIs that would form their own ontological categories on encountering the world, by using cognitive algorithms they’d developed, or something like this. Hence they wouldn’t be able to share much, and so on. That seems substantially different to “Actually they just read the entire Internet in a way that lets them understand what we’re talking about”.
Eliezer Yudkowsky:
Again: Anachronism. The Yudkowskian thesis is that what AIs learn from data end up being local to that AI and to that company, rather than being saleable Hansonian Cyc-content that gets bought and combined into AIs no single company could ever afford to build.
The translation of the Hansonian vision into modern terms would be big companies trading single LLM layers around, that get assembled into an AI that no single firm could afford to make, which is why it's impossible for RSI to happen inside one firm without the global economy.
The Hansonian thesis is so alien to the modern experience, which is in reality more Yudkowskian than the original Yudkowsky, that you are no longer able to parse his words in terms of the world you know.
Aashish Reddy:
As I say here [in the previous section on “What counts as local?”]: "You're reading me here as being far more supportive of the Hansonian position than I am, I think". I explain why in Point 1 above. So sticking to the Yudkowskian position. I am thinking here of claims like the following:
I say plenty of sympathetic stuff about the Yudkowskian position here [in the appendix], and I think it was far more right than the Hansonian position on net. In my original reply on the thread you’re replying to, you’ll see that on my interpretation, Shulman gives a decisive (Yudkowskian!) argument against Hanson’s position, or at least, challenges him to argue “on [your] turf”, which he never does — at least not to any reasonable degree. But the way in which his framing was at odds with yours is not a decisive refutation in the same way; it’s just turned out to be closer to reality. I thus find it a bizarre claim that “reality [is] more Yudkowskian than the original Yudkowsky”!
Eliezer Yudkowsky:
“If two AIs both see an apple for the first time, and they both independently form concepts about that apple, and they both independently build some new cognitive content around those concepts, then their thoughts are effectively written in a different language.” = NEURALESE!
What this means is that the "cognitive content" ends up as NEURALESE, which you cannot trade around between different companies, unlike Hanson's Cyc-content, which he thought would have [to] be built and combined by many different specialized companies with no one AGI!
Aashish Reddy:
Does neuralese make it “difficult for [AIs] to trade new knowledge as raw cognitive content without having a special language designed just for sharing knowledge”? I’m confused by the claim here. Of course different AIs store/encode the concepts associated with “apple” differently in their weights. There’s still plenty of room for knowledge transfer between AIs though...
Eliezer Yudkowsky:
HANSON DID NOT KNOW ABOUT WEIGHTS. THIS IS BEFORE DEEP LEARNING TOOK OFF. Hanson thought there would be human-readable data handwritten by the company associated with the apple concept. I WAS ARGUING THE THING YOU TAKE FOR GRANTED.
His position is not "AIs can talk to each other". I would've agreed with that claim. His claim is that COMPETENCE WITH APPLES takes the form of HUMAN-HANDCRAFTED DATA that gets TRADED AND ASSEMBLED BETWEEN COMPANIES.
For God's sake read "I <3 Cyc" and then look up how Cyc was being built. IT USED NO DEEP LEARNING. IT HAD NO WEIGHTS.
Aashish Reddy:
Do you think I think Hanson foresaw deep learning? I am pretty sure I didn't say this. But just in case I'm being unclear: I do not think Hanson saw deep learning coming or had a proto-concept of weights or anything in this vicinity.
I read "I <3 Cyc" and was as bemused by it as you were, and think/hope that had I not been 3-4 years old at the time of this exchange, I'd have been equally as bemused then as well. I agreed with your point that substituting the words for other words would throw a wrench in its "knowledge" and so it's hard to see how anyone could call it an AGI project. Of course, Hanson's broader point was that "knowing lots" is important and so AI developers should figure out some way of getting lots of content into their systems, and we can say he was right that this was important and comically wrong re how we ended up getting knowledge into the systems. It is very obvious that Yudkowsky won the Cyc-exchange!
However the quotes I mentioned suggest that AIs that independently form concepts would struggle to share knowledge “without having a special language designed just for sharing knowledge”, and this creates barriers to trade. I do not think this turned out to be accurate, even given neuralese?
Models all form concepts by predicting human text, so their ontologies are in the relevant sense anchored to ours. There is no special language of the kind you claim necessary; it’s just natural language! This is why knowledge can leak between labs (eg via outputs/distillation) even if they’re not trading weights. Or to put it another way, you were right on “internal representations are local”, but this doesn’t imply “knowledge is local”, and the second is the relevant one for the local vs global debate. And I think the reason is directionally Hansonian: learning from the Internet means importing a _shared_ontology.
But like, set aside Hanson! I don’t think he won the debate! I mostly read the book as you laying out your position, and Hanson occasionally nudging you to 1) Make your argument clearer, 2) Explain at a meta-level what you’re doing and why you’re licensed to use the abstractions you are, 3) Respond to some object-level pressures he put on your arguments.
I don’t think he laid out an object-level argument of his own that was anywhere near as fleshed out, and he got some important object-level things wrong. And so I’m hardly saying “Hanson was right” (as you keep attributing to me!); I’m saying, “In places where Yudkowsky was wrong, we can illuminate that via the pressures Hanson put on his argument”.
What would vindicate the original picture?
Aashish Reddy (continued):
Now I thought you might reply that just because the course of events so far doesn’t look Yu[dk]owskian, that doesn’t mean it won’t become so. I am genuinely surprised to hear you say that “reality [is] more Yudkowskian than the original Yudkowsky”. If you claim I’m misunderstanding your argument, perhaps you can help me clarify your actual position by answering these:
[The exchange ends with these questions. I would welcome the thoughts of others!]
Appendix: What do we mean by FOOM?
[This was a separate branch of the original thread, referred to several times above]
corsaren:
What definition of FOOM is even being used here? I.e., are we talking the superexponential/bounded-time singularity stuff?
Also, longer question: one of the biggest objections I’ve had to RSI/FOOM theories as of late is the idea that “improving capabilities” is inherently recursive. For things like raw processing speed, sure, but idk how far that gets you on its own, and so much of intelligence seems to be about (spiky) knowledge of the world and building the right mental models, and improving in that way doesn’t seem obviously recursive.
E.g., I can get better at proposing new drug molecules, but that doesn’t make me better at getting better at proposing new drug molecules, and it certainly doesn’t make me better at getting better at business negotiations. Curious if this came up at all as I’ve been meaning to read through to old literature to see if this objection has already been well-addressed.
Aashish Reddy:
Which, come to think of it, I should perhaps write up in a similar format to this post.
I will write up these reasons separately at some point. There was this other exchange: