The weighty conclusion of the "recursive self-improvement" meme is not “expect seed AI”. The weighty conclusion is “sufficiently smart AI will rapidly improve to heights that leave humans in the dust”.

Note that this conclusion is still, to the best of my knowledge, completely true, and recursive self-improvement is a correct argument for it.

This whole discussion seems relevant to me because it feels like it keeps coming up when you and Eliezer talk about why prosaic AI alignment doesn't help, sometimes explicitly ("Even if this helped with capabilities produced by SGD, why would it help in the regime that actually matters?") and often because it just seems to be a really strong background assumption for you that leads to you having a very different concrete picture of what is going to happen.

It doesn't seem like recursive self-improvement is a cheap lower bound argument, it seems like you really think that what I think of as the "normal, boring" world just isn't going to happen. So I'm generally interested in talking about that and get clear about what you think is going on here, and hopefully get some predictions on the record.

This also gives me the sense that you feel quite strongly about your view of recursive self-improvement. If you had a 50% chance on "something like boring business as usual with SGD driving crucial performance improvements at the crucial time" then your dismissal of prosaic AI alignment seems strange to me.

(ETA: there's actually a lot going on here, I'd guess this is like 1/4th of the total disagreement.)

Robin Hanson was claiming things along the lines of ‘The power is in the culture; superintelligences wouldn’t be able to outstrip the rest of humanity.’

Worth noting that Robin seems to strongly agree that "recursive self-improvement" is going to happen, it's just that he has a set of empirical views for which that name sounds silly and it won't be as local or fast as Eliezer thinks.

Relatedly, Eliezer saying "Robin was wrong for doubting RSI; if other crazy stuff will happen before RSI then he's just even more wrong" seems wrong. In Age of Em I think Robin speculates that within a few years of the first brain emulations, there will be more alien AI systems which are able to double their own productivity within a few weeks (and then a few weeks later it will be even crazier)! That sure sounds like he's on board with the part of RSI that is obvious, and what he's saying is precisely that other crazy stuff will happen first, essentially that we will use computers to replace the hardware of brains before we replace the software. (The book came out in 2016 but I think Robin has had the basic outline of this view since 2012 or earlier.)

The big update over the last decade has been that humans might be able to fumble their way to AGI that can do crazy stuff before it does much self-improvement. 

This feels to me like it's still missing a key part of the disagreement, at least with people like me. As best I can tell/guess, this is also an important piece of the disagreement with Robin Hanson and with some of the OpenAI or OpenPhil people who don't like your discussion of recursive self-improvement.

Here's how the situation seems to me:

• "Making AI better" is one of the activities humans are engaged in.
• If AI were about as good as things at humans, then AI would be superhuman at "making AI better" at roughly the same time it was superhuman at other tasks.
• In fact there will be a lot of dispersion, and prima facie we'd guess that there are a lot of tasks (say 15-60% of them as a a made up 50% confidence interval) where AI is superhuman before AI R&D.
• What's more, even within R&D we expect some complementarity where parts of the project get automated while humans still add value in other places, leading to more continuous (but still fairly rapid, i.e. over years rather than decades) acceleration.
• That said, at the point when AI is capable of doing a lot of crazy stuff in other domains, "AI R&D" is a crazy important part of the economy, and so this will be a big but not overwhelmingly dominant part of what AI is applied to (and relatedly, a big but not overwhelmingly dominant part of where smart people entering the workforce go to work, and where VCs invest, and so on).
• The improvements AI systems make to AI systems are more like normal AI R&D, and can be shared across firms in the same way that modern AI research can be.

As far as I can make out from Eliezer and your comments, you think that instead the action is crossing a criticality threshold of "k>1," which suggests a perspective more like:

• AI is able to do some things and not others.
• The things AI can do, it typically does much better/faster/cheaper than humans.
• Early AI systems can improve some but not all parts of their own design. This leads to rapid initial progress, but diminishing returns (basically they are free-riding on parts of the design already done by humans).
• Eventually AI is able to improve enough stuff that there are increasing rather than diminishing returns to scale even within the subset of improvements that the AI is able to make.
• Past this point progress is accelerating even without further human effort (which also leads to expanding the set of improvements at which the AI is superhuman). So from here the timescale for takeoff is very short relative to the timescale of human-driven R&D progress.
• This is reasonably likely to happen from a single innovation that pushes you over a k>1 threshold.
• This dynamic is a central part of the alignment and policy problem faced by humans right now who are having this discussion. I.e. prior to the time when this dynamic happens most research is still being done by humans, the world is relatively similar to the world of today, etc. 
• The improvements made by AI systems during this process are very unlikely modern R&D, and so can't be shared between AI labs in the same way that e.g. architectural innovations for neural networks or new training strategies can be.

I feel like the first picture is looking better and better with each passing year. Every step towards boring automation of R&D (e.g. by code models that can write mediocre code and thereby improve the efficiency of normal software engineers and ML researchers) suggests that AI will be doing recursive self-improvement around the same time it is doing other normal tasks, with timescales and economic dynamics closer to those envisioned by more boring people.

On what I'm calling the boring picture, "k>1" isn't a key threshold. Instead we have k>1 and increasing returns to scale well before takeoff. But the rate of AI progress is slow-but-accelerating relative to human abilities, and therefore we can forecast takeoff speed by looking at the rate of AI progress when driven by human R&D.

You frame this as an update about "fumbling your way to an AGI that can do crazy stuff before it does much self-improvement," but that feels to me like it's not engaging with the basic argument at issue here: why would you think the AI is likely to be so good at "making further AI progress" relative to human researchers and engineers? Why should we be at all surprised by what we've seen over recent years, where software-engineering AI seems like it behaves similarly to AI in other domains (and looks poised to cross human level around broadly the same time rather than much earlier)? Why should this require fumbling rather than being the default state of affairs (as presumably imagined by someone more skeptical of "recursive self-improvement").

My impression of the MIRI view here mostly comes from older writing by Eliezer, where he often talks about how an AI would be much better at programming because humans lack what you might call a "codic cortex" and so are very bad programmers relative to their overall level of intelligence. This view does not seem to me like it matches the modern world very well---actual AI systems that write code (and which appear on track to accelerate R&D) are learning to program using similar styles and tools to humans, rather than any kind of new perceptual modality.

(As an aside, in most other ways I like the intelligence explosion microeconomics writeup. It just seems like there's some essential perspective that isn't really argued for but suffuses the document, most clear in its language of "spark a FOOM" and criticality thresholds and so on.)

Also: It’s important to ask proponents of a theory what they predict will happen, before crowing about how their theory made a misprediction. You're always welcome to ask for my predictions in advance.

I'd be interested to get predictions from you and Eliezer about what you think is going to happen in relevant domains over the next 5 years. If we aren't able to get those predictions, then it seems reasonable to just do an update based on what we would have predicted if we took your view more seriously (since that's pretty relevant if we are now deciding whether to take your views seriously).

If you wanted to state any relevant predictions I'd be happy to comment on those. But I understand how it's annoying to leave the ball in your court, so here are some topics where I'm happy to give quantitative predictions if you or Eliezer have a conflicting intuition:

• I expect successful AI-automating-AI to look more like AI systems doing programming or ML research, or other tasks that humans do. I think they are likely to do this in a relatively "dumb" way (by trying lots of things, taking small steps, etc.) compared to humans, but that the activity will look basically similar and will lean heavily on oversight and imitation of humans rather than being learned de novo (performing large searches is the main way in which it will look particularly unhuman, but probably the individual steps will still look like human intuitive guesses rather than something alien). Concretely, we could measure this by either performance on benchmarks or economic value, and we could distinguish the kinds of systems I imagine from the kind you imagine by e.g. you telling a story about fast takeoff and then talking about some systems similar to those involved in your takeoff story.
• I expect that the usefulness and impressiveness of AI systems will generally improve continuously. I expect that in typical economically important cases we will have a bunch of people working on relevant problems, and so will have trend lines to extrapolate, and that those will be relatively smooth rather than exhibiting odd behavior near criticality thresholds.
• At the point when the availability of AI is doubling the pace of AI R&D, I expect that technically similar AI systems will be producing at least hundreds of billions of dollars a year of value in other domains, and my median is more like $1T/year. I expect that we can continue to meaningfully measure things like "the pace of AI R&D" by looking at how quickly AI systems improve at standard benchmarks.
• I expect the most powerful AI systems (e.g. those responsible for impressive demonstrations of AI-accelerated R&D progress) will be built in large labs, with compute budgets at least in the hundreds of millions of dollars and most likely larger. There may be important innovations about how to apply very large models, but these innovations will have quantitatively modest effects (e.g. reducing the compute required for an impressive demonstration by 2x or maybe 10x rather than 100x) and so a significant fraction of the total value added / profit will flow to firms that train large models or who build large computing clusters to run them.
• I expect AI to look qualitatively like (i) "stack more layers," (ii) loss functions and datasets that capture cognitive abilities we are interested in with less noise, (iii) architecture and optimization improvements that yield continuous progress in performance, (iv) cleverer ways to form large teams of trained models that result in continuous progress. This isn't a very confident prediction but it feels like I've got to have higher probability on it than you all, perhaps I'd give 50% that in retrospect someone I think is reasonable would say "yup definitely a significant majority of the progress was in categories (i)-(iv) in the sense that I understood them when that comment was written in 2022."

It may be that we agree about all of these predictions. I think that's fine, and the main upshot is that you shouldn't cite anything over the next 5 years as evidence for your views relative to mine. Or it may be that we disagree but it's not worth your time to really engage here, which I also think is quite reasonable given how much stuff there is to do (although I hope then you will have more sympathy for people who misunderstood your position in the future).

Perhaps more importantly, if you didn't disagree with me about any 5 year predictions then I feel like there's something about your position I don't yet understand or think is an error:

• Why isn't aligning future AI systems similar to aligning existing AI systems? It feels to me like it should be about the (i) aligning the systems doing the R&D, (ii) aligning the kinds of systems they are building. Is that wrong? Or perhaps: why do you think they will be building such different systems from "stack more layers"? (I certainly agree they will be eventually, but the question seems to just be whether there is a significant probability of doing stack more layers or something similar for a significant subjective time.)
• Why does continuous improvement in the pace of R&D, driven by AI systems that are contributing to the same R&D process as humans, lead to a high probability of incredibly fast takeoff? It seems to me like there is a natural way to get estimated takeoff speeds from growth models + trend extrapolation, which puts a reasonable probability on "fast takeoff" according to the "1 year doubling before 4 year doubling" view (and therefore I'm very sympathetic to people disagreeing with that view on those grounds) but puts a very low probability on takeoff over weeks or by a small team.

we too predict that it's easy to get GPT-3 to tell you the answers that humans label "aligned" to simple word problems about what we think of as “ethical”, or whatever. That’s never where we thought the difficulty of the alignment problem was in the first place. Before saying that this shows that alignment is actually easy contra everything MIRI folk said, consider asking some MIRI folk for their predictions about what you’ll see

I partly agree with this. Like you, I'm often frustrated by people thinking this is where the core of the problem will be, or where the alignment community mistakenly thought the core of the problem was. And this is especially bad when people see systems that all of us would expect to work, and thereby getting optimistic about exactly the same systems that we've always been scared of.

That said, I think this is an easy vibe to get from Eliezer's writing, and it also looks to me like he has some prediction misses here. Overall it feels to me like this should call for more epistemic humility regarding the repeated refrain of "that's doomed."

Some examples that I feel like don't seem to match recent results. I expect that Eliezer has a different interpretation of these, and I don't think someone should interpret a non-response by Eliezer as any kind of acknowledgment that my interpretation is correct.

• Here: "Speaking of inexact imitation: It seems to me that having an AI output a high-fidelity imitation of human behavior, sufficiently high-fidelity to preserve properties like "being smart" and "being a good person" and "still being a good person under some odd strains like being assembled into an enormous Chinese Room Bureaucracy", is a pretty huge ask."
  
  I think that progress in language modeling makes this view look much worse than it did in 2018. It sure looks like we are going to have inexact imitations of humans that are able to do useful work, to continue to broadly agree with humans about what you "ought to do" in a way that is common-sensically smart (such that to the extent you get useful work from them it's still "good" in the same way as a human's behavior). It also looks like those properties are likely to be retained when a bunch of them are collaborating in a "Chinese Room Bureaucracy," though this is not clear. And it looks quite plausible that all of this is going to happen significantly before we have systems powerful enough that they are only imitating human behavior instrumentally (at which point X-and-only-X becomes a giant concern).
  
  The issue definitely isn't settled yet. But I think that most people outside of LW would look at the situation and say "huh, Eliezer seems like he was too confidently pessimistic about the ability of ML to usefully+safely imitate humans before it became a scary optimization daemon."
   
• Here: "Similar remarks apply to interpreting and answering "What will be its effect on _?" It turns out that getting an AI to understand human language is a very hard problem, and it may very well be that even though talking doesn't feel like having a utility function, our brains are using consequential reasoning to do it. Certainly, when I write language, that feels like I'm being deliberate. It's also worth noting that "What is the effect on X?" really means "What are the effects I care about on X?" and that there's a large understanding-the-human's-utility-function problem here. In particular, you don't want your language for describing "effects" to partition, as the same state of described affairs, any two states which humans assign widely different utilities. Let's say there are two plans for getting my grandmother out of a burning house, one of which destroys her music collection, one of which leaves it intact. Does the AI know that music is valuable? If not, will it not describe music-destruction as an "effect" of a plan which offers to free up large amounts of computer storage by, as it turns out, overwriting everyone's music collection? If you then say that the AI should describe changes to files in general, well, should it also talk about changes to its own internal files? Every action comes with a huge number of consequences - if we hear about all of them (reality described on a level so granular that it automatically captures all utility shifts, as well as a huge number of other unimportant things) then we'll be there forever."
  
  This comment only makes sense if you think that an AI will have difficulty predicting "which consequences are important to a human" or explaining those consequences to a human with any AI system weak enough to be safe. Yet it seems like GPT-3 already has a strong enough understanding of what humans care about that it could be used for this purpose.
  
  Now I agree with you that there is a version of this concern that is quite serious and real. But this comment is very strongly framed about the capabilities of an AI weak enough to be safe, not about the difficulty of constructing a loss function that incentivizes that behavior, and some parts of it appear not to make sense when interpreted as being about a loss function (since you could use a weak AI like GPT-3 as a filter for which consequences we care about).
   
• Same comment: "[it] sounds like you think an AI with an alien, superhuman planning algorithm can tell humans what to do without ever thinking consequentialistically about which different statements will result in human understanding or misunderstanding. Anna says that I need to work harder on not assuming other people are thinking silly things, but even so, when I look at this, it's hard not to imagine that you're modeling AIXI as a sort of spirit containing thoughts, whose thoughts could be exposed to the outside with a simple exposure-function. It's not unthinkable that a non-self-modifying superhuman planning Oracle could be developed with the further constraint that its thoughts are human-interpretable, or can be translated for human use without any algorithms that reason internally about what humans understand, but this would at the least be hard"
  
  This really makes it sound like "non-self-modifying" and "Oracle" are these big additional asks, and that it's hard to use language without reasoning internally about what humans will understand. That actually still seems pretty plausible, but I feel like you've got to admit that we're currently in a world where everyone is building non-self-modifying Oracles that can explain the consequences of their plans (and where the "superhuman planning algorithms" that we deal with are either simple planning algorithms written by humans, as in AlphaZero, or are mostly distilled from such planning algorithms in a way that would not make it harder to get them to express their views in natural language). And that's got to be at least evidence against the kind of view expressed in this comment, which is strongly suggesting that by the time we are building transformative AI it definitely won't look at all like that. The closer we now are to powerful AI, the stronger the evidence against becomes.

FWIW, I think (a somewhat different form of) recursive self improvement is still incredibly important to consider in alignment. E.g., an RL system’s actions can influence the future situations it encounters, and the future situations it encounters can influence which cognitive patterns the outer optimizer reinforces. Thus, there’s a causal pathway by which the AI’s actions can influence its own cognitive patterns. This holds for any case where the system can influence its own future inputs.

It seems clear to me that any alignment solution must be robust to at least some degree of self-modification on the part of the AI.

So why was MIRI's technical research so focused on self reference, tiling, etc?

A conversation I've had several times in recent weeks is with people who argue that we can create human-level intelligences (who are safe, because they're only human-level) and somehow use them to get alignment/pivotal act, or something like "just stop there".

And I think the recursive self-improvement is one answer to why human-level AIs are not safe. Actual humans attempt to improve their intelligence already (e.g. nootropics), it's just hard with our architecture. I expect unaligned human-level AIs to try the same thing and have much more success because optimizing code and silicon hardware is easier than optimizing flesh brains.
 

...we too predict that it's easy to get GPT-3 to tell you the answers that humans label "aligned" to simple word problems about what we think of as “ethical”, or whatever. That’s never where we thought the difficulty of the alignment problem was in the first place. Before saying that this shows that alignment is actually easy contra everything MIRI folk said, consider asking some MIRI folk for their predictions about what you’ll see.

From my perspective it seems like you're arguing that 2+2 = 4 but also 222 + 222 = 555, and I just don't understand where the break happens. Language models clearly contain the entire solution to the alignment problem inside them. Certainly we can imagine a system that is 'psychopathic': it completely understands ethics, well enough to deceive people, but there is no connection between its ethical knowledge and its decision making process. Such a system could be built. What I don't understand is why it is the only kind of system that could be (easily) built, in your view. It seems to me that any of the people at the forefront of AI research would strongly prefer to build a system whose behavior is tied to an understanding of ethics. That ethical understanding should come built in if the AI is based on something like a language model. I therefore assert that the alignment tax is negative. To be clear, I'm not asserting that alignment is therefore solved, just that this points in the direction of a solution.

So, let me ask for a prediction: what would happen if a system like SayCan was scaled to human level? Assume the Google engineers included a semi-sophisticated linkage between the language model's model of human ethics and the decision engine. The linkage would focus on values but also on corrigibility, so if a human shouts "STOP", the decision engine asks the ethics model what to do, and the ethics model says "humans want you to stop when they say stop". I assume that in your view the linkage would break down somehow, the system would foom and then doom. How does that break down happen? Could you imagine a structure that would be aligned, and if not, why does it seem impossible?

> Robin Hanson was claiming things along the lines of ‘The power is in the culture; superintelligences wouldn’t be able to outstrip the rest of humanity.’

This argument hasn't been disproven yet. In fact, still seems quite credible, unless I am mistaken?

Why did past-MIRI talk so much about recursive self-improvement? Was it because Eliezer was super confident that humanity was going to get to AGI via the route of a seed AI that understands its own source code?

I doubt it. My read is that Eliezer did have "seed AI" as a top guess, back before the deep learning revolution. But I don't think that's the main source of all the discussion of recursive self-improvement in the period around 2008. 

Rather, my read of the history is that MIRI was operating in an argumentative argument where:

• Ray Kurzweil was claiming things along the lines of ‘Moore’s Law will continue into the indefinite future, even past the point where AGI can contribute to AGI research.’ (The Five Theses, in 2013, is a list of the key things Kurzweilians were getting wrong.)
• Robin Hanson was claiming things along the lines of ‘The power is in the culture; superintelligences wouldn’t be able to outstrip the rest of humanity.’

The memetic environment was one where most people were either ignoring the topic altogether, or asserting ‘AGI cannot fly all that high’, or asserting ‘AGI flying high would be business-as-usual (e.g., with respect to growth rates)’.

The weighty conclusion of the "recursive self-improvement" meme is not “expect seed AI”. The weighty conclusion is “sufficiently smart AI will rapidly improve to heights that leave humans in the dust”.

Note that this conclusion is still, to the best of my knowledge, completely true, and recursive self-improvement is a correct argument for it.

I disagree here. Foom or hard takeoff do not follow from recursive self-improvement.

Recursive self-improvement induced foom implies that marginal returns to cognitive reinvestment are increasing not diminishing across the relevant cognitive intervals. I don't think that position has been well established.

Furthermore, even if marginal returns to cognitive reinvestment are increasing, it does not necessarily follow that marginal returns to real world capability from cognitive investment are also increasing across the relevant intervals. For example, marginal returns to predictive accuracy in a given domain diminish, and they diminish at an exponential rate (this seems to be broadly true across all relevant cognitive intervals).

This is not necessarily to criticise Yudkowsky's arguments in the context in which it appeared in 2008 - 2013. I'm replying as a LessWronger who has started thinking about takeoff dynamics in more detail and is dissatisfied with those arguments and the numerous implicit assumptions Yudkowsky made that I find unpersuasive when laid out explicitly.

I mention this so that it's clear that I'm not pushing back against the defence of RSI, but expressing my dissatisfaction with the arguments in favour of RSI as presented.