If Anyone Builds It could have been an explanation for why the MIRI worldview is still relevant nearly two decades later, in a world where we know so much more about AI. Instead, the authors spend all their time shadowboxing against opponents they’ve been bored of for decades, and fail to make their own case in the process.

Hm. I'm torn between thinking this is a sensible criticism and thinking that this is missing the point.

In my view, the core MIRI complaint about 'gradualist' approaches is that they are concrete solutions to abstract problems. When someone has misdiagnosed the problem, their solutions will almost certainly not work, and the question is just where they've swept the difficulty under the rug. Knowing so much more about AI as an engineering challenge while having made no progress on alignment the abstraction--well, the relevance of the MIRI worldview is obvious. "It's hard, and if you think it's easy you're making a mistake."

People attempting to solve AI seem overly optimistic about their chances of solving it, in a way consonant with them not understanding the problem they're trying to solve,  and not consonant with them having a solution that they've simply failed to explain to us. The book does talk about examples of this, and tho you might not like the examples (see, for example, Buck's complaint that the book responds to the safety sketches of prominent figures like Musk and LeCun instead of the most thoughtful versions of those plans) I think it's not obvious that they're the wrong ones to be talking about. Musk is directing much more funding than Ryan Greenblatt is.

The arguments for why recent changes in AI have alignment implications have, I think, mostly failed. You may recall how excited people were about an advanced AI paradigm that didn't involve RL. Of course, top-of-the-line LLMs are now trained in part using RL, because--obviously they would be? It was always cope to think they wouldn't be? I think the version of this book that was written two years ago, and so spent a chapter on oracle AI because that would have been timely, would have been worse that the book that tried to be timeless and focused on the easy calls.

But the core issue from the point of view of the New York Times or the man on the street is not "well, which LessWrong poster is right about how accurately we can estimate the danger threshold, and how convincing our control schema will be as we approach it?". It's that the man on the street thinks things that are already happening are decades away, and even if they believed what the 'optimists' believe they would probably want to shut it all down. It's like the virologists talking amongst themselves about the reasonable debate over whether or not to do gain-of-function research, and the rest of society looked in for a moment and said "what? Make diseases deadlier? Are you insane?".

The fundamental architecture, training methods and requirements for progress for modern AI systems are all completely different from the technology Yudkowsky imagined in 2008, yet nothing about the core MIRI story has changed. 

We could say — and certainly Yudkowsky and Soares would say — that this isn’t important, because the essential dynamics of superintelligence don’t depend on any particular architecture. But that just raises a different question: why does the rest of the book talk about particular architectures so much? Chapter two, for example, is all about contingent properties of present day AI systems. It focuses on the fact that AIs are grown, not crafted — that is, they emerge through opaque machine learning processes instead of being designed like traditional computer programs. This is used as evidence that we should expect AIs to have strange alien values that we can't control or predict, since the humans who “grow” AIs can’t exactly input ethics or morals by hand. This might seem broadly reasonable — except that this was also Yudkowsky’s conclusion in 2006, when he assumed that AIs would be crafted. Back then, his argument was that during takeoff, when an AI rapidly self-improves into superintelligence, it would undergo a sudden and extreme value shift. Yudkowsky and Soares still believe this argument, or at least Soares did as of 2022. But if this is true, then the techniques used to build older, dumber systems are irrelevant — the risk comes from the fundamental nature of superintelligence, not any specific architecture.

I think Clara misunderstands the arguments and how they've changed. There's two layers to the problem: In the first layer, which was the one relevant in the old days, the difficulties are mainly reflective stability and systematic-bias-introduced-through-imperfect-hypothesis-search.^[1] Sufficient understanding and careful design would be enough to solve these, and this is what agent foundations was aiming at. How difficult these problems are does depend on the architecture, as with other engineering safety problems, and MIRI was for a while working on architectures that they hoped would solve the problems (under the heading HRAD, highly reliable agent design).

The second layer of argument became relevant with deep learning: that, if grown rather than engineered, the inner alignment issue becomes almost unavoidable,^[2] and the reflective stability issue doesn't go away. The reflective stability issue doesn't depend on FOOM, as Clara claims, it just depends on the kind of learning-how-to-think-better or reasoning-about-how-to-think-better that humans routinely do.
The book focuses on the second layer of argument, but does mention the first (albeit mostly explained via analogy):

It's also discussed more in the supplemental material (note non-dependence on FOOM). And it emphasises at the end that the reflective stability issue (layer 1) is disjunctive with the inner alignment issue (layer 2).

Importantly, the second layer of argument did change the conclusion. It caused Yudkowsky to update negatively on our chances at succeeding at alignment,^[3] because it adds that second layer of indirection between our engineering efforts and the ultimate goals of a super-intelligence.

I find it unpleasant how aggressive Clara is in this article, especially given the shallow understanding of the argument structure and how it has changed.

1. ^{^}

   Where this is a difficult problem to explain, but extreme examples of it are optimization daemons and the malign prior.

2. ^{^}

   Because you've deliberately created a mesa-optimizer.

3. ^{^}

   I'm sure there was a tweet or something where he said something like "the success of deep learning was a small positive update (because CoT) and a large negative update (because inscrutable)". Can't find it.

That is, we will have one opportunity to align our superintelligence. That's why we'll fail. It's almost impossible to succeed at a difficult technical challenge when we have no opportunity to learn from our mistakes. But this rests on another implicit claim: Currently existing AIs are so dissimilar to the thing on the other side of FOOM that any work we do now is irrelevant.

I think this is an explicit claim in the book, actually? I think it's at the beginning of chapter 10. (It also appears in the story of Sable, where the AI goes rogue because it does a self-modification that creates such a dissimilarity.)

I think "irrelevant" is probably right but something like "insufficient" is maybe clearer. The book describes people working in interpretability as heroes--in the same paragraph as it points out that being able to see that your AI is thinking naughty thoughts doesn't mean you'll be able to design an AI that doesn't think naughty thoughts.