I think Will MacAskill's summary of the argument made in Chapter 4 of IABIED is inaccurate, and his criticisms don't engage with the book version. Here's how he summarises the argument:
The evolution analogy:
Illustrative quote: “To extend the [evolution] analogy to AI: [...] The link between what the AI was trained for and what it ends up caring about would be complicated, unpredictable to engineers in advance, and possibly not predictable in principle.”
Evolution is a fine analogy for ML if you want to give a layperson the gist of how AI training works, or to make the point that, off-distribution, you don’t automatically get what you trained for. It’s a bad analogy if you want to give a sense of alignment difficulty.
Y&S argue:
- Humans inventing and consuming sucralose (or birth control) would have been unpredictable from evolution’s perspective, and is misaligned with the goal of maximising inclusive genetic fitness.
- The goals of a superintelligence will be similarly unpredictable, and similarly misaligned.
On my reading, the argument goes more like this:[1]
The analogy to evolution (and a series of AI examples) is used to argue that there is a complicated relationship between training environment and preferences (in a single training run!), and that we don't have a good understanding of that relationship. The book uses "complications" to refer to weird effects in the link between training environment and resulting preferences.
From this, the book concludes: Alignment is non-trivial, and shouldn't be attempted by fools.[2]
Then it adds some premises:[3]
The book doesn't explicitly draw a conclusion just from purely the above premises, but I imagine this is sufficient to conclude some intermediate level of alignment difficulty, depending on the difficulty of spotting and patching the most troublesome complications.
The book adds:[4]
From this it concludes: It's unrealistically difficult to remove all complications, and shouldn't be attempted with anything like current levels of understanding.[5]
So MacAskill's summary of the argument is inaccurate. It removes all of the supporting structure that makes the argument work, and pretends that the analogy was used by itself to support the strong conclusion.
He goes on to criticise the analogy by pointing at (true) dis-analogies between evolution and the entire process of building an AI:
But our training of AI is different to human evolution in ways that systematically point against reasons for pessimism.
The most basic disanalogy is that evolution wasn’t trying, in any meaningful sense, to produce beings that maximise inclusive genetic fitness in off-distribution environments. But we will be doing the equivalent of that!
That alone makes the evolution analogy of limited value. But there are some more substantive differences. Unlike evolution, AI developers can:
- See the behaviour of the AI in a very wide range of diverse environments, including carefully curated and adversarially-selected environments.
- Give more fine-grained and directed shaping of individual minds throughout training, rather than having to pick among different randomly-generated genomes that produce minds.
- Use interpretability tools to peer inside the minds they are creating to better understand them (in at least limited, partial ways).
- Choose to train away from minds that have a general desire to replicate or grow in power (unlike for evolution, where such desires are very intensely rewarded).
- (And more!)
So these dis-analogies don't directly engage with the argument. If they were directly engaging with the first part of the argument, they would be about the predictability of a single training run, rather than the total AI research process.
The dis-analogies could be reinterpreted as engaging with the other premises: 1 and 3 could be interpreted as claims that we do have (potentially yet to be invented) methods of detecting complications, and 2 and 4 could be interpreted as claims that we similarly have methods to patch complications. But the examples of unintended LLM weirdness given throughout the book are enough to show that at least current techniques aren't working well for detection or patching of weirdness.
More importantly, the examples of particularly difficult complications entirely ignored, despite their necessity for the conclusion.
I've reordered it for logical clarity.
From the book: "If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation."
These are mentioned in the book (e.g. in footnote 2), and the first is briefly supported with examples. The second isn't supported in this chapter.
In the paragraphs following "So far, we’ve only touched on the sorts of complications that would arise in the preferences trained directly into an AI."
"Problems like this are why we say that if anyone builds it, everyone dies."
A little bit late, but there are more reasons why I think the evolution analogy is particularly good and better than the selective breeding analogy.
Evolution basically ended up optimizing the brain such that it has desires that were instrumental to genetic fitness. So we end up with these instrumental sub-preferences for high calorie food or sex. Then we go through a huge shift of the distribution or environment from a very constrained hunter gatherer society to a technologically advanced civilization. This isn't just a random shift but a shift toward and ... (read more)