Summary and Table of Contents

The goal of this post is to discuss the so-called “sharp left turn”, the lessons that we learn from analogizing evolution to AGI development, and the claim that “capabilities generalize farther than alignment” … and the competing claims that all three of those things are complete baloney. In particular,

Section 1 talks about “autonomous learning”, and the related human ability to discern whether ideas hang together and make sense, and how and if that applies to current and future AIs.
Section 2 presents the case that “capabilities generalize farther than alignment”, by analogy with the evolution of humans.
Section 3 argues that the analogy between AGI and the evolution of humans is not a great analogy. Instead, I offer a new and (I claim) better analogy between AGI training and, umm, a weird fictional story that has a lot to do with the evolution of humans, but it’s definitely not evolution, I swear. I draw some lessons from this improved analogy, including reasons for both reassurance and concern about an AGI “sharp left turn”.
Section 4 presents the opposite case that “alignment generalizes farther than capabilities”.
Section 5 tries to reconcile the two competing claims, partly in terms of different assumptions regarding the level of AI autonomy versus supervision. I wind up mostly pessimistic, but suggest some approaches that might work.

By the way, I categorized this post as part of the 2023 lesswrong annual review, as it’s kinda a response to, and riff on, the following:

Nate Soares (@So8res): A central AI alignment problem: capabilities generalization, and the sharp left turn (2022). (…which elaborates on something also mentioned in Eliezer Yudkowsky AGI Ruin: A List of Lethalities (2022))
@Quintin Pope: Evolution provides no evidence for the sharp left turn (2023), plus Quintin Pope on the AXRP podcast (2023)
Beren Millidge (@beren): Alignment likely generalizes further than capabilities (2024)
Nate Soares: Humans aren’t fitness maximizers (2022)
@Kaj_Sotala: Genetic fitness is a measure of selection strength, not the selection target (2023)
@jacob_cannell: Evolution Solved Alignment (what sharp left turn?) (2023)
@KatjaGrace: Have we really forsaken natural selection? (2024)

All these posts are well worth reading, but hopefully this post will be self-contained. There are othe...

2023 Longform Reviews