Cross-posted to the EA forum https://forum.effectivealtruism.org/posts/aSDnzAm85a3Pi87rm/what-role-should-evolutionary-analogies-play-in

This is based on a research project that I’ve been working on under the supervision of Dr. Vael Gates. This work is funded by the Long-Term Future Fund. I would like to thank Vael Gates for their support and supervision, and Tilman Räuker for their valuable feedback.

Epistemic status: exploratory, fairly high uncertainty. My background is in experimental physics, and I only have limited experience with biology or AI.

The motivation for this project was a question from Technical AGI Safety Research Outside AI by Richard Ngo.

What are the most compelling arguments for and against discontinuous versus continuous takeoffs? In particular, how should we think about the analogy from human evolution, and the scalability of intelligence with compute?

In short, it seems that many people are quite confused about the use of evolution as an analogy in forecasting AI takeoff speeds (if not, then I certainly am!) and I hope to clarify things through this post. In particular, I’m focusing mostly on “how should we think about the analogy from human evolution”, although I think this itself is a big enough question that I haven’t given a complete answer to it. I’ve made an effort to accurately represent existing arguments and stay up to date with new ones, but at the end of the day these are still my interpretations of them. Comments and feedback are very welcome, and all mistakes are my own.

Table of Contents

0 Overview
1 Evolutionary arguments for takeoff speeds
- 1.1 Hominid variation
- 1.2 Changing selection pressures
- 1.3 Cumulative cultural evolution
  - 1.3.1 My opinions on this debate
- 1.4 Brains and compute
  - 1.4.1 Side note: Dog-level AI
- 1.5 Biological anchors
- 1.6 One algorithm and Whole Brain Emulation
- 1.7 Anthropic effects
- 1.8 Drivers of intelligence explosions
- 1.9 Summary
2 Evaluating analogies and disanalogies
- 2.1 A non-exhaustive list of analogies and disanalogies
  - Analogy: Timescales
  - Analogy: Necessary compute
  - Analogy: Cumulative cultural evolution
  - Analogy: Intelligence is non-hard
  - Analogy: Thresholds
  - Analogy: Number of neurons
  - Analogy: Structure of the learning system
  - Disanalogy: Training environments
  - Disanalogy: Optimisation

...