LESSWRONG
LW

881
Wikitags

The Hanson-Yudkowsky AI-Foom Debate

Edited by Wei Dai, Paul Crowley, et al. last updated 5th Apr 2022

In late 2008, an extensive and long-awaited debate about the Technological Singularity occurred on Overcoming Bias, mainly between Robin Hanson and Eliezer Yudkowsky. It focused on the likelihood of hard AI takeoff ("FOOM"), the need for a theory of Friendliness, and the future of AGI, whole brain emulation, and recursive self-improvement in general. A stand-up debate on the same themes took place at Jane Street Capital in June 2011, and Hanson made a further post in 2014.

See Also

  • AI Takeoff
  • Intelligence explosion
  • Outside view

The best resource for most readers new to the debate will be MIRI's 2013 The Hanson-Yudkowsky AI-Foom Debate eBook, available in PDF, EPUB, and MOBI formats. This includes

  • the original series of blog posts,
  • a transcript of the 2011 debate
  • Kaj Sotala's summary and analysis of the debate, and
  • a 2013 technical report on AI takeoff dynamics (“intelligence explosion microeconomics”) written by Yudkowsky.

Prologue

  • Fund UberTool? by Robin Hanson
  • Engelbart as UberTool? by Robin Hanson
  • Friendly Teams by Robin Hanson
  • Friendliness Factors by Robin Hanson
  • The Weak Inside View by Eliezer Yudkowsky
  • Setting The Stage by Robin Hanson
  • The First World Takeover by Eliezer Yudkowsky
  • Abstraction, Not Analogy by Robin Hanson
  • Whence Your Abstractions? by Eliezer Yudkowsky

Main sequence

  • AI Go Foom by Robin Hanson
  • Optimization and the Singularity by Eliezer Yudkowsky
  • Eliezer's Meta-Level Determinism by Robin Hanson
  • Observing Optimization by Eliezer Yudkowsky
  • Life's Story Continues by Eliezer Yudkowsky
  • Emulations Go Foom by Robin Hanson
  • Brain Emulation and Hard Takeoff by Carl Shulman
  • Billion Dollar Bots by James Miller
  • Surprised by Brains by Eliezer Yudkowsky
  • “Evicting” brain emulations by Carl Shulman
  • Cascades, Cycles, Insight... by Eliezer Yudkowsky
  • When Life Is Cheap, Death Is Cheap by Robin Hanson
  • ...Recursion, Magic by Eliezer Yudkowsky
  • Abstract/Distant Future Bias by Robin Hanson
  • Engelbart: Insufficiently Recursive by Eliezer Yudkowsky
  • Total Nano Domination by Eliezer Yudkowsky
  • Dreams of Autarky by Robin Hanson
  • Total Tech Wars by Robin Hanson
  • Singletons Rule OK by Eliezer Yudkowsky
  • Stuck In Throat by Robin Hanson
  • Disappointment in the Future by Eliezer Yudkowsky
  • I Heart CYC by Robin Hanson
  • Is the City-ularity Near? by Robin Hanson
  • Recursive Self-Improvement by Eliezer Yudkowsky
  • Whither Manufacturing? by Robin Hanson
  • Hard Takeoff by Eliezer Yudkowsky
  • Test Near, Apply Far by Robin Hanson
  • Permitted Possibilities, & Locality by Eliezer Yudkowsky
  • Underconstrained Abstractions by Eliezer Yudkowsky
  • Beware Hockey Stick Plans by Robin Hanson
  • Evolved Desires by Robin Hanson
  • Sustained Strong Recursion by Eliezer Yudkowsky
  • Friendly Projects vs. Products by Robin Hanson
  • Is That Your True Rejection? by Eliezer Yudkowsky
  • Shared AI Wins by Robin Hanson
  • Artificial Mysterious Intelligence by Eliezer Yudkowsky
  • Wrapping Up by Robin Hanson
  • True Sources of Disagreement by Eliezer Yudkowsky
  • The Bad Guy Bias by Robin Hanson
  • Disjunctions, Antipredictions, Etc. by Eliezer Yudkowsky
  • Are AIs Homo Economicus? by Robin Hanson
  • Two Visions Of Heritage by Robin Hanson
  • The Mechanics of Disagreement by Eliezer Yudkowsky

Conclusion

  • What Core Argument? by Robin Hanson
  • What I Think, If Not Why by Eliezer Yudkowsky
  • Not Taking Over the World by Eliezer Yudkowsky

Postscript

  • We Agree: Get Froze by Robin Hanson
  • You Only Live Twice by Eliezer Yudkowsky
  • Debating Yudkowsky by Robin Hanson
  • Foom Debate, Again by Robin Hanson

Jane Street debate

A stand-up debate between Hanson and Yudkowsky on the same themes took place on Wednesday June 30 2011 at Jane Street Capital.

  • Video with better quality audio 1.57G
  • Poor quality WMA audio 97M (dead link)
  • Poor quality MP3 audio 97M (dead link)
  • Full transcript
  • Partial transcript by Chris Hallquist

Summary and analysis

In Spring 2012 Luke Muehlhauser asked Kaj Sotala to write a summary and analysis of the debate; he made a draft of that analysis available in November 2012. The final version of the summary is included in the ebook.

Post-book material

Not included in MIRI's 2013 book on the debate:

  • Robin Hanson's 2014 blog post on the subject in response to the publication of "Superintelligence", I Still Don’t Get Foom
  • Katja Grace's 2014 hosted discussion on Less Wrong, Superintelligence 6: Intelligence explosion kinetics
  • Eliezer's 2017 post, AlphaGo Zero and the Foom Debate and Robin Hanson's reply, What Evidence Is AlphaGo Zero Re AGI Complexity?
Discussion
Discussion