LESSWRONGTags
LW

Holden Karnofsky

EditHistory
Discussion(0)
Help improve this page(1 flags)

Holden Karnofsky is a co-founder of effective altruism pioneers GiveWell.

In May 2012, Karnofsky posted Thoughts on the Singularity Institute (SI), which became the most-upvoted article ever on Less Wrong. It offered a detailed critique of what is now the Machine Intelligence Research Institute, and spawned a great deal of discussion.

MIRI staff posted two replies:

  • Eliezer Yudkowsky, Reply to Holden on 'Tool AI'
  • Luke Muehlhauser, Reply to Holden on The Singularity Institute

Paul Crowley ("ciphergoth") posted discussion articles for each point raised:

  • Objection 1: it seems to me that any AGI that was set to maximize a "Friendly" utility function would be extraordinarily dangerous.
  • Objection 2: SI appears to neglect the potentially important distinction between "tool" and "agent" AI.
  • Objection 3: SI's envisioned scenario is far more specific and conjunctive than it appears at first glance, and I believe this scenario to be highly unlikely.
  • Is SI the kind of organization we want to bet on?
  • Other objections to SI's views

Other discussion:

  • Phil Goetz, Holden's Objection 1: Friendliness is dangerous

Related pages

  • Tool AI