[ Question ]

What are the reasons to *not* consider reducing AI-Xrisk the highest priority cause?

by capybaralet1 min read20th Aug 201927 comments


Existential RiskCause Prioritization

ETA: I'll be adding things to the list that I think belong there.

I'm assuming a high level of credence in classic utilitarianism, and that AI-Xrisk is significant (e.g. roughly >10%), and timelines are not long (e.g. >50% ASI in <100years). ETA: For the purpose of this list, I don't care about questioning those assumptions.

Here's my current list (off the top of my head):

  • not your comparitive advantage
  • consider other Xrisks more threatening (top contenders: bio / nuclear)
  • infinite ethics (and maybe other fundamental ethical questions, e.g. to do with moral uncertainty)
  • S-risks
  • simulation hypothesis
  • ETA: AI has high moral value in expectation / by default
  • ETA: low tractability (either at present or in general)
  • ETA: Doomsday Argument as overwhelming evidence against futures with large number of minds

Also, does anyone want to say why they think none of these should change the picture? Or point to a good reference discussing this question? (etc.)

New Answer
Ask Related Question
New Comment

4 Answers

Other reasons that people may have (I have some of these reasons, but not all):

  • not a classical utilitarian
  • don't believe those timelines
  • too distant to feel an emotional tie to
  • unclear what to do even if it is a priority
  • very high discount rate for future humans
  • belief that moral value is relative with cognitive ability (an extremely smart AI may be worth a few quitillion humans in a moral/experiential sense)

Of these, I think the one that I'm personally least moved by while acknowleging it as one of the better arguments against utilitarianism is the last. It's clear that there's SOME difference in moral weight for different experiences of different experiencers. Which means there's some dimension on which a utility monster is conceivable. If it's a dimension that AGI will excel on, we can maximize utility by giving it whatever it wants.

Without rejecting any of the premises in your question I can come up with:

Low tractability: you assign almost all of the probability mass to one or both of "alignment will be easily solved" and "alignment is basically impossible"

Currently low tractability: If your timeline is closer to 100 years than 10, it is possible that the best use of resources for AI risk is "sit on them until the field developers further" in the same sense that someone in the 1990s wanting good facial recognition might have been best served by waiting for modern ML.

Refusing to prioritize highly uncertain causes in order to avoid the Winner's Curse outcome of your highest priority ending up as something with low true value and high noise

Flavours of utilitarianism that don't value the unborn and would not see it as an enormous tragedy if we failed to create trillions of happy post-Singularity people (depending on the details human extinction might not even be negative, so long as the deaths aren't painful)

I'll add one more:

  • Doomsday Argument as overwhelming evidence against futures with large number of minds

Also works against any other x-risk related effort and condones a carpe-diem sort of attitude on the civilizational level.

Here's Will MacAskill's answer.