This isn't even related to the law of large numbers, which says that if you flip many coins you expect to get close to half heads and half tails. This is as opposed to flipping 1 coin, where you expect to always get either 100% heads or 100% tails.

I personally expected that P(AI) would drop-off roughly linearly as n increased, so this certainly seems counter-intuitive to me.

An inflection point for probability estimates of the AI takeoff?

by Prismattic 1 min read29th Apr 201146 comments


Suppose that your current  estimate for possibility of an AI takeoff coming in the next 10 years is some probability x.  As technology is constantly becoming more sophisticated, presumably your probability estimate 10 years from now will be some y > x.  And 10 years after that, it will be z > y.  My question is, does there come a point in the future where, assuming that an AI takeoff has not yet happened in spite of much advanced technology, you begin to revise your estimate downward with each passing year?  If so, how many decades (centuries) from now would you expect the inflection point in your estimate?