The question (from FTX Future Fund) is what is $

P(misalignment x-risk|AGI)’: Conditional on AGI being developed by 2070, humanity will go extinct or drastically curtail its future potential due to loss of control of AGI.

I’m new to this community but I don’t think we should be computing the probability of a ‘risk’, so my terminology after this will be ‘P(misalignment x-event|AGI)’

By the way, the framing of this piece is very antagonizing toward AI-longtermists (you’ve been warned), because I truly believe you just need someone to yell at you and tell you that you are wrong—and in this case also take your money.

I believe that the fundamental mistake in all AI-longtermist thinking has to do with overconfidence. AI-longtermists binarize the events of our future into “misaligned AI x-event” or “no misaligned AI x-event”, and thus they attempt correct for overconfidence by making the probability of “misaligned AI (or not)” tend toward 50/50. This is a generous attempt, but they fail to consider that they were overconfident in their original binarization.

Indeed, they fail to consider that within the category of “no misaligned AI x-event” there is a whole spectrum of possible ways we can coexist with AGI, there are so many different unknown unknown possibilities that exist. Indeed, a more drastic error is the overconfidence that these are the only two real categories. Suppose people become even more connected to their computers—think replacing search engines and social media with AGI—perhaps inseparably so, then the question is ill-posed, because people as we are familiar with today, no longer exist. We become some melding of mind and machine.

And yet, when it comes to “misaligned AGI x-event”, there are two images everyone fixates on: the computer in the server hacking its way out and seizing control of the world’s cyberspace, or the killer robot, who is programmed to defend world and decides the best way to do so is to destroy. It’s like people just like watching movies!!! The bigger “AGI” related risk is probably an “AGI” robot working in a biohazard lab trips (or perhaps swerves to avoid running into a toddler) and breaks a vial and releases some super-smallpox into the world. I would argue that this “AGI toddler saving tripping event” doesn’t even fall in the category of “misaligned AGI x-event”, it’s more garden-variety “robot has safety measure, fo...

Posts

Wikitag Contributions

Comments