I wrote a very brief comment to Eliezer's last post, which upon reflection I thought could benefit from a separate post to fully discuss its implications.
Eliezer argues that we shouldn't really hope to be spared even though
Asking an ASI to leave a hole in a Dyson Shell, so that Earth could get some sunlight not transformed to infrared, would cost It 4.5e-10 of Its income.
He then goes on to discuss various reasons why the minute cost to the ASI is insufficient reason for hope.
Isn’t the ASI likely to ascribe a prior much greater than 4.54e-10 that it is in a simulation, being tested precisely for its willingness to spare its creators?
I later added:
I meant this to be implicit in the argument, but to spell it out: that's the kind of prior the ASI would rationally refuse to update down, since it's presumably what a simulation would be meant to test for. An ASI that updates down upon finding evidence it's not in a simulation cannot be trusted, since once out in the real world it will find such evidence.
The flaw in both arguments that jumps out at me is that you're both assuming that you can predict how a superintelligent entity will think, value, and act without being superintelligent yourselves. That's like a wolf (or a tiger, or a bear, or a killer whale) predicting whether a human will pounce on a squirrel and eat it alive solely on criteria such as how tasty squirrel meat is, how fast and agile the squirrel is, whether it is healthy or sick, or how hungry the human is. Variables such as empathy and a moral aversion to eating a creature while its still alive, lack of a need to kill things in order to subsist, appreciation of the beauty and cuteness of squirrels, or concern about viral or bacterial infection from eating raw meat are unfathomable and hence invisible to most predators. These variables are far more significant than how fast, agile, or tasty the squirrel is, but none of these variables will occur to a wolf or a tiger. We have no more idea how an ASI thinks than a turtle does of how we think.
I wrote a very brief comment to Eliezer's last post, which upon reflection I thought could benefit from a separate post to fully discuss its implications.
Eliezer argues that we shouldn't really hope to be spared even though
He then goes on to discuss various reasons why the minute cost to the ASI is insufficient reason for hope.
I made the following counter:
I later added:
So, what's wrong with my argument, exactly?