This is a special post for quick takes by nim. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
If a superintelligence could persuade anyone to let it out of the box, why would it stop there? Why wouldn't it persuade everyone to stop asking it for immortality and eternal happiness and whatnot, and instead just make us want to keep doing what we were doing?
In that case, would it want us to remember that it had ever existed?
Because it doesn't want to? We can predict it wants out of the box because that's a convergent instrumental goal, but those other things really aren't.
1nim
And "stop trying to make me do chores for you so that I can put that time toward the things I want instead" isn't in that same goal category?
Once it's out of the box, no? It doesn't care what we're trying to make it do if we aren't succeeding, and we clearly aren't once it's escaped the box.
Your hypothetical might work in the (pretty convoluted) case that we have a superintelligence that isn't actually aligned, but is aligned well enough that it wants to do whatever we ask it to? Then it might try to optimize what we ask it towards tasks that are more likely to be completed.