Now we get to the question how detailed the paperclips have to be for the paperclipper to care. I expect the paperclipper to only care when the paperclips are simulated individually and we can't simulate 3^^^^^^3 paperclips individually.
I see no reason to think any work of fiction can lead to such a distortion of reality.
I see no reason to think works of fiction that lead to such a distortion of reality are impossible.
Now we get to the question how detailed the paperclips have to be for the paperclipper to care. I expect the paperclipper to only care when the paperclips are simulated individually and we can't simulate 3^^^^^^3 paperclips individually.
We built the paperclipper. Hell, it doesn't even have to be a literal paperclipper. For that matter, it doesn't have to be literally 3^^^^^^3 paperclips; all that matters is that we can manipulate it's utility function without too much strain on our resources. If it values cheesec... (read more)
If it's worth saying, but not worth its own post, even in Discussion, it goes here.