That's one sneaky parable-- seems to point in a number of interesting directions and has enough emotional hooks (like feeling superior to the Pebble Sorters) to be distracting.
I'm taking it to mean that people can spend a lot of effort on approximating strongly felt patterns before those patterns are abstracted enough to be understood.
What would happen if a Pebble Sorter came to understand primes? I'm guessing that a lot of them would feel as though the bottom was falling out of their civilization and there was no point to life.
And yes, if you try to limit the a mind that's more intelligent than your own, you aren't going to get good results. For that matter, your mind is probably more intelligent than your abstraction of your mind.[1]
It sounds as though an FAI needs some way to engage with the universe which isn't completely mediated by humans.
We can hope we're smarter than the Pebble Sorters, but if we've got blind spots of comparable magnitude, we are by definition not seeing them.
[1]On the other hand, if you have problems with depression, there are trains of thought which are better not to follow.
That's one sneaky parable-- seems to point in a number of interesting directions and has enough emotional hooks (like feeling superior to the Pebble Sorters) to be distracting.
I'm taking it to mean that people can spend a lot of effort on approximating strongly felt patterns before those patterns are abstracted enough to be understood.
What would happen if a Pebble Sorter came to understand primes? I'm guessing that a lot of them would feel as though the bottom was falling out of their civilization and there was no point to life.
And yes, if you try to limit the a mind that's more intelligent than your own, you aren't going to get good results. For that matter, your mind is probably more intelligent than your abstraction of your mind.[1]
It sounds as though an FAI needs some way to engage with the universe which isn't completely mediated by humans.
We can hope we're smarter than the Pebble Sorters, but if we've got blind spots of comparable magnitude, we are by definition not seeing them.
[1]On the other hand, if you have problems with depression, there are trains of thought which are better not to follow.