Also, the math of AIXI assumes the environment is separably divisible - no matter what you lose, you get a chance to win it back later.
Does this mean that we don't even need to get into anything as esoteric as brain surgery – that AIXI can't learn to play Sokoban (without the ability to restart the level)?
Belief in disbelief, perhaps.
The religious version
You may want something like "the Christian version". Ancient Greek paganism was a religion.
- This is our second important observation
The numbering seems to have gotten borked somehow.
I somehow managed to read this before chapter 6. With context, "who was too young" hits harder.
The fact that this is #5 turns out to be darkly appropriate.
Suppose I built a machine with artificial skin that felt the temperature of the cup and added ice to cold cups and lit a fire under hot cups.
Should this say "added ice to hot cups and lit a fire under cold cups"?
"Oh, right," said Xenophon, "How about 'Systems that would adapt their policy if their actions would influence the world in a different way'?"
"Teacup test," said Socrates.
This seems wrong. The artificial-skin cup adds ice or lights fire solely according to the temperature of the cup; if it finds itself in a world where ice makes tea hotter and fire makes tea colder, the cup does not adapt its strategies.
I gesture vaguely at Morality as Fixed Computation, moral realism, utility-function blackboxes, Learning What to Value, discerning an initially-uncertain utility function that may not be straightforward to get definitive information about.