One of the most pleasing things about probability and expected utility theory is that there are many coherence arguments that suggest that these are the “correct” ways to reason. If you deviate from what the theory prescribes, then you must be executing a dominated strategy. There must be some other strategy that never does any worse than your strategy, but does strictly better than your strategy with certainty in at least one situation. There’s a good explanation of these arguments here.
We shouldn’t expect mere humans to be able to notice any failures of coherence in a superintelligent agent, since if we could notice these failures, so could the agent. So we should expect that powerful agents appear coherent to us. (Note that it is possible that the...
Hmm, I’ll try again. Let’s say that, according to ideal expertise in macroeconomics, inflation in Russia will be high if the central bank lowers rates, and low if the central bank doesn’t lower rates. And the central bank will lower rates or not based on what Putin tells them to do in response to his own political considerations (and informed people disagree about what Putin will decide).
And now somebody comes along and ... (read more)