Solomonoff Induction could be well-calibrated across mathematically possible universes. If a hypothesis has a probability 10%, you should expect it to be true in 10% of the universes.
Important thing is that Solomonoff priors are just a starting point in our reasoning. Then we update on evidence, which is at least as important as having reasonable priors. If it does not seem well calibrated, that is because you can't get good calibration without using evidence.
Imagine that at this moment you are teleported to another universe with completely different laws of physics... do you expect any other method to work better than Solomonoff Induction? Yes, gradually you get data about the new universe and improve your model. But that's exactly what you are supposed to do with Solomonoff priors. You wouldn't predictable get better results by starting from different priors.
It appears to just be a formalization of Occam's Razor, which itself is just a rule of thumb.
To me it seems that Occam's Razor is a rule of thumb, and Solomonoff Induction is a mathematical background explaining why the rule of thumb works. (OR: "Choose the most simple hypothesis that fits your data." Me: "Okay, but why?" SI: "Because it is more likely to be the correct one.")
But if it turned out not to be well-calibrated, it would not be a very good "recipe for truth." What am I missing?
You can't get a good "recipe for truth" without actually looking at the evidence. Solomonoff Induction is the best thing you can do without the evidence (or before you start taking the evidence into account).
Essentially, the Solomonoff Induction will help you avoid the following problems:
Getting inconsistent results. For example, if you instead supposed that "if I don't have any data confirming or rejecting a hypothesis, I will always assume its prior probability is 50%", then if I give you two new hypotheses X and Y without any data, you are supposed to think that p(X) = 0.5 and p(Y) = 0.5, but also e.g. p(X and Y) = 0.5 (because "X and Y" is also a hypothesis you don't have any data about).
Giving so extremely low probability to a reasonable hypothesis that available evidence cannot convince you otherwise. For example if you assume that prior probability of X is zero, then with proper updating no evidence can convince you about X, because there is always an alternative explanation with a very small but non-zero evidence (e.g. the lords of Matrix are messing with your brain). Even if the value is technically non-zero, it could be very small like 1/10^999999999, so all the evidence you could get within your human life could not make you change your mind.
On the other hand, some hypotheses do deserve very low prior probability, because reasoning like "any hypothesis, however unlikely, has prior probability at least 0.01" can be exploited by a) Pascal's mugging, b) constructing multiple mutually exclusive hypotheses which together have arbitrarily high probability (e.g. "AAA is the god of this world and I am his prophet", "AAB is the god of this world and I am his prophet"... "ZZZ is the god of this world and I am his prophet").
Thank you for your reply. It does clear up some of the virtues of SI, especially when used to generate priors absent any evidence. However, as I understand it, SI does take into account evidence - one removes all the possibilities incompatible with the evidence, then renormalizes the probablities of the remaining possibilities. Right?
If so, one could still ask - after taking account of all available evidence - is SI then well-calibrated? (At some point it should be well-calibrated, right? More calibrated than human beings. Otherwise, how is it useful... (read more)
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.