A wise man, therefore, proportions his belief to the evidence.
-David Hume, An Enquiry Concerning Human Understanding
The eighth virtue is humility
-Eliezer Yudkowsky
A note: I may remake this a longer multi-part essay at some point, but I think starting here would be a good place since I see emphasis on this kind of knowledge overmuchin related communities.
A Telling of Priors, Perspectives and Quantum Worlds:
Let’s imagine a scenario to explore how one considers priors:
I approach a friend (Al) with a coin sitting on my thumb, heads up, I flip the coin and cover it before they see. I then ask Al “heads or tails?” They call heads, to which I ask “how confident are you in your prediction?” Al answers, “oh, I’d say about 51%, I know that fair coins tend to land on the side facing up around 51% of time since I have no reason to think this coin is any different, I would make the same estimation.”
Another friend (Bri) walks past, they see I am covering my hand but have no knowledge of the coin flip or what might be under there. I ask Bri, “what is under my hand?” They guess, “a coin, tails side up.” Asked how confident they are, Bri proclaims: “I am certain, I guess I’d put the odds at 99%, I don’t know why I just have a strong gut feeling about it.”
I think most readers will rationally agree that Al is making a more reasonable prediction, he is deriving a prior estimate by examining what information he has and making inferences from empirical data accessible to him as to the likeliest outcome. From my own empirical training, I would consider this the ‘correct’ approach. Bri, meanwhile, has insufficient information to draw any empirical prior probabilities. What does she do instead? Like a good Bayesian, she just guesses. Now, what is the true prior probability? If we step back as an outside observer, it is either heads or tails, it is already determined, there is nothing probabilistic going on, however, the information available to Al gave him no way to estimate that and we can say with confidence that if we repeated the experiment we would expect Al to guess correctly roughly 51% of the time, matching his estimated probability, while we would have no way of placing any estimation on Bri’s approach which is a pure Bayesian guess.
Stepping back into the scenario, I uncover my hand revealing that it is a coin tails facing up. Bri declares victory and takes it as a true affirmation of her Bayesian prior as she guessed, very nearly correctly, that it was certain to be a coin facing tails up. Al, however, also does not seem to think he was wrong, even knowing the true prior probability was 100% he views his estimate of 51% as a correct deduction of the empirical evidence available to him. At the same time, he still holds that Bri’s prediction—despite being correct—was irrational, is this view valid?
My contention is that Bri’s method is wrong, the ‘correct’ answer from Bri would have been “I don’t know” and “while I have my hunches, I have no reason to assign any degree of confidence.” Priors not backed by any empirical reasoning may sometimes match our results but when they are not backed with rigor and empirical observations, they are no more trustworthy than any other intuition—which doesn’t mean intuition is useless, but it is not a rational approach to take at face value. One can use processes of Bayesian inference (such as utilizing Monte Carlo experiments) to derive a strong foundation for a prior, but until those are done priors that aren’t supported even where one personally extremely strong priors should be reported as unknowns. For example, I agree with Phil Plait in writing on the risk of dying by alien invasion: “Because of the lack of data, we have a true unknown here. Personally, using just my guts and hunches, I would put the odds in the range of billions-to-one against, but that is not very scientific. So to be truly skeptical, as any real scientist is, I will have to leave this blank, and hope that advances in astrobiology will allow us to make some safe estimates of the odds sometime soon." (Phil Plait, Death from the Skies). Like Plait, I have a very strong prior, going on my intuition and guesses—which I certainly could rationalize if asked—I would place myself as >99.999999% certain that I will not die in an alien invasion. But, since I lack any empirical data and rigor to justify that prior, if asked my response would not be one of confidence but one of “I don’t know,” this is the epistemic humility traditionally advocated by empiricists and the sciences. To be a skeptic, of the empirical sort, is not to reject Bayesian inferences, but to portion your claims with humility and not claim to encapsulate what is unknown. I believe this also leads, at times, to a lack of humility among rationalists in theories and preferences.
While an outdated example, consider for instance the many-worlds-interpretation, championed as obvious by Yudkowsky.
First, is this a fair view of ‘Science’? I don’t believe so. The Copenhagen interpretation (to my understanding, not as a professional physicist) typically is taken to posit collapse as part of the model not measurable reality. It is a convenient way to model what is happening but the only thing that is posed as ‘reality’ is the actual observations not the conceptual framework for modeling the occurrences. This view may be encapsulated by Dirac: “one should put one's trust in a mathematical scheme, even if the scheme does not appear at first sight to be connected with physics,” and even older to Francis Bacon “[i]n these matters, therefore, truth and usefulness are the very same thing; and practical applications of scientific results are of greater value as pledges of truth than as contributing to the comforts of life.” That is, science’s ‘rejection’ of the many worlds interpretation is one of epistemology, i.e. “the interpretation is model is mathematically identical, applied to reality it makes identical claims and predictions, therefore it should be thought of just as an interpretation up to philosophy, until and unless there is some deviation, whether you want to use the word ‘decoherence’ or ‘collapse’ is irrelevant semantics.” This is something of a positivist view, but one that is simply based in humility and not demanding hypotheses to explain things that do not affect observations. It is not “anti-scientific” to believe in MWI or wave collapse because of reasons of parsimony or a belief in “the insanity of a global single world on a gut level” it is just a-scientific. The good scientist who ascribes to empirical methods and prioritizes the discernable model of reality which can be formalized and tested and usefully turned to a practical understanding, when it comes to interpreting the cause of model until there is some reason to prefer things one way or another they, like Newton, feign no hypothesis. If two proposed interpretations are materially identical, identical in every mathematical way, the scientist just considers them tentatively identical, until there is a way to discern a preference.
-Sir Isaac Newton
-David Hume, An Enquiry Concerning Human Understanding
-Eliezer Yudkowsky
A note: I may remake this a longer multi-part essay at some point, but I think starting here would be a good place since I see emphasis on this kind of knowledge overmuchin related communities.
A Telling of Priors, Perspectives and Quantum Worlds:
Let’s imagine a scenario to explore how one considers priors:
I approach a friend (Al) with a coin sitting on my thumb, heads up, I flip the coin and cover it before they see. I then ask Al “heads or tails?” They call heads, to which I ask “how confident are you in your prediction?” Al answers, “oh, I’d say about 51%, I know that fair coins tend to land on the side facing up around 51% of time since I have no reason to think this coin is any different, I would make the same estimation.”
Another friend (Bri) walks past, they see I am covering my hand but have no knowledge of the coin flip or what might be under there. I ask Bri, “what is under my hand?” They guess, “a coin, tails side up.” Asked how confident they are, Bri proclaims: “I am certain, I guess I’d put the odds at 99%, I don’t know why I just have a strong gut feeling about it.”
I think most readers will rationally agree that Al is making a more reasonable prediction, he is deriving a prior estimate by examining what information he has and making inferences from empirical data accessible to him as to the likeliest outcome. From my own empirical training, I would consider this the ‘correct’ approach. Bri, meanwhile, has insufficient information to draw any empirical prior probabilities. What does she do instead? Like a good Bayesian, she just guesses. Now, what is the true prior probability? If we step back as an outside observer, it is either heads or tails, it is already determined, there is nothing probabilistic going on, however, the information available to Al gave him no way to estimate that and we can say with confidence that if we repeated the experiment we would expect Al to guess correctly roughly 51% of the time, matching his estimated probability, while we would have no way of placing any estimation on Bri’s approach which is a pure Bayesian guess.
Stepping back into the scenario, I uncover my hand revealing that it is a coin tails facing up. Bri declares victory and takes it as a true affirmation of her Bayesian prior as she guessed, very nearly correctly, that it was certain to be a coin facing tails up. Al, however, also does not seem to think he was wrong, even knowing the true prior probability was 100% he views his estimate of 51% as a correct deduction of the empirical evidence available to him. At the same time, he still holds that Bri’s prediction—despite being correct—was irrational, is this view valid?
My contention is that Bri’s method is wrong, the ‘correct’ answer from Bri would have been “I don’t know” and “while I have my hunches, I have no reason to assign any degree of confidence.” Priors not backed by any empirical reasoning may sometimes match our results but when they are not backed with rigor and empirical observations, they are no more trustworthy than any other intuition—which doesn’t mean intuition is useless, but it is not a rational approach to take at face value. One can use processes of Bayesian inference (such as utilizing Monte Carlo experiments) to derive a strong foundation for a prior, but until those are done priors that aren’t supported even where one personally extremely strong priors should be reported as unknowns. For example, I agree with Phil Plait in writing on the risk of dying by alien invasion: “Because of the lack of data, we have a true unknown here. Personally, using just my guts and hunches, I would put the odds in the range of billions-to-one against, but that is not very scientific. So to be truly skeptical, as any real scientist is, I will have to leave this blank, and hope that advances in astrobiology will allow us to make some safe estimates of the odds sometime soon." (Phil Plait, Death from the Skies). Like Plait, I have a very strong prior, going on my intuition and guesses—which I certainly could rationalize if asked—I would place myself as >99.999999% certain that I will not die in an alien invasion. But, since I lack any empirical data and rigor to justify that prior, if asked my response would not be one of confidence but one of “I don’t know,” this is the epistemic humility traditionally advocated by empiricists and the sciences. To be a skeptic, of the empirical sort, is not to reject Bayesian inferences, but to portion your claims with humility and not claim to encapsulate what is unknown. I believe this also leads, at times, to a lack of humility among rationalists in theories and preferences.
While an outdated example, consider for instance the many-worlds-interpretation, championed as obvious by Yudkowsky.
First, is this a fair view of ‘Science’? I don’t believe so. The Copenhagen interpretation (to my understanding, not as a professional physicist) typically is taken to posit collapse as part of the model not measurable reality. It is a convenient way to model what is happening but the only thing that is posed as ‘reality’ is the actual observations not the conceptual framework for modeling the occurrences. This view may be encapsulated by Dirac: “one should put one's trust in a mathematical scheme, even if the scheme does not appear at first sight to be connected with physics,” and even older to Francis Bacon “[i]n these matters, therefore, truth and usefulness are the very same thing; and practical applications of scientific results are of greater value as pledges of truth than as contributing to the comforts of life.” That is, science’s ‘rejection’ of the many worlds interpretation is one of epistemology, i.e. “the interpretation is model is mathematically identical, applied to reality it makes identical claims and predictions, therefore it should be thought of just as an interpretation up to philosophy, until and unless there is some deviation, whether you want to use the word ‘decoherence’ or ‘collapse’ is irrelevant semantics.” This is something of a positivist view, but one that is simply based in humility and not demanding hypotheses to explain things that do not affect observations. It is not “anti-scientific” to believe in MWI or wave collapse because of reasons of parsimony or a belief in “the insanity of a global single world on a gut level” it is just a-scientific. The good scientist who ascribes to empirical methods and prioritizes the discernable model of reality which can be formalized and tested and usefully turned to a practical understanding, when it comes to interpreting the cause of model until there is some reason to prefer things one way or another they, like Newton, feign no hypothesis. If two proposed interpretations are materially identical, identical in every mathematical way, the scientist just considers them tentatively identical, until there is a way to discern a preference.