People tend to form what I guess I'm going to call default-hypothesis-reference-classes. This is what they see as the "default" hypothesis that new hypotheses have to compete against. Obviously this isn't how a perfect bayesian reasoner would act, but we're only human.
These weird reference classes can be tricky, because you can get "sucked into them" and stop noticing that they're weird.
I started thinking of this when thinking about the CICO model of weight gain/loss. Calories-in-calories-out sounds obviously true under thermodynamics. There's about 2 × 1010 Calories in a gram of matter, and humans need about 2000 Calories per day, so make sure not to eat more than a tiny fraction of a gram per day and you're sure to lose weight.
Well, not quite. In my opinion, the most "natural" default reference class for weight gain or loss is mass-in-mass-out (MIMO), which is obviously true, if not super useful. This has to be true - everything we know about physics tells us it's impossible for your body to gain or lose mass unless it comes from or goes somewhere.
What people mean when they say CICO is that different foods have different amounts of bioavailable energy, and weight gain is dependent on difference between the bioavailable energy you eat and the amount of heat your body produces. This doesn't seem like it necessarily has to be true! People gain weight when they grow, and that's partially because they're eating actual minerals that are deposited into their bones. (And these minerals largely don't go anywhere once they're deposited.) So that's a straightforward counterexample to CICO.
We could imagine the same thing happening with fat gain or loss. Your body could store energy by accumulating fat, but not remove the actual fat molecules from your body when their chemical energy is consumed. (I'm not saying that this actually happens, just that it's not forbidden by thermodynamics.) Or our model for what kinds of energy are bioavailable could be wrong, and if so we'd give wrong predictions when trying to apply CICO even if the theory itself were true.
What I'm trying to say is, if your previous "default" reference class for models of weight gain was MIMO, CICO would seem weird and not that special to you. CICO is very likely more useful than MIMO, but it wouldn't intuitively seem like it should be the default theory all other theories are at best a special case of.
Once I started looking for weird reference classes, I started seeing them all the time. For example, I once saw some people discussing a study that examined the effect of new housing on "the asking price for 3-bedroom condos". What seemed weird to me is that the study chose a refencence class, "the asking price", which intuitively seems like not the thing we usually want to talk about, which is the sale price.
I think this one was discussed more broadly, but at the beginning of the pandemic I saw a lot of people saying "this country had good outcomes, but their population density is very low" or things like that. But population density is an imperfect metric for what we really care about, which is how clustered a population is. Canada's population density is very low, but you could delete 90% of canada and probably only lose 10% of their population. This fictional decimated-canada would probably have had similar covid outcomes to real-canada, but with a 9x higher population density. So the reference class of "the difficulty of managing covid in a country is dependent on the country's population density" seems like a sketchy choice for a default reference class of covid-management-difficulty prediction generators.