In The 5-Second Level and SOTW: Be Specific, Eliezer mentions being influenced by this example:

"Beware, demon!" he intoned hollowly.  "I am not without defenses."
"Oh yeah?  Name three."

-- Robert Asprin, Another Fine Myth

Examples of "proof by theory"

That someone has a theory that supports something is evidence for something.

Examples

1. Once 3 people tell us something, we believe it. Some people think it, so it's true. Even knowing they are in cahoots and trying to manipulate us. I cannot source the study, but try it. It is scarily effective.

2. Ancel Keys formulated his dietary fat / heart disease hypothesis in the 1950s. Over a period of 3-4 years he moved from "hypothesis" to "almost certain" even though no new evidence arose in support of the hypothesis. It appears that every time he wrote on the issue, he noted that he himself, a very intelligent and credible authority, believed the theory, which seemed to weigh in favour of the theory. He cited his own previous papers which then added to the weight of the case, in his mind. [Keys may also have been influenced by the fact that his chief rival John Yudkin believed that sugar was the chief culprit, which view was therefore clearly wrong (theory in this case as anti-evidence). We are still sorting through the wreckage of his catastrophe].

3. Teenage fashions in clothes and politics. Teenagers are very concerned about acceptance by the group, and at the same time they have little experience and knowledge. So they seek cues from those around them as to what fashion statements and political opinions are acceptable. They are seeking cues from those around them, who are just as clueless as they are. Result: strongly held but more or less random fashions and opinions. One late teen recently told me he considers himself fortunate indeed to have been born at that one magic time when his peer group adhered to basically every right and true political and social opinion.

4. Contagion in financial markets. Didier Sornette has had some success in modeling the structure of financial bubbles and crashes based on the premise that speculators are very anxious about the direction of prices and highly uncertain about them at the same time. They have very little good information about future prices. In Sornette's model, traders take cues from traders they are in contact with, resulting in violently fluctuating "phase changes" in investor opinion leading to log-periodic hyper-exponential price moves. Again the opinions of other traders are taken as data when in fact they have little information content.

When I was taught the incompleteness theorem (proof that there are true mathematical claims that cannot ever be proven), I wished for an example of one of its unprovable claims. Math is a very strange territory. You will often find proofs of the existence of extraordinary things, but no instance of those extraordinary things. You can know with certainty that they're out there, but you might never get to see one. Without examples, we must always wonder if the troublesome cases can be confined to a very small region of mathematics and maybe this big impressive theorem will never really actually impinge on our lives in any way.

The problem is, an example of incompleteness would have to be a true claim that nobody could prove. If nobody could prove it, how would we recognise it as a true claim?

Well, how do we know that the sun will rise again tomorrow? We know that it rose before, many times, it's never failed, there's no reason to suspect it wont rise again. We don't have a metaphysical proof that the sun will rise again tomorrow, but we don't really need one. There is no proof, but the evidence is overwhelming.

It occurred to me that we could say a similar thing about the theorem P ≠ NP. We have tried and failed to prove or disprove it for so long that any other field would have accepted that the evidence was overwhelming and moved on long ago. A physicist would simply declare it a law of reality.

I was quite happy to find my example. It wasn't some weird edge case. It's a theorem that gets used every day by computer scientists to triage their energies, see, if you can prove that a problem you're trying to solve is equivalent or stronger than a known NP problem, you would be well advised to assume it's unsolvable, even though we wont ever be able to prove it (although, admittedly, we haven't been able to prove that we wont ever be able to prove it, that too seems fairly evident, if not guaranteed)