Viliam's Shortform

Viliam

This is a special post for quick takes by Viliam. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

If you dismiss ideas coming from outside academia as non-scientific, you have a point. Those ideas were not properly tested, peer-reviewed, etc.

But if you dismiss those ideas as not worth scientists' attention, you are making a much stronger statement. You are effectively making a positive statement that the probability of those ideas being correct is smaller than 5%. You may be right, or you may be wrong, but it would be nice to provide some hint about why you think so. Are you just dissing the author; or do we have an actual historical experience that among ideas coming from a certain reference group, less than 1 in 20 turns out to be correct?

Why 5%? Let's do the math. Suppose that we have a set of hypotheses out of which about 5% are true. We test them, using a p=0.05 threshold for publishing. That means, out of 10000 hypotheses, about 500 are true, and let's assume that all of them get published; and about 9500 are false, and about 475 of them get published. This would result in approximately 50% failure in replication... which seems to be business as usual in certain academic journals?

So if it is perfectly okay for scientists to explore ideas that have about 5% chance of being correct, then by saying that certain idea should not be explored scientifically, you seem to suggest that its probability is much smaller.

Note that this is different from expecting an idea to turn out to be correct. If an idea has a 10% chance of being correct, it means that I expect it to be wrong, and yet it makes sense to explore the idea seriously.

(This is a condensed version of an argument I made in a week old thread, so I wanted to give it a little more visibility. On top of that, I suspect that status concerns can make a great difference in scientist's incentives. Exploring ideas originating in academia that have a 5% chance of being right is... business as usual. Exploring ideas originating outside of academia that have a 5% chance of being right will make you look incompetent if they turn out to be wrong, which indeed is the likely outcome. No one ever got fired for writing a thesis on IBM, so to speak.)

AlphaFold doesn't come out of academia. That doesn't make it non-scientific. As Feymann said in his cargo-cult science speech, plenty of academic work is not properly tested. Being peer-reviewed doesn't make something scientific.

Conceptually, I think you are making a mistake when you treat ideas and experiments as the same and equate the probability of an experiment finding a result as the same as the idea being true. Finding a good experiment to do to test an idea is nontrivial.

A friend of mine was working in a psychology lab and according to ... (read more)

3phdead2y

I think its important to disambiguate searching for new problems and searching for new results. 1. For new results: while I have as little faith in academia as the next guy, I have a web of trust in other researchers who I know do good work, and the rate of their work being correct is much higher. I also give a lot of credence to their verification / word of mouth on experiments. This web of trust is a much more useful high pass filter for understanding the state of the field. I have no such filter for results outside of academia. When searching for new concrete information, information outside of academia is not worth scientists interests due to lack of trust / reputation 2. When it comes to searching for new hypotheses / problems, an important criterion is how much you personally believe in your direction. You never practically pursue ideas with 10% probability: you ideally pursue ideas you think have a fifty percent probability but your peers believe have a 15% probability. (This assumes you have high risk tolerance like I do, and are okay with a lot of failure. Otherwise, do incremental research). For problem generation, varied sources of information are useful, but the belief must come intrinsically. 3. When searching for interesting results to verify and replicate, its open season. As a result, I think that ideas outside academia are not useful to researchers unless the researchers in question have a comparative advantage at synthesizing those ideas into good research inspiration. As for nonideal reasons for ignoring results outside academia, I would more blame reviewers rather than vague "status concerns" and a general low appetite for risk tolerance despite working in an inherently risky profession of research.

3DirectedEvolution2y

Well, ideas from outside the lab, much less academia, are unlikely to be well suited to that lab’s specific research agenda. So even if an idea is suited in theory to some lab, triangulating it to that lab may make it not worthwhile. There are a lot of cranks and they generate a lot of bad ideas. So a < 5% probability seems not unreasonable.