And I agree, research as it is currently practiced is (to the best of my knowledge) very unoptimised. There are easily correctable inefficiencies in research that could be eliminated, to increase the speed of converging on the correct theories. Merely practicing Bayesian updating is not enough to eliminate all the inefficiences in research. I get the sense that more is possible. but as best as I can tell, the methods I propose in this post is an optimum one. ,
Table of Contents
A Mathematical Model of Scientific Research
Efficient Experimental Selection
Optimised Experiment Design
Optimised Hypothesis Selection
Brief Summary
A Mathematical Model of Scientific Research
Introduces the model I use to analyse scientific research, and lays the foundations for the entire project. Without this paper, none of the rest would be intelligible.
Efficient Experimental Selection
When confronted with a research problem, what atrategy should we adopt when selecting experiments in order to converge on the correct hypothesis as soon as possible? How do we optimise our experiments in order to make the experimental process as efficient as possible?
Optimised Experiment Design
How do we design our experiments so that they are optimal? So that we only consider the best experiments at any point in time. How do we further optimise our research to make it as efficient as possible?
Optimised Hypothesis Selection
How do we select our hypotheses so that we only consider the best hypotheses? How do we prevent ourselves from wasting time on hypothesis that are not true (and not likely to be true). How do we optimise our research in the very hypotheses we investigate?
I ~expect to publish the first post within a (week p = (0.33)/fortnight p = (0.5), month p = (0.9)) of writing this~ have vastly underestimated the project, and have no idea when I'll finish. I think the first paper should be done before the end of this year (I'm too busy to write it right now, but I'm about halfway done, and would be free again in December). As for the remaining posts, I do not know how many they will be (apart from the 3 I've highlighted above), when the next post would be out (final year undergrad student, so I'm far busier than I like) ~or if I would ever complete this series~ I intend to make an extraordinary effort to see the series . After I write the first post (and possibly others), I may open the series to community input (in the sense that others can add to the series).
Arguably the biggest source of inefficiency in scientific research is perverse incentives. (See http://online.liebertpub.com/doi/pdf/10.1089/ees.2016.0223) This is a sociological problem rather than a statistical problem.
The research payoff curve is fat-tailed (some research, leads to $10^12, other research leads to -$10^2. This makes traditional optimisation methods dangerous. (Think like a Hollywood Producer or Venture Capitalist rather than a Engine designer. One "hit" can compensate for a hell of a lot of misses.)
Once a hypothesis (or hypothesis space) has been formulated to the degree that there are computable probabilities, your research work is 9/10ths done. The bit after that can arguably be outsourced or automated.
Traditional science is too slow Yudkowsky says. Traditional science finds it acceptable for you to waste 30 years on a false theory as long as you disconfirm it at the end. Traditional science allows you to privilege hypothesis as long as you duly test them. Traditional science does not prevent you from 'discovering' magic explanations. Traditional science is insufficient. Traditional science has been tested and found wanting.
And I agree, research as it is currently practiced is (to the best of my knowledge) very unoptimised. There are easily correctable inefficiencies in research that could be eliminated, to increase the speed of converging on the correct theories. Merely practicing Bayesian updating is not enough to eliminate all the inefficiences in research. I get the sense that more is possible. but as best as I can tell, the methods I propose in this post is an optimum one. ,
Table of Contents
Brief Summary
A Mathematical Model of Scientific Research
Introduces the model I use to analyse scientific research, and lays the foundations for the entire project. Without this paper, none of the rest would be intelligible.
Efficient Experimental Selection
When confronted with a research problem, what atrategy should we adopt when selecting experiments in order to converge on the correct hypothesis as soon as possible? How do we optimise our experiments in order to make the experimental process as efficient as possible?
Optimised Experiment Design
How do we design our experiments so that they are optimal? So that we only consider the best experiments at any point in time. How do we further optimise our research to make it as efficient as possible?
Optimised Hypothesis Selection
How do we select our hypotheses so that we only consider the best hypotheses? How do we prevent ourselves from wasting time on hypothesis that are not true (and not likely to be true). How do we optimise our research in the very hypotheses we investigate?
I ~expect to publish the first post within a (week p = (0.33)/fortnight p = (0.5), month p = (0.9)) of writing this~ have vastly underestimated the project, and have no idea when I'll finish. I think the first paper should be done before the end of this year (I'm too busy to write it right now, but I'm about halfway done, and would be free again in December). As for the remaining posts, I do not know how many they will be (apart from the 3 I've highlighted above), when the next post would be out (final year undergrad student, so I'm far busier than I like) ~or if I would ever complete this series~ I intend to make an extraordinary effort to see the series . After I write the first post (and possibly others), I may open the series to community input (in the sense that others can add to the series).