Estimating the cost-effectiveness of research

by owencb2 min read11th Dec 20142 comments


Fermi Estimation
Personal Blog

At a societal level, how much money should we put into medical research, or into fusion research? For individual donors seeking out the best opportunities, how can we compare the expected cost-effectiveness of research projects with more direct interventions?

Over the past few months I've been researching this area for the Global Priorities Project. We've written a variety of articles which focus on different parts of the question. Estimating the cost-effectiveness of research is the central example here, but a lot of the methodology is also applicable to other one-off projects with unknown difficulty (perhaps including political lobbying). I don't think it's all solved, but I do think we've made substantial progress.

I think people here might be interested, so I wanted to share our work. To help you navigate and find the most appropriate pieces, here I collect them, summarise what's contained in each, and explain how they fit together.

  • I gave an overview of my thinking at the Good Done Right conference, held in Oxford in July 2014. The slides and audio of my talk are available; I have developed more sophisticated models for some parts of the area since then.
  • How to treat problems of unknown difficulty introduces the problem: we need to make decisions about when to work more on problems such as research into fusion where we don't know how difficult it will be. It builds some models which allow principled reasoning about how we should act. These models are quite crude but easy to work with: they are intended to lower the bar for Fermi estimates and similar, and provide a starting point for building more sophisticated models.
  • Estimating cost-effectiveness for problems of unknown difficulty picks up from the models in the above post, and asks what they mean for the expected cost-effectiveness of work on the problems. This involves building a model of the counterfactual impact, as solvable research problems are likely to be solved eventually, so the main effect is to move the solution forwards. This post includes several explicit formulae that you can use to produce estimates; it also explains analogies between the explicit model we derive and the qualitative 'three factor' model that GiveWell and 80,000 Hours have used for cause selection.
  • Estimating the cost-effectiveness of research into neglected diseases is an investigation by Max Dalton, which uses the techniques for estimating cost-effectiveness to provide ballpark figures for how valuable we should expect research into vaccines or treatments for neglected diseases to be. The estimates suggest that, if carefully targeted, such research could be more cost-effective than the best direct health interventions currently available for funding.
  • The law of logarithmic returns discusses the question of returns to resources into a field rather than on a single question. With some examples, it suggests that as a first approximation it is often reasonable to assume that diminishing marginal returns take a logarithmic form.
  • Theory behind logarithmic returns explains how some simple generating mechanisms can produce roughly logarithmic returns. This is a complement to the above article: we think having both empirical and theoretical justification for the rule helps us to have higher confidence in it, and to better understand when it's appropriate to generalise to new contexts. In this piece I also highlight areas for further research on the theoretical side, into when the approximation will break down, and what we might want to use instead in these cases.
  • How valuable is medical research? written with Giving What We Can, applies the logarithmic returns model together with counterfactual reasoning to produce an estimate for the cost-effectiveness of medical research as a whole.

I've also made a thread in LessWrong Discussion for people to discuss applications of one of the simpler versions of the cost-effectiveness models, to get Fermi estimates for the value of different areas.


2 comments, sorted by Highlighting new comments since Today at 1:11 AM
New Comment

This is an extremely important question to ask and to research, not to speak of answering it properly. It complements the EA community very nicely and can then help to answer questions like "Should I donate to GiveWell or MIRI?" I am very interested in your results.

This is interesting, especially the conclusion from the "How valuable is medical research?" post that the cost/benefit of medical research is, to "a rough 90% confidence interval, of between $2,000 and $36,000 per (Quality-Adjusted Life Year)". It would be interesting to look at medical research's effect on maximum life span -- how long the most long-lived person has lived -- because medical advances tend to reduce mortality, not increase the life span record (see this Stanford Center of Longevity talk).

A simple model of economic growth (see Mankiw, "Macroeconomics") is that growth in Output (e.g GDP) is the sum of growth in Capital, Labor, and Total Factor Productivity. A case study in this text argues that South Korea's boom from the 1950s to the 1990s was due to investment in capital (rebuilding after the Korean War) and in labor (large increases in education), and technology improvement did not significantly affect economic growth.

My naive reading is that Capital (e.g. factory equipment) and Labor (e.g. workers) are both limited by resource availability on Earth but offer relatively large returns when resources are available, while Total Factor Productivity (e.g. technology) can improve without limit but is relatively expensive. Thus, someone wanting to min/max the global economy would go for a lot of population growth and infrastructure projects until the carrying capacity of the Earth was reached, then would invest the remaining capacity in R&D. Unfortunately, this becomes a political and coordination problem: how do get people to agree with your definition of "effectiveness", and how do you motivate millions of people to tackle a shared goal while avoiding corruption, oppression, etc.?