What science needs

by PhilGoetz6 min read2nd Dec 201284 comments

42

Practice & Philosophy of ScienceWorld Optimization
Frontpage

Science does not need more scientists.  It doesn't even need you, brilliant as you are.  We already have many times more brilliant scientists than we can fund.  Science could use a better understanding of the scientific method, but improving how individuals do science would not address most of the problems I've seen.

The big problems facing science are organizational problems.  We don't know how to identify important areas of study, or people who can do good science, or good and important results.  We don't know how to run a project in a way that makes correct results likely.  Improving the quality of each person on the project is not the answer.  The problem is the system.  We have organizations and systems that take groups of brilliant scientists, and motivate them to produce garbage.

I haven't got it all figured out, but here are some of the most-important problems in science.  I'd like to turn this into a front-page post eventually, but now I'm going to post it to discussion, and ask you to add new important problems in the comments.

Egos

A lot of LWers think they want to advance scientific understanding.  But I've learned after years in the field that what most scientists want even more is prove how smart they are.

I couldn't tell you how many times I've seen a great idea killed because the project leader or someone else with veto power didn't want someone else's idea or someone else's area of expertise to appear important.  I've been "let go" from two jobs because I refused when my bosses flat-out told me to stop proposing solutions for the important problems, because that was their territory.

I don't mean that you should try to stop people from acting that way.  People act that way.  I mean you should admit that people act that way, and structure contracts, projects, and rewards so that these petty ego-boosts aren't the biggest rewards people can hope to get.

Too many "no"-men

The more people your project has who can say "no", the worse the results will be.  This is one reason why Hollywood feature films are stupid, why start-ups do good work, and why scientific projects are so often a waste of money.  Good ideas are inherently unpopular.  Most of the projects that I've worked on have been crippled because every good idea ran into someone with veto power who didn't want to do things differently, or didn't want somebody else to get credit for solving the problem.  See "Egos".

Saying "no" to bad projects is important, but once the project is underway, there is a bias to say "no" more than "yes", even after adjusting for the number of times you can say "yes" in total.  Requiring consensus is especially pernicious.  You can't get good results when everbody on the project has to say "yes" to new ideas.

Jurisdiction arguments

Team members often disagree about whose expertise particular decisions fall under.  Most people see how their expertise applies to a problem more easily than they can see how someone else's expertise applies to a problem.  What usually happens is that territorial claims are honored from the top of the org chart on down, and by seniority.  For example, I worked for a computer game company where the founder hired a scriptwriter, then came up with his own story ideas and told the scriptwriter to implement them.  The implementation had no text; the scriptwriter took the story ideas and produced descriptions of scenes acted out with body language.  The animators thought that body motion fell completely within their jurisdiction, so they felt free to rework whatever they saw differently.  The scriptwriter had very little chance for creative input, no control over anything, and very little job satisfaction.

This is a common problem for computer scientists and mathematicians.  Computer scientists and mathematicians see themselves as people who understand how to most-effectively take a set of data, and arrive at the desired results.  This includes figuring out what data to look at, and in the best case, means being involved in the proposal writing to look at possible problems to address, and determine which problems are soluble and which ones are not based on information theory.  This never happens.  People in other specialties see computer scientists as a kind of lab technician to bring on after they've figured out what problem to address, and what data and general algorithm to use.  They see statisticians as people to consult when the project is done and they're writing up the results.  They aren't even aware that these other disciplines can do more than that.

A classic example is the Human Genome Project.  Some people you never hear about, including my current boss, came up with algorithms to take whole-genome shotgun data and assemble it.  Craig Venter went to the leaders of the Human Genome Project and explained to them that, using this approach, they could finish the project at a fraction of the cost.  Anybody with a little mathematical expertise could look at the numbers and figure out on the back of a napkin that, yes, this could work.  But all the decision-makers on the HGP were biologists.  I presume that they didn't understand the math, and didn't believe that mathematicians could have useful insights into biological problems.  So they declared it impossible—not difficult, but theoretically impossible—and plowed ahead, while Craig split off to use the shotgun approach.  Billions of taxpayer dollars were wasted because a few people in leadership positions could not recognize that a problem in biology had a mathematical aspect.

Muzzling the oxen

"Thou shalt not muzzle the ox when he treadeth out the corn."  — Deuteronomy 25:4

I believe that a large number of the problems with scientific research are tolerated only because nothing is at stake financially.  Government agencies have tried very hard to ensure that people do work for their contracts.  You have to say in the proposal what you're going to do, and itemize all your costs, and do what you said you would do, and write reports once a month or once a quarter showing that you're doing what you said you would do.  This results in unfortunate obvious stupidities.  We can spend $30,000 to have an employee write a piece of software that we could have bought for $500, or to solve a problem that a consultant could have solved for $500, but we can't buy the software or hire the consultant because they aren't listed in the contract and the employee is.

But the bigger problem is that the strict financial structure of scientific research makes it illegal to motivate scientists by giving them a percentage of resulting profits.  You simply can't write up a budget proposal that way.  So managers and team members indulge their prejudices and fantasies because the little bit of self-esteem boost they get from clinging to their favorite ideas is worth more to them than the extra money they would earn (zero) if the project produced better results.  Examples of petty prejudices that I've seen people wreck good work to preserve:  top-down over bottom-up design, emacs over vim (I was in a shop once where the founders forbade people from using vim, which had an astonishingly destructive effect on morale), rule-based over statistical grammars, symbolic logic over neural networks, linguistics expertise as more important than mathematical expertise, biological expertise as more important than mathematical expertise, and, always, human opinions gathered from a few hundred examples as more valid than statistical tests performed on millions of samples.

When I read about machine learning techniques being applied in the real world, half the time it's by trading firms.  I haven't worked for one, so I don't know; but I would bet they are a lot more receptive to new ideas because, unlike scientists, they care about the results more than about their egos.  Or at least, an appreciable fraction as much as they care about their egos.

Entry costs

Everybody in science relies on two metrics to decide who to hire and who to give grants to:  What their recent publications are, and what school they went to.  It is possible to go to a non-top-ranked school and then get on important projects and get publication credits.  Someone who just left our company on Friday worked the magic of cranking out good research publications while working as a programmer, always taking on only projects that had good publication potential and never getting stuck with the horrible life-sucking, year-sucking drudgery tasks of, say, converting application X from using database Y to database Z.  I just don't know how he did it.

For the most part, that doesn't happen.  You don't become a researcher; you start out as a researcher.  You need to stay in school, or stay on as a postdoc, until you have your own track record publications and have won your own grant.  You need people to read those publications.  You don't get to work on important projects and get your work read and get a grant because you're brilliant.  You get these things because your advisor works the old-boy network for you.  Whatever your field is, there is a network of universities that are recognized as leaders in that field, and you are more-or-less assured of failure in your career (especially in academia or research) unless you go to one of those universities, because you won't get published in good journals, you won't get read much, and you won't get a big grant.

There are exceptions.  Fiction writers and computer programmers don't need to go to a fancy university; they need credits and experience.  (Computer programmers.  But don't get a Ph.D. in computer science from a non-elite university and imagine you're going to do research; it won't happen.)  Good stories can sort of be recognized; basic knowledge about Enterprise Java can be measured.  Companies have recognized the monetary value of doing so.  But grant review panels and companies don't really know how to rate scientists or managers, so they try to get somebody from MIT or from Wharton, because nobody ever got fired for buying a Xerox.

The value of scientists to their companies may or may not be reflected in their salaries, but the value of those select universities is certainly reflected in the price of tuition.  If your college of choice costs you less than $55,000/yr to attend, including room and board, it will not lead you to success.  Unfortunately, the U.S. government won't loan you more than $10,000/yr for tuition.

(One interesting exception is in cosmology.  I did a study of successful physicists, as measured by their winning the Nobel or being on the faculty at Harvard.  I found that after 1970, no one was successful in physics unless they went to an elite undergraduate college, with a few exceptions.  The exceptions were astrophysicists who went to college in Arizona or Hawaii, where there are inexpensive colleges that are recognized as leading institutions in astronomy because they have big telescopes.)

The single-biggest problem with science today is finding relevant results.  I have had numerous discussions with experts in a field who were unaware of recent (and not-so-recent) important results in their field because they relied on word-of-mouth and a small set of authoritative journals, while I spent half an hour with Google before our meeting.  To take a spectacularly bad example, the literature showing that metronidazole kills Borrelia burgdorferi cysts, while penicillin, doxycycline, amoxicillin, and ceftriaxone do not, is over ten years old; yet metronidazole is never prescribed for Lyme disease while the latter are.

Attention is the most-valuable resource in the twenty-first century.  Producing a significant result is not hard.  Getting people to pay attention to it is.  Scientometric analysis of scientific publications shows that producing more and more papers in a field has very little impact on the number of papers cited (a proxy for number of results used), probably because scientists basically read up to one paper per day chosen from one or two leading journals, and that's it.  They aren't in the habit of regularly, actively searching for things relevant to their work; and frankly, there isn't much motivation to do that, since using Google to answer a specific question is like using excavation equipment to search for a needle in a haystack.

42