The Paradox of Expert Opinion

by Emrik5 min read26th Sep 20219 comments


Filtered EvidenceGoodhart's LawWorld Modeling

The best-informed opinions tend to be the most biased ones.

(This is not just the sophistication effect, although it is tangentially relevant.)

If you want to know whether string theory is true and you're not able to evaluate the technical arguments yourself, who do you go to for advice? Well, seems obvious. Ask the experts, the string theorists themselves. They're likely the ones most informed on the issue, and the people most likely to be able to talk coherently about it. Unfortunately, they're also a group heavily selected for believing in the hypothesis. It's unlikely they would bother becoming string theorists in the first place unless they believed in it.

Another example is religious experts. If you want to know if God exists, who do you ask? Again, the people most informed on the issue, religious experts, have also been heavily selected for the fact that they believe in it. Not every expert decided to study religion because they were believers, but it's enough of an effect that 70% of philosophers of religion in the PhilPaper Survey accept or lean towards theism, compared to only 16% of all philosophers.

More relevant to this crowd may be the question of whether to take transformative AI seriously. The experts who have thought the most about this question are likely to be the ones who take the risk seriously. This means that the most technically eloquent arguments are likely to come from the supporter side, in addition to hosting the greatest volume of persuasion. Note that this will stay true even for inane causes like homeopathy: I'm a firm disbeliever, but if I were forced to participate in a public debate right now, my opponent would likely sound much more technically literate on the subject.

And if you take an outside-view approach and look at surveys of the most informed AI risk experts, there are obvious problems with that. Surveys including AI researchers of all stripes are likely better since they've not been filtered for belief as strongly. I'm not saying this is new. The people who run these surveys are well-aware of this effect, and run them with the understanding that this is imperfect information.

(If we put our Kuhnian hats on, we may start to speculate about any popular paradigm in science. The standard model? You could argue that physicists as a group are selected for belief in the standard model, and "dissent" often gets reclassified as "failure to learn". I won't argue for it, but it's a fun hat to wear to parties.)

Systemic bias is inevitable

This paradox stays true even in worlds where all researchers are perfectly rational (assuming they start out with the same priors). Let everyone have a Solomonoff inductor with infinite computational power in their brains, and it will still be true.

As long as

  1. researchers are exposed to different pieces of evidence (aka evidential luck), and
  2. choose which field of research to enter based on something akin to Expected Value of Information,[1] and
  3. the field has higher EVI the more you accept its premises,

then the experts of that field will be a group that's been selected for belief in those premises, to some extent.

If you're confused about why (1) is a precondition: it's because if all the researchers were perfectly rational and exposed to the same evidence, they would all end up with the same beliefs and there would be no subgroup of beliefs to select for. Whereas if (1) is true, then you can have perfectly rational researchers arriving at different beliefs due to evidential luck, and now the expert-making mechanism can select for specific beliefs and produce a systemic bias.

Regressional Goodhart amplifying personal biases

But in the real world, none of us are perfectly rational. And in selecting experts, you are implicitly selecting for higher personal biases, worsening the Paradox. Too see why, let's consider the same situation as I outlined above, except now researchers choose which fields to enter based on more things than just EVI, in addition to having personal biases that corrupt their estimation of EVI.

Let a researcher's motivation  to study a particular field be a combination their personal biases , and the  they would have assigned to it if they were completely rational,[2] and whatever additional randomness is left , such that .

Now, if you have a process of selection that optimises for , such as the process for producing experts, then that process will be selecting for higher values of all the three factors. Which means that the people with the most motivation to study a field will statistically be more biased in favour of its premises. This is an example of Regressional Goodhart or Winner's Curse.

Conformity pressure inside research communities

Once a student has managed to get into an advanced research community (having gone through systemic and personal bias filters that partially sifts out disagreement with the field's premises), they will now be faced with a wide variety of hard-to-avoid conformity pressures. This has been extensively covered before and I won't rehash the case here, except for linking to information cascades.

Adversarial Goodhart amplifying deception

In the real world, there are also well-known problems with the incentives experts often face, especially in academia. Thus, Adversarial Goodhart:

When you optimize for a proxy, you provide an incentive for adversaries to correlate their goal with your proxy, thus destroying the correlation with your goal.

Whatever metric we use to try to determine expertise (let's roughly define this as "how good they actually are") in a topic, researchers are going to have an incentive to optimise for that metric. And since we can't observe expertise directly, we're going to have to rely very heavily on proxy measures.

Empirically, it looks like those proxies include: number of citations, behaving conspicuously "professional" in person and in writing, confidence, how difficult their work looks, and a number of other factors. Now, we care about actual expertise , but, due to the proxies above, the metric  will contain some downward or upward error  such that .

When we reward researchers for having a high , we incentivise them to optimise for both  and . They can do this by actually becoming better researchers, and/or by increasing the Error—how much they seem like an expert in excess of how expert they are. When we pick an individual with a high , that individual is also more likely to have a high . Adversarial Goodhart makes us increasingly overestimate expertise the higher up on the metric distribution we go.

(Of course, you could argue that academia is much too honourable an institution to succumb to mere incentive gradients. Surely, if we look at the real world, we see no evidence at all of absolutely catastrophic academic alignment failures...)

  1. Of course, how human researchers choose fields of research is much more complicated, but EVI is a significant part of it. Humans tend to research what they find interesting, yes, but the feeling of interestingness is partly based on intuitions about EVI. So this premise isn't completely off base. ↩︎

  2. This is so that we can more clearly see the additional effect personal biases play over and above the systemic bias described in the previous section. ↩︎


9 comments, sorted by Highlighting new comments since Today at 6:03 AM
New Comment

The examples make me think of reference class tennis.

A central question related to this post is "which reference class should you use to answer your question?" A key point is that it depends on how much selection pressure there is on your reference class with respect to your query.

The examples don't work. For string theory, the math of it is meaningful regardless of whether it holds about our world, and in physics consensus reliably follows once good evidence becomes available, so the issue is not experts misconstruing evidence. For homeopathy, there is a theoretical argument that doesn't require expertise. For religion, the pragmatic content relevant to non-experts is cultural/ethical, not factual. The more technical theological claims studied by experts are in the realm of philosophy, where some of them are meaningful and true, it's their relevance that's dubious, but philosophical interest probably has validity to it vaguely analogous to that of mathematical interest.

I'm not sold yet on why any of the examples are bad?

I know very little of string theory, so maybe that's the one I think is most likely to be a bad example. I assume string theorists are selected for belief in the field's premises, whether that be "this math is true about our world" or "this math shows us something meaninfwl". Physicists who buy into either of those statements are more likely to study string theory than those who don't buy them. And this means that a survey of string theorists will be biased in favour of belief in those premises.

I'm not talking inside view. It doesn't matter to the argument in the post whether it is unreasonable to disagree with string theory premises. But it does matter whether a survey of string theorists will be biased or not. If not, then that's a bad example.

Math studied by enough people is almost always meaningful. When it has technical issues, it can be reformulated to fix them. When it's not yet rigorous, most of its content will in time find rigorous formulations. Even physics that disargees with experiment can be useful or interesting as physics, not just as math. So for the most part the wider reservations about a well-studied topic in theoretical physics are not going to be about truth, either mathematical or physical, but about whether it's interesting/feasible to test/draws too much attention/a central enough example of physics or math.

This paradox stays true even in worlds where all researchers are perfectly rational.

No, if there is common knowledge of rationality then Aumann's Agreement Theorem beats evidential luck.

Maybe the "everyone's rational" world still allows different people to have different priors, not just evidence?

Right, good point. Edited to point out that same priors (or the same complexity measure for assigning priors) is indeed a prerequisite. Thanks!

This is a good point in the sense that communication between the researchers could in theory make all of them converge to the same beliefs, but it assumes that they all communicate absolutely every belief to everyone else faster than any of them can form new beliefs from empirical evidence.

But either way it's not a crux to the main ideas in the post. My point with assuming they're perfectly rational is to show that there are systemic biases independent of the personal biases human researchers usually have.

(Edit: It was not I who downvoted your comment.)