(Cross-posted from my site.)

My goal here is to quickly describe a potential problem that groups of people may have when trying to collectively figure out the truth of some subject.

The basic idea is that people may use each others' beliefs to inform their own beliefs, without properly accounting for the fact that others' beliefs are based on some of the same evidence as their own beliefs.

In what follows I present a simplified example of how this might happen, a bit of math to guide our thinking, and some illustrative plots.

The basic takeaway is that a failure to "de-correlate" others' beliefs from the supporting evidence leads to sub-optimal epistemics. In particular:

- Groups that make this error may be overconfident in their beliefs.
- This can make it harder to recover accurate beliefs if the first people to explore a topic reach conclusions far from the truth.

## An illustrative example

Suppose that Alice is trying to figure out whether the following claim is true:

AGI will be developed sometime in the next 20 years.

Alice starts with some prior, and after investigating this claim updates her belief to be more confident that it is true.

Later, Bob comes along and wants to know whether this claim is true. He investigates the claim, using, for the most part, the same sources of information that Alice used, and arrives at a belief similar to Alice's posterior. He then notes that Alice also updated her belief to be more confident of the claim's truth, and uses that as additional evidence; he thus ends up even more confident than Alice that AGI will be developed in the next 20 years.

Bob's last step is of course an error: if Alice and Bob used the same information to form their beliefs, Bob shouldn't use Alice's belief to inform his own belief -- the information expressed through Alice's posterior belief is the same as the information in the evidence that Bob already reviewed. Bob needs to "de-correlate" the information he sees from the information in Alice's belief -- in this case, he will find that the two sources of information are perfectly correlated, meaning that they are redundant.

We probably aren't so naive as to *completely* fail to decorrelate the information in other people's beliefs from what we see; but it seems likely that we make this mistake to some extent. We may, therefore, want to be a bit skeptical of common beliefs formed in a group where raw evidence is scarce relative to the discussion about how to interpret that evidence. (The claim about AGI was chosen intentionally, with this in mind.)

## A model

It might help to think about how this might happen in a more formal way. If you don't like math, feel free to skip ahead.

Let's suppose that we have people who want to figure out the proper level of credence, , for some claim. As a prior, each person has that is uniformly distributed between 0 and 1.

Now suppose that there are pieces of independent evidence, drawn from . (Each has a probability of being 1 and a probability of being 0.)

In order, each player receives the signal and reviews the beliefs of everyone who already received their signals. (Player observes signal and posteriors of players .)

Denoting the common prior as , we can calculate using Bayes' theorem that player 1, upon observing forms the posterior

In general, if Player properly de-correlates previous signals, they should end up with

The danger is that players repeat-count earlier signals: e.g., player 3 sees beliefs of players 1 and 2 and thinks the two beliefs are based on independent evidence, when in reality player 2 uses the evidence given to player 1. If players (wrongly) assume complete independence of beliefs, player ends up with

Again, the basic idea is that earlier signals get too much attention.

## Some simulations

Below are two plots generated using the model presented in the previous section.

The first plot shows the posteriors that result from a group of 10 people who receive independent evidence. The correct belief in this example is .

You can see that that incorrectly updating beliefs yield a posterior that is too narrow -- it confidently predicts the wrong value.

The second plot shows a case where , but we start with a train of unlikely (possibly mistaken) observations in the other direction.

Using the wrong updating method means that we put too much weight on those initial observations and move back toward the correct belief more slowly.

Related to this, there is a paper I once read, but have no way of locating again, that considered the following model situation (to the best of my recall).

Suppose that there is a disease for which there are two treatments, A and B, that may differ in efficacy. There are many doctors, each of whom comes across this disease from time to time and will prescribe A or B. Each doctor knows the results they have had with the cases they have seen, and based on that can form a preference for A or B. Each doctor also knows all the other doctors’ recommendations — A or B — but nothing about the other doctors’ cases. How much should each doctor weigh the other doctors’ recommendations, given that they know that everyone’s recommendation is a factor in everyone else’s?

It turns out that no matter how many other doctors there are, the weight that each doctor should attach to them should be no more than if there were just a single other doctor.

There is a meta-probability that both Alice and Bob should have, which is that they've misinterpreted the evidence - failed to be Bayesean in their mapping of their sensory input to causal probabilities, or made other errors in modeling or calculation.

Alice's posterior

IS SOME EVIDENCEfor Bob that he has made fewer of these errors than he might fear, or at least made the same errors as Alice. That would support his posteriors a bit. Better, of course, is a discussion of evidence to identify the overlap and novelty in the reasoning.None of which REALLY matters, as neither Alice nor Bob are actually updating this rigorously.

Closely related: https://en.wikipedia.org/wiki/Information_cascade

Thanks for pointing this out -- this is helpful to position this in idea space. Basically what I'm describing here is an information cascade where actors fail to realize that others' beliefs are not independent.