Uninformed Elevation of Trust

by shminux1 min read28th Dec 20209 comments

18

Rationality
Frontpage

I don't know if there is a standard name for this phenomenon (other than the related Gell-Mann amnesia effect, e.g. "The Sequences are great, except in my area of expertise, where they are terrible.").

Here is the gist: we trust the data as much as we trust the source, regardless of how much the source trusts the data.

This sounds unobjectionable on the surface. We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data whenever we have no independent means of checking the veracity of the data. What is lost in this near-automatic logic is one small piece: the credence the source itself assigns to the data

The (faulty) Bayesian math here is pretty straightforward and is left as an exercise to the reader.

A few examples: 

  • An investment tip from an insider is perceived as reliable even if the insider themselves would consider it a speculation at best.
  • Life on Venus!
  • US election fraud!
  • A tupperware party.
  • A potential hire introduced by a trusted friend or a coworker.
  • A CFAR workshop. Just kidding, those are super trustworthy.

I think the titular description, Uninformed Elevation of Trust, tends to capture the essence of what happens, quickly and naturally, whenever we forget or neglect to engage our critical reasoning full on. It's an elevation of trust, not just adjustment of it, because the scenario where we skip the critical evaluation happens more the more we trust the source of data. We just naturally assume that whatever they tell us is as trustworthy as they themselves are, even though it's obvious that it should be discounted by the factor equal to the source's degree of belief in what they tell us. 

Please feel free to offer your own examples and counter examples. 

18

9 comments, sorted by Highlighting new comments since Today at 12:15 AM
New Comment

The Sequences are great, except in my area of expertise, where they are terrible

Can I get an example of a section of The Sequences where someone with the relevant area of expertise would say that that it's terrible?

That's not relevant to the subject of the article, but, since you asked, the pattern is that if you talk to a philosopher, they point out a million holes in the philosophical aspects of the Sequences, if you talk to a mathematician, they will point to the various mathematical issues, if you talk to a statistician, they will rant about the made-up divide between Bayesian and frequentist approaches, if you talk to a physicist, they will point out the relevant errors and mischaracterizations in the parts that deal with physics. Basically, pick a subject area and talk to a few experts, and you will see it. 

Roll to disbelieve.

The value of specific examples is that we can check whether the critics seem to know what they're talking about, both in their field (do they understand the ground truth) and regarding the sequences themselves (do they know what Eliezer is saying). Simply telling us there are many examples does not, I believe, fulfill the intent of the question. Which is fine, you have no obligation to answer it, but I think it's worth pointing out.

To be clear, I'm sure you can find people in each of those groups making reach of those criticisms. I do not believe those criticisms would be consensus in each of those groups. Certainly not all of them and on the level of "this is terrible". I remember, for example, physicists talking about the quantum mechanics sequence like "yeah, it's weirdly presented and I don't agree with the conclusion, but the science is basically accurate".

In retrospect, I should not have mentioned the Sequences as an example, it's a sensitive topic here. I personally learned a lot from them in the areas outside my expertise. 

I'm confused... is this supposed to be an ironic demonstration of Gell-Mann amnesia, or...?

The bit in the title about the Sequences? Yes.

There's a lot of that in the comments, not that anyone reads the comments.

Here is the gist: we trust the data as much as we trust the source, regardless of how much the source trusts the data.

Rant incoming, apologies. This is, sadly, not correct from the get-go. In general, besides your example which more closely is attributed to some form of psychological bias, we tend to lend source importance based on the lack of trust from a source. However, that implies there is any source vetting whatsoever.


I am sure there are a number of individuals here who have worked intelligence, and I am lucky enough to have both worked intelligence and ditched intelligence, so I'm not very interested in my NDA.


There is something incredible about being source of raw intelligence to the point where I have trouble trusting anything I do not hear and verify for myself so many years later. This election drove me absolutely insane. Not only for a particular side's tendencies, but everyone's failure at source vetting. Abandon the 'source'. Find out the true source. I ordered 50 year old magazines to double check a transcription. Nice collector's piece, though. The transcription was right btw.


The source itself is not relevant unless you are the collector; then what matters is not how you present it, but who it goes to, and how it goes to them. However, when you are consuming collected intelligence/information, you have to weigh the medium against the data. All information is neccesarily useful, when used correctly, even misinfo. Misinformation and propaganda neccesarily tells the bias of the consumer/creator, and can lead you in trends, since propaganda tends to be single or sparsely-sourced.


For instance: You can get a reasonable understanding of an event minus all the important bits through the news media.

You can get zero understanding of an event, and a great amount of confusion, from social media.

When I read a news article, depending on the importance of the event (ie: Trump signs stimulus bill!), I will go make sure it is consistent across reporters. If it is, it tells me one of two possibilities:

a. The source all the reporters got was similar or the same. Specific details can tell you this quite easily.

b. All the reporters are in league together and conspiring against this particular news item


Since b is highly improbable (aside from the possibility of accidental conspiring, which is completely in realms of possibility), I generally stick with a.

If it's not very important, I don't waste my time. It was probably a waste of time reading the article; news reporters are very droll. Thankfully, they have mastered the art of the thesis statement.

Given my experience with classified info, how can I rate news media's accuracy or otherwise on relevant subjects they have reported on related to my career?

Absolutely awful, and generally mischaracterizing, if not completely libel. They are easily one of the most dangerous groups that can be unleashed on anything that has the word secret anywhere near it. It is really hard to manipulate the media; they would prefer to not report something if it is not potentially breaking. It's why I think the social media conspiracy is a particularly good one; I can completely believe the or a algorithm can be trained to filter out specific posts, because it's not very hard to do, and those posts are very predictable. I absolutely think it's happening. Was there voter fraud? Probably. Who did it? Probably not us. To bet on there being no foreign meddling in the USA's elections is already a lost bet.

Trump referred to this as more secure than Afghanistan's election. Well, we designed that for them. It was so bad, they agreed to just both be president. I wish I was joking. Ghani just bullied the other individual out and he got the title Peace Negotiator. Thankfully, Trump will get no such title.

This sounds unobjectionable on the surface. We tend to equate the reliability of the data with the subjectively perceived trustworthiness of the source of data whenever we have no independent means of checking the veracity of the data. What is lost in this near-automatic logic is one small piece: the credence the source itself assigns to the data

You have no idea how the source assigns credence to the data. It is easy to obfuscate or lie. Lying happens even with people you trust. Imagine how it goes for everyone you don't care about. Well, we all know.

This all neglects source protections, which aren't as important unclassified. Suffice to say, there is nothing more important than the source itself, yet nothing as completely worthless. The importance lies in not understanding why the source is how it is, but rather in keeping it consistent.


Bad information is just as useful as good, what matters is whether you are cognizant of its quality without too much due effort (given the expiry date on data). That is most characterized by subject matter expertise.


/rant

I think we are talking about different phenomena here. My point is that, if an average person trusts a source, they tend to assume that the validity of the data are the same as the validity of the source (high!) even if the source themselves takes their own data with the grain of salt, and advises others to do so. You are not an average person by any means, so your approach is different, and much better. 

My personal experience with anything published is that it is 50% lie and 50% exaggeration. And yet I still rely on the news to get a picture of something I have no first-hand access to. But that's the usual Gell-Mann amnesia. I am describing something different: forgetting to account of the source's own uncertainty even if it is spelled out (like in the case of potential life artifacts on Venus) or easily queried or checked (e.g. for a stock tip from a trusted source: "how much did you personally invest in it based on your data?").