I think this is somewhat useful already, as a sort of true-in-spirit rant against the media that is a helpful reminder of things most of us probably already knew.
I'd love to see this grow into a more ambitious project:
1. Get a random collection of media articles you haven't read before, ideally on topics you are unfamiliar with.
2. Spend some amount of time (30min?) on each one, using the current List of Rules to read between the lines and write a short analysis proposing various hypotheses, e.g. "Probably so-and-so had some pretty good reasons for what they did -- I'm guessing it was X or Y. As for the headline of this article, my guess is that the exact opposite is true, because while the events cited in the article did happen and are evidence for the titular claim, they aren't enough evidence to overcome the standard argument, which wasn't even mentioned in the article, presumably because they didn't have a good rebuttal to it."
2.5. In parallel, have some guinea pig volunteers (ideally a diverse group) read your rules & ask some clarifying questions, and then read those same articles and write their own reports as best as they can, making an attempt to follow the rules & noting when they think a rule is unhelpful/counterproductive/false, or just unclear.
3. Compare notes and see how much overlap there was between your own interpretation of the rules and everyone else's.
4. Spend several hours doing a deep-dive investigation into each article, attempting to get to the bottom of who is right -- are the hypotheses you each conjectured in your analyses correct? By the end of this you should consider yourself, if not an expert on the subject, at least fairly well-informed.
5. Reflect on everything that's happened thus far & come up with a new and improved List of Rules that reflects what you've learned about what works and what doesn't. Make some testable predictions for how the new List of Rules will outperform the old List in the upcoming experiment.
6. Run an experiment where fresh guinea pig volunteers are sorted into a control and experimental group, the control group sees the old List and the experimental group sees the new List, then both read & analyze some new articles.
7. Publish the results, perhaps with a List of Rules: Third Edition that incorporates your learnings.
I imagine you won't want to do this because it's a lot of work and you have other things you'd rather do, but if you did do this, I'd be grateful and would read the best-performing List of Rules with keen interest and probably noticeably shift my reading-behavior accordingly. I'd also signal-boost the List to all my friends etc.
Interesting idea.
I'll do a simplified trial run. I went in to Reuters and picked this top article, Brazilian troops clear pro-Bolsonaro camp after protesters storm capital, which is about a subject that I know ~nothing about.
Now according to Zvi, the rules seem to say:
If I was seeing this interpretation in the wild, my median guess would be that it was a batshit crazy conspiracy theory. But again I don't know much about the situation so I would be curious if anyone knows anything that can be said about it.
They also talk about the protestors entering government buildings, but never about any people working in those buildings being afraid or hurt, so according to Zvi's rules this would imply that the buildings were empty or something.
I don't know about the other stuff, but https://www.vox.com/world/2023/1/9/23546507/brazil-bolsonaro-lula-capital-invasion-january-8 says
Congress was in recess at the time, leaving the building mostly empty.
It might be interesting to do a more systematic investigation into how often the media makes errors. Like pick a random set of articles, list the claims and implications they make, and evaluate what proportion of them are true.
"How the media makes errors"?
I think Zvi's point is that errors do not dominate media deceptions. Their writing is a made up of conscious choices that mostly follow clear rules.
I understand OP to be including "misleading implications" as part of the thing to be counted. An additional complication is that the degree of misinformation in media varies widely by subject matter and relevance; everyday articles about things with minimal Narrative impact are usually more reliable. For that reason a random sample of articles probably looks better than a sample of the most impactful and prominent articles.
Excellent point on the selective subject matter placement of articles with misleading implications. Thank you. I should have thought that through.
Scott has now convinced me that the media does not state literal lies in the voice of the article. However, while we're using the phrase 'bounded distrust', I do think that there are two things that are unbounded:
Scott analyzes a couple of examples in his follow-up post Sorry, I Still Think I Am Right About The Media Very Rarely Lying. Pushing narratives of voter fraud, covid being less important than the flu, Clinton being 99% likely to win, etc etc, were all harmful to different amounts in different ways. I haven't thought about which ones are the worst, but my claim is that the terrible-ness of the narrative being pushed is not bounded by much more than what they can get away with people buying in the current culture, while not stating explicit lies in the author's voice. I don't have a cite for this, but my guess is that people performing very good acts can be punished and have a mob set on them and have the goodness of their acts inverted pretty casually in the establishment narrative. Again, not by telling explicit lies, but by present exceedingly misleading information that is selected to be consistent and supportive of a narrative that they can sell.
I think this is a lot of what is captured when people say things like "I wouldn't put anything past the media". Not that they explicitly lie, but that they will support arbitrarily inaccurate and unethical narratives.
I often use the heuristic that if two sources with opposing Narratives both claim that a certain fact is true, it is strong evidence that the fact is indeed true. Are there cases where this heuristic fails? E.g. where both sides claim a fact is true (likely with different motives), but it is actually false?
A category that comes to mind is when one side is a little terrorist group, who wants to claim that they're big and important and powerful, and another side is an authoritarian government (or some branch thereof), who also wants to claim that the little terrorist group is big and important and powerful so they can justify requests for more funding and for encroachment on civil liberties to fight the terrorists. A third side would be those opposing the requests of the second side (i.e. civil liberty advocates and small-budget advocates), but they're not always represented.
Generally, be looking for unrepresented third sides that one of the first two sides is exploiting (and the other doesn't care about or is also exploiting).
Bounded Distrust is an important addition to my personal lexicon, and this is a decent explanation of how to use it with news organizations. Zvi is perhaps a bit cynical, but the thesis is in part that this level of cynicism is warranted.
I haven't been using Bounded Distrust as much when thinking about news organizations, but I do use it when thinking about other vectors for information. (Including people.) That's a bit odd, since the original essays (both Scott's and Zvi's) are very much about news agencies. The general lesson is something like, what ways do you expect an information source to drift from ground truth or mislead?
I'd like to recommend some essay on Bounded Distrust for the Best Of LessWrong collection for adding a useful term to my vocabulary and a checklist of how to use it for the news.
I've been editing the post at the Substack version (https://thezvi.substack.com/p/how-to-bounded-distrust) but I don't see the option to switch back to the usual non-HTML editor on this one, so mods please reimport and let me know what I'm missing.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
The old Soviet joke is that Pravda always lies and so it is useful, whereas The New York Times is not as useful because it sometimes tells the truth.
Although it's a joke it actually has quite surprising implications.
This seems to imply that it would have been easier to obtain trustworthy knowledge from Pravda then the NYT?
Scott Alexander points out that the media, from The New York Times to Infowars, very rarely lies explicitly and directly.
Alas, the media often misleads. It implies and insinuates that which is not. It abuses the language. It selectively omits. It is highly motivated by partisanship and ideology and its own interests. It does not do or understand the research. It is terrible at interpreting science. It confuses cause and effect. It purports to use technically accurate data to show, even prove, conclusions known to be false, in ways that are designed to mislead and obviously in bad faith.
Nor does it much care.
They cite someone else, and claim this excuses them from all responsibility. All they said was ‘police say this guy is guilty’ or ‘this ‘expert’ found irregularities he says shows fraud.’
The rules are not what they used to be.
Then there are the op-ed pages and headlines, which are far worse.
This leads to a situation of Bounded Distrust, which I analyze at length here. I then work through some examples here. If you want to think about the problem in detail, start at these links.
A shorter, more practical version was needed.
This attempts to offer that. It leaves a lot out. Consider reading the long version.
Some special rules about the headline. They also apply to op-eds. Headlines are:
Not chosen by the author.
Allowed to lie.
Allowed to blatantly contradict the article’s content.
Laying out a Narrative the source wants you to believe.
The body of a news article is more reliable. The rules are simple. The article:
Has a Narrative, likely revealed by the headline.
Is not allowed to lie, in a way that could count as being physically falsified.
Is not allowed to assert facts without reliable sources.
Is allowed to do almost anything else.
Is often part of an implicit conspiracy to suppress true information or spread false information, without explicit denial of the true info or explicit claims of the false info.
Is allowed to repeat any claim if it attributes that claim to its source.
Can call anyone an expert. Expert consensus means three people. ‘Some investors’ and similar phrases mean two (as does ‘surrounded by.’)
Is allowed to withhold or not seek relevant information, selectively quote, frame, insinuate, imply, condemn via association, misconstrue. This includes calling true things lies or ‘misinformation’ if they imply disliked things.
Is allowed to change meanings of words in different contexts or over time.
Will draw conclusions in ways that defy logic, or that would be obvious errors to anyone with ordinary skill in the art. This is allowed.
Can and will find an ‘expert’ to support anything they want.
Will maximally shape the story to fit the Narrative and reality tunnel.
Will face potential negative reputational and other consequences for breaking these rules, and sometimes choose to break all of them.
Won’t face consequences for breaking these rules if everyone else is also breaking them to the same degree in similar spots.
Will face other negative consequences for insufficient Narrative support.
When the expected consequences of rule breaking exceed any plausible benefits from breaking the rules, you can mostly trust that the rules above are followed.
When the stakes are so high that the consequences could be seen as worth paying for either the reporter or the outlet, they might do that, which can be called using the one time. You must be extra careful.
The reporter is allowed to lie in order to get the story, the way a cop can lie during their investigation. Both often do so.
As described in the long version, now that you roughly know the rules the media uses, there are three general approaches.
Careful reading of media in combination with other sources.
Stop caring so much about the news unless it impacts you physically.
Outsource the work to some combination of other sources, including here.
The remainder of the post is a guide to using the first strategy, when it is needed.
There is no getting around the need to consider and examine the original source.
For each source at all levels, and each class of source, one must track what rules they can be assumed to be following.
Any superficially credible source screwing up and endorsing a statement can start an information cascade no one will feel responsible for. See the origins of the false ‘more athletes died in the last year than in the last 38 years’ claim. This type of logic-washing does not only applies to one side.
As long as the direct source is named, the original (primary) ‘source’ could be mistaken, lying, non-credible. Circular citations are a thing. E.g. Wikipedia cites X, Y cites Wiki, Z cites Y, Wiki cites Z instead of X, everyone forgets who X was, and Y and Z are media. Or the original source is selectively quoted. Within the rules.
As long as they quote their source, nothing more is required. There is zero obligation for media to verify their source is not spouting obvious nonsense.
If the source is a politician, assume they lie, about everything, all the time.
Some sources, especially governments and corporations, have different rules, and in some contexts engage in bounded lying where they shift expectations a fixed amount in a positive direction. This is where 'good harvest’ means ‘we will all starve’ and ‘glorious harvest’ means ‘good harvest.’ Watch out for using the one time.
Also examine the direct source. The media outlet or reporter will, on rare occasions, choose to ‘say that which is not’ and take the consequences. Ask: Do they see the stakes as high enough to consider this? If yes for the reporter but not the outlet, would the editors and fact checkers catch it?
The old Soviet joke is that Pravda always lies and so it is useful, whereas The New York Times is not as useful because it sometimes tells the truth.
Once you realize articles are sculpted to be maximally supportive of Narrative, it becomes possible to read them as a Soviet would Pravda. Every word is present for a reason.
A stronger version, with less qualifications or weasel words, would have been against the rules. This tells you where you are at.
Every piece of evidence that was found and helps the Narrative will be present. It will be taken out of context, sculpted, engineered to do this to the extent possible.
Every known detail that is not present would not support the Narrative, or at least would not support it sufficiently to justify the space necessary to include it.
Odd word choices (given the house style) are not coincidences. The standard word choice could not be used. The lack of direct statements can be very strong evidence. Pay attention to Exact Words, the use of weasel words, the Law of No Evidence and Suspiciously Specific Denials.
‘Legal reasons’ can also explain such choices.
Drawing vague flimsy associations between the target and Bad People tells you that this was the best they could do.
The choice to write the article or say anything at all is also a choice. Ask why.
This is similar to when a lawyer cannot tell you to do something that carries any legal risk, or someone is avoiding providing medical care or advice. Think about what they are being careful not to say.
This is a very different Bayesian calculus. Notice what is missing or unsaid and what qualifiers could not be left out. Ask why is this being told to me rather than something else.
Where this leads to a ‘that’s funny’ investigate further.
When you see the kinds of attacks and tricks the rules favor, that’s all they have.
For non-media sources, one must figure out what rules set applies and act accordingly.
Not that simple. The problem gets easier with practice, but is anti-inductive.
For further reading: On Bounded Distrust, An Exercise in Bounded Distrust.