I often encounter a seemingly implicit belief that listening to someone's opinion or idea is something that can be done in exchange for merely the time spent doing so. This idea seemed so obviously true when I glanced upon it in my mind, I was almost certain it was wrong.
I - We don't learn basic things rationally
We sometimes model ourselves as rational actors, disseminating information to form beliefs and ideas. Under this model, we might think of ourselves as having some priors about facets of reality, listening to someone and evaluating the logical consistency of their ideas, then contextualizing those ideas based on our priors, looking into the supporting evidence (if any), then finally updating our priors and creating new ones.
Take as an example:
A: We should introduce a carbon tax in order to reduce our negative impact on the environment.
B: Hmh [I do have a fairly strong prior that carbon-containing gas releasing industry affects our environment for the worst. I also have a fairly strong prior that taxing a behavior disincentivizes it. But I also have a fairly strong prior that taxing industry is a very complex process, thus we need fairly solid data to have a high certainty about a given taxation strategy]. What you're saying seems reasonable, has any country experimented with such a tax?
A: Oh yes, dozens have have, including large countries with reliable data collection, such as Sweeden, Argentina, and Canada.
B: That's curious. I'll look into the data and see if and how said the carbon tax was correlated with air quality, the replacement of carbon-polluting industries with more environmentally friendly forms of industry, and government investments into air-cleaning projects. [proceeds to investigate more than alter his beliefs based on the evidence]
As far as my limited understanding of human psychology goes, I find no reason to believe the above is anywhere close to our default modus operandi.
The most closely related and scientifically measurable evidence here are a multitude of phenomena that fall under the umbrella term priming.
Priming our brains for specific patterns of thought seldom implicates what we'd call "reason" or even conscious thinking. Bring someone to a part where everyone speaks Spanish and they'll be primed to understand Spanish; they will still "understand" other languages, of course, but the context switching required to communicate in them will require a bit of time and effort.
I think the Wikipedia article is a good entry point, but if you want more specific examples of irrefutable priming effects, I think the strongest you'd find are those under the umbrella of repetition priming. Areas critical to our everyday life where repetition priming is an important driver of cognition are reading and face recognition. But you're likely to find repetition priming plays a useful role in movements, navigation, and memory. It constitutes the basis of large swaths of neuroscience.
Even going beyond priming, I think there's at least contentious evidence that once we hear a lie, even if it's debunked later, even if the teller then states it's a lie, even if it's prefaced as a lie... we are still quite likely to remember it as a real fact about the world.
The experiments here are a bit shabby, and I'd point you to an amazing SSC article that digs into the issue further (read up to chapter III for the relevant bits).
However, I will remind you that the researchers mentioned above are trying to look at whether or not the truth is more powerful than fiction when it's known as such. The only takeaway I want is that we are far from perfect at distinguishing between past information as true or false even if we "know" it's truth value with 100% certainty.
Moving into the realm of hopefully-shared experiences, think about the process of language learning and schooling children.
It's quite obvious that "reason" is not very much implicated in language acquisition. Yes, it might serve to iron out grammatical details or to acquire the ability to form complex syllogisms in a language, but conscious reasoning is unrelated to the basics of learning to speak and understand. I can't tell you "how" I learned English the same way I can tell you "how" to find the area of a triangle, or "how" the Harvard computer architecture works. Even if I can come up with post-factum rules based on how I and others speak and write, I'm still not getting at the deepest part of language learning, the association of sounds with complex internal concepts.
Take the words:
Seeing these 3 constructs on the page will trigger search processes which, might return associate words, events, images, feelings... etc. We have close to zero conscious control as to whether or not these thought processes begin upon seeing the words, or as to what they end up returning, what the words imprint into our thinking.
Finally, think back to the things that you were told in school. Not things you learned, not things you reasoned about, just things that you happened to read or be told. Maybe it's the history of your country, some famous book, or some facts about the natural world. These things likely have an impact upon your life, whether you like it or not, if you happen to stumble upon questions or concepts that relate to them for which you have no better reference.
If I am told in class about WW2 and the Natzis, or about Stalin and his secret services, my mind will be guided to those if I think "what's a very bad person" or "what's an example of a violent conflict between countries". This will happen long before I have enough training in ethics to understand why genocides are "worst" than voluntarily letting millions of people in Africa die from disease and starvation.
If I read Romeo and Juliet, hearing or thinking about love or dedication or family bonds or suicide might harken me back to the text. Unless I happen to also read a bunch of other things related to those concepts, my mind will anchor to the closest bit of information it can remember.
If I am told about Democracy being a system of governance that provides freedom and which came about as the peak solution to running a country based on enlightenment thinking, I am likely to remember that association.
This is not to say that the examples we are told about in school are incorrect or unideal.
It might be that Rome and Juliet is a great template for romantic relationships (barring the suicide bit), it might be that Stalin and Hitler are great examples of mass-murderers, it might be that democracy is something that brings freedom and prosperity. I'm fairly convinced at least the later 2 are true
But I don't remember having "reasoned" to those conclusions. I didn't read 100 romance novels to compare Rome and Juliet against. I didn't research 100 mass-murderers to compare Stalin against. I didn't read up on 100 alternative systems of governance and their effects upon freedom or prosperity.
Some people with interest in those specific topics might have, much like I read up on, say, mathematics and computer science. But nobody has the time to become an expert in everything, so we are stuck with uncritically acquired information in a lot of areas.
II - Information environment
Hence I think our information environment might dictate a lot of what we believe. This is obvious when we are kids and have comparatively few and comparatively weak priors, as well as lack most of the "reasoning" ability we have now. However, based on my understanding of priming, language interpretation, and my own experience, I'm fairly secure in speculating our information environment might be fairly relevant to how our beliefs change in adulthood. We're probably less susceptible to our information environment as adults, but we are less susceptible to learning new things in general as we age.
In particular, I think the information environment we expose ourselves to is very prone to creating beliefs in areas where we have otherwise weak priors.
If you look at most areas where quackery seems to prosper, they are those not well covered by science. Or even better, they concern themselves with things that are so subjective or poorly defined that no two men can agree on their exact properties, thus forever shrouding themselves in the impermeable armor of vaguery.
Tea-leaf readers, numerologists, astrologers, and other such charlatans make "predictions" about amorphous concepts such as "love" and "happiness" and "prosperity". Homeopathy and crystal healing address "improving the immune system" and "increasing energy levels", not curing a very specific form of cancer or treating malaria. Religion concerns itself with concepts such as good, evil, desert, and what happens to consciousness once we die.
Note: Some readers might not get the half-joke if I include cosmology and nutritional epidemiology in the above list, so I refrained from doing so
It's hard to obtain good priors in those areas, so when in need of a belief, left empty-handed by any serious scientific endeavor, our brain might turn itself to quackery. Not for lack of reasoning skills, but for the very same reasons it might turn itself to Romeo and Juliet when thinking of love at first sight, or the same reason it subconsciously affixes certain concepts to the word "MATTRESS", because our brains tend to prefer an answer, a "something", anything, to lack of knowledge.
But chief among all examples is advertising, a multi-trillion dollar industry that concerns itself mainly with some form of efficiently infecting us with false or skewed beliefs.
Granted, some ads amount to "Here's a problem you thought unsolvable here's a solution to it", but most advertising is chiefly focused on things like:
- Brand reinforcement (e.g. almost everybody knows BMW is a car for people with self-esteem issues, but unless everybody knows that everybody knows, unless the arbitrary belief is enforced, the BMW brand losses it's meaning)
- Increase the likelihood of involuntary memory lookup (e.g. all diabetes drinks are the same, but when people think "I'd like some sugar water" they usually think "I'd like a coca-pepsi" ... any ad that is designed to be funny/interesting/shocking and has basically nothing to do with the subject matter falls into this category)
- False decision-critical information that will never be fact-checked (e.g. "Did you know the opposing political party leader was disrespectful to the veteran-doctor-babies !?", "9/10 dentists said FluroGush is a mouth water they recommend, it also kills 105.6% of all BBBBAD BACTERIA, buy FluroGush now")
Ads are the prime example of thought tumors. They embed themselves into our minds and we are left little option of stopping them once they've been seen. By design, they are made to avoid the filter of reason.
Most people seem to have some understanding that ads are "bad", but they shelf this under "ads are annoying" or "my data privacy is important" not under "this is seeding false ideas into my brains and will make my map of the world less accurate, potentially snowballing into catastrophic failures of reason".
I've mainly addressed the information environment which we ourselves construct. We have a choice in installing an adblocker, we have a choice in not consuming media where "we are the product", we have a choice in not reading the blogs of quacks, we have a choice in not listening to the random garble of everyone on the internet.
But the worst of problems comes from the information we are exposed to by our social and physical environment.
For example, Tokyo has a very art-nouveau aesthetic for me, with its huge neon billboards, every square centimeter of space covered in some bright colored poster, and ad videos blurting out from screens lodged every nook and cranny. But that's because I can't read Japanese, I assume that, if I was able to, I'd be maddening... then again, most residents seem to adopt a "headphone in + head down" posture 90% of the time, so I guess that might be a viable coping strategy (as far as I've seen the ads seem to be fighting back by encroaching on the pavement)
Or what about the people we hang out with? At least, assuming you're a person with a fairly diverse friend group.
The "problem" with friends is that one can't really ignore them as one would an ad or news report on TV. Sure, you can get away with saying "look, this is, like, a very very boring subject and I don't want to talk about it", but if half of what someone wants to address relies on something you consider to be "likely bullshit", then there's no getting away from discussing it, it's that or toning down the friendship.
One idea that some seem to rally behind is that of "just listening" to someone, the idea that sometimes people just need an empathic fleshboard to vent upon. That would be all well and good if we were purely rational actors, but we aren't. Every time someone tells us something, especially a friend, i.e. someone we trust, a small part of our brain now beliefs that. Say the words and trigger the sensory queues and arrange the internal monologue in just the right order, and even the most absurd of thoughts you ever encountered might pop out.
To give a quick example of what I'm talking about, remember the carbon tax discussion? This is how I really imagine that sort of discussion:
A: We should introduce a carbon tax in order to reduce our negative impact on the environment. [Because someone told me so in a video on youtube, and now it feels good to have a new piece of information and my brain instinctively wants to share it]
B: Hmh [I have no idea what those terms are, I heard of "global warming" and "carbon causing climate change" a few times, so I guess they have to do with that amorphous cloud of concpets]. So you're saying [retrive catchphrases] it's a way to fight global warming !?
A: Yeah, I think so [because I share that same blob of concepts where climate change == global warming == environment impact and where anything that affects one positively affects the other 2 the same way]
B: Oh, ok, I guess that's cool. [I will now proceed to do no further research on this because the topic doesn't interest me one bit, but whenever someone brings up environmental issues or climate change the concept of a carbon tax, even though I'm not sure what it implies, will now pop to mind, potentially with some nice sound around it]
III - Oh god, is this really bad?
If you're with me thus far, hopefully, we can agree on 4 things:
- We aren't the conscious arbiter of most of the information we learn.
- Whatever information our environment happens to blur at us, we might remember and might accidentally assign some degree of truth to.
- There more intensely we listen, the more likely that information is to get embedded.
- We have to draw a line somewhere, in order to continue reading fiction, hanging out with people we like, changing our mind when it has to be changed and debating opposing viewpoints.
For one, the critique above seems to be damming many forms of fiction. I recently finished reading "Unsong", which, in case you aren't familiar, is a book that has among its premises:
- Most Abrahamic religions and their holy texts are real, in a very, very, very literal way.
- All forms of quackery, from numerology to fortune-telling, to interpreting the bible, to ouija boards are not only real, but they constitute a layer of reality more fundamental than the law of physics and the constraint of mathematics.
- Knowing "secret" words and finding vague schizophrenic interpretations of words and phrases is the most powerful force in the universe.
- An anthropomorphic-....ish God exists and all the problems that come with that, from immortal souls to theodicy, to the apocalypse are real.
This book includes a surprisingly large swath of the nonsensical ideas one might encounter in the real world. And it makes them seem compelling, it makes characters based on these arts that you get attached to, it waves an amazing story around them, it's ending is so touching it made me listen to American Pie 20 or 30 times over the next few days, it brings joy and causes interesting and seemingly profound thought during and after it's reading.
Daniel Denette once compared religion to music and said that, if we found out music lead to dangerous or violent behavior, we ought to discourage and ban it. This is an interesting analogy and it served to show me a point where I break with traditional "rationalism", in that I truly believe some form of art might have close to intrinsic value (intrinsic not in a "supernatural" sort of way but in a "critique of pure reason" sort of way). I feel this way about music and I also feel this way about some fiction, be it lord of the rings or the foundation series.
Looking at information as "harmful until proven true" leaves fiction on a perilous cliff edge, marks it off as a dirty pleasure.
Perhaps worst, this viewpoint means that certain forms of discussion and debating are dangerous. I was always of the opinion that one should, as long as they are bored, try to reasonably discuss with whomever crosses their path on the internet... in a collaborative and non-aggressive sort of way, mind you. The more misinformed the person seems the more likely at least one of you stands to learn from that conversation.
But if we view information as harmful until proven otherwise, "getting to someone's level" could be dangerous, unless you have above-average certainty that "their level" is in an epistemically superior place to yours.
Another thing this perspective casts a shadow upon is acquiring new perspectives (no pun intended), for it seems likely that we're especially vulnerable to learn wrong things about subjects we've had no or little prior exposure to.
On the whole though, this "harmful before proven innocent" approach to information is most damaging because, if taken seriously, it provides a way to rationalize being stuck with outright psychotic beliefs.
On one hand, we've got:
- Being stuck with potentially delusional beliefs due to fear of engaging other perspectives.
- Viewing any discussion and debate as potentially having significant downsides upon our sanity.
- Lossing access to fiction, music, and potentially other things that give meaning to life.
- Lossing friends which hold beliefs that seem too outrageous or naive.
On the other hand, we've got:
- Acquiring beliefs you would not endorse had you had the background or time to compare them with alternatives.
- Hampering various mechanisms of your mind by creating some extra useless connections and potential for various false beliefs.
- Acting on wrong beliefs that were irrationally acquired, or at least necessitating some willpower and mindfulness to stop yourself from doing so.
Now, if you look at the downsides in the first section you will note they seem to describe a kind of cultish behavior one would associate with radical political movements and religion. I think this vague pattern matching is enough to scare me away from them.
But looking even further, accepting behavioral shifts of this sort might shift a life away from "one worth living" under many common systems of value, mine included.
Finally, depending on your starting point and the way the world progresses, "being stuck delusional beliefs" might mean "thinking marmite tastes bad" or "thinking that a hunt for bigfoot is the best way to pursue happiness and success in life". While we might believe ourselves to be in the former case, with few and mild delusional beliefs, the intensity of that belief is probably tightly correlated with how delusional we actually are.
Granted, I do believe there are practical ways to probe for delusional beliefs, but I digress, that's a topic for another article.
IV - Reprival and synthesis
To summarize what I've proposed thus far, I've outlined what I think is an often unspoken but in hindsight obvious proposition: That the information you expose yourself to, no matter what your reason "thinks" or would think about it, might sometimes (often?) embed itself into your mind and set or even affect priors in wholly unreasonable ways.
I concluded that "protecting" yourself from entering in contact with very suboptimal or outright wrong sources of information is impossible without seemingly becoming a soulless uncaring robot. Even then, unless you had really good priors and a map that covered a lot of territory, things might not work out.
But there might still be things that we can do to protect ourselves from some or even most such invasive incorrect beliefs without becoming an asshole or halting our engagement in public forms.
So allow me to walk close to the edge of the abyss I'm preaching caution of, by speculating some potential solutions.
First I think there's the obvious stuff which I'm sure everybody must be doing until I realize many seemingly brilliant people aren't:
- Use an open-source ad-free adblocker on every single browser, anonymous tab, or not. Ublock origin works on every major browser and platform, it's open-source, community-driven, and amazingly good. Also, use Firefox, on top of it not being built by an evil monopoly that wants to centralize the internet via AMP, it has anti-tracking and anti-ad features as well as many quality of life addons (e.g. youtube background play for android and youtube download)
- If you have an option between paying a few measly dollars or being bombarded with ads, and adblocker is not an option, pay up if you can afford it.
- Don't leave "background" info dumps running, at least no shitty info dumps, don't fall asleep to the news, don't autoplay trash-talking shows on youtube when bored and tired.
- As much as possible, avoid consuming any content where you are the product. If all your friends are doing it, much like with heroin, at least make sure you're doing it safely (e.g. if you need Facebook to communicate with people, there are solutions like messanger.com and the messenger lite app that are limited to the chatting and calling)
- If you're consuming media for the sake of being entertained, it's likely not fulfilling any other purpose, so make sure you're entertained and not doing it due to inertia.
- Avoid discussions that heavily relate to sex and politics, because people tend to switch to full-on tribal-signaling and tribal-propaganda (politics is war, arguments are soldiers... etc, etc). We've had hundreds of years of democracy to optimize political memes that are irrational yet catchy. While they needn't necessarily be harmful or wrong, they needn't be correct and they seem optimized for instilling hatred for political outgroups and blind loyalty for the political ingroup... so, yeah, those are ideas I'd keep very very far away.
As an author what can be done to reduce the "contamination" of your audience with unwarranted beliefs, happily enough, often coincides with what many academics and non-fiction no-bullshit writers would consider to be common courtesy.
Don't make appeals to emotions too strongly or too often. That's not to say once's writing shouldn't contain some jokes or at least underhanded jabs here and there, nor that some topics don't require summoning up some staple of love, anger, fear, or awe.
But if a piece is heavily affective this probably serves to lower the reader's guard. After all, it seems common sense that reason works best in a calm and somewhat detached state.
To that, I'd add frontloading evidence and citing negative findings. There's a very obvious line between being a complete bore and hiding your sources at the bottom of the page, under the ads, with no working links to them.
I think the rule of thumb should be to peak interest and very gently hint at but not draw any important conclusions in the abstract. Then frontload as much evidence as possible in the first section, in order to allow the reader to "disagree" with the principal pillars upon which your opinion stands early on. Ideally giving some slight encouragement for people to read the sources, which might be as simple as adding hyperlinks in the exact spot where you cite it and citing in the proper place. This way, if the reader disagrees with the evidence presented (or is very suspicious and lacks the necessary time to read it) they can stop reading and hopefully forget all about it.
Follow that with outlining as many of the edge cases and shaky premises your reader must "grant" for the point to stand. Only then venture into the chain of reasoning, the conclusion, and anything that might flow from that.
One can even encourage the reader to reason by employing micro-constructs such as "question as premise => evidence => incomplete syllogism => finalize in the conclusion"... but this is easier said than done.
Finally, I'd argue that any suggestion based on your conclusion that hasn't been experimentally proven ought to go at the very end and be clearly demarcated as speculation... much the same way this writing guide is.
On the whole, I think it serves to engage in the kinds of communities where these kinds of rules are implicitly observed. You'll notice all of these guidelines are sliding scales, so just try to aim for as high as you can manage.
As a consumer of foreign ideas, besides what I wrote above, I think it might pay dividends to do a few things:
- Learn the "establishment" position and the arguments for it before learning the "wake up sheeple" position. On the whole, I think it's safe to assume establishment positions are better than random. So if you end up dropping the subject at least you'll be left with some probabilistically good beliefs about it. Granted, in some cases, the establishment position itself might be hard to pinpoint.
- Spread horizontally by branching off on the work of or recommendations of authors you really like and have reason to believe hold expertise in those domains.
I do concede that we have an amazing ability not to believe everything we hear. But I am in doubt as to its efficiency.
You can read a lot of fiction and never spare a second thought to the rules of thaumaturgic magic or wrap-plasma drives when thinking about engineering in the real world. Presumably thanks to this ability to compartment "fictive information" away from "real information". But it's not at all obvious to me "how" I do this, I don't do any rituals nor feel an immediate shift in state of mind when I start consuming fiction.
It's obvious that this shift is not perfect. Again, referring to the mind-killer, you will note that most political shows are a mix of 3 things:
- Appeal to emotions (usually humor and anger)
- Fictional stories
- Real stories
You needn't even limit yourself to political shows, as far as I know, this is the case with most political discussion boards (e.g. reddit frontpage or subreddits of less popular political factions).
Maybe I'm wrong though, I started religiously distancing myself from anything politics in the last 2 years to the point of ignorance, but looking back in memory I can't figure out a single partisan political propaganda outlet that didn't include a mix of fiction and reality. I'd love to know if someone actually collected data on this.
These political propagandists do label fiction as such, you can't legally say "The opposition leaders climbed down into the basement of his castle and murdered our sacred cows then proceeded to murder children"... that's liable, but not if prefixed with "it's like" or suffixed with "ha ha ha". But it doesn't seem like anyone watching them shelves this under "fiction".
I've decided not to trust this ability to separate fact from fiction unless I get a significant gain of happiness or "meaning" from doing so. So, as mentioned before, something like reading Tolkein is in, falling asleep to the sounds of a soap opera is out. I think the dangerous point here is when one chooses to consume fiction instead of partaking in the sensory experience of the real world, even if said fiction is not inspiring or enjoyable. I understand that many of you might think this to be a fictional behavior, but I've personally noticed it quite a lot and I think we can catch ourselves sometimes doing it if we are mindful.
Last but not least I think there's an ability one might be able to acquire to hold things as "unknown" and be comfortable with that.
I think that this whole problem might be embedded into the broader issue that we are suckers for explaining and understanding, and prefer scraps of garbage over admitting to ourselves we can't know something.
I spoke about this more broadly in "Costs and benefits of metaphysics" and I think a whole article could be dedicated to modern philosophical skepticism and a subsequent prescription to accept some things just can't be known.
However, I should stop with the prescriptive part of this article here, it's the one that I have the