“Knowledge, like all things, is best in moderation," intoned the Will. "Knowing everything means you don't need to think, and that is very dangerous.”
― Garth Nix, Lady Friday^[1]

In the pursuit of knowledge, there are two bad attractors that people fall into. 

One of them is avoiding ever knowing anything. "Oh I could of course be wrong! Everything is only suggestive evidence, I can't really claim to know anything!"

The second is to really lean into believing in yourself. "I know this to be true! I am committed to it, for constantly second-guessing myself will lead to paralysis and a lack of decisiveness. So I will double down on my best hypotheses."

The former is a stance fearful of being shown to be wrong, and the ensuing embarrassment, so avoids sticking one's neck out. The latter is also fearful of being shown to be wrong, and so takes the tack of not thinking about ways one could be wrong.

I do not claim to solve this problem in full generality, but there is one trick that I use to avoid either of these mistakes: Believe two things. In particular, two things in tension.

What's an example of this?

Sometimes I notice how powerful human reasoning has been. We've discovered calculus, built rocket ships to the moon, invented all of modern medicine, etc. Plus, I notice all of the evidence available to me via the internet—so many great explanations of scientific knowledge, so many primary sources and documentation of events in the world, so many experts writing and talking. In such a headspace, I am tempted to believe that on any subject, with a little work, I can figure out what is true with great confidence, if I care to.

At other times, I notice that people have made terrible choices for a long time. There was so much rising crime until people figured out that lead in paint and gas caused lower IQ and increased violence. People thought obesity was primarily an issue with character rather than a medical issue. People on the internet constantly say and believe inane and false things, including prestigious people. I myself have made many many dumb and costly mistakes that I didn't need to.

I could choose to believe that I am a master of reality, a powerful rationalist that will leave no stone unturned in my pursuit of truth, and arrive at the correct conclusion.

Or I could believe we are all deeply fallible humans, cursed to make mistake after mistake while the obvious evidence was staring us in the face.

My answer is to believe both. I understand very little of what is true and I can come to understand anything. The space between these two is where I work, to move from one to the other. I shall not be shocked when I observe either of these, for they are both happening around me regularly.

The unknown is where I work. In Scott Garrabrant's sequence on Cartesian Frames, he frames knowledge and power as a dichotomy; either you can know how a part of the world is, or it can be in multiple states and you can have power over which state that part of the world ends up in. Similarly, I take two pieces of knowledge with opposing implications and associations; in holding both of these opposing beliefs, they set up a large space in the middle for me to have power over, where the best and worst possible outcomes are within my control.

A different route that people take to avoid fooling themselves, is to believe a claim (like "I can come to understand anything if I try hard enough") and then to remember that it's conditional on trying hard enough. They try to hold on to that concrete claim, and add a fuzzy uncertainty around what it means for a given situation. I find this is less effective than holding onto two concrete claims that are in tension, where the two claims imply that there are other things that you don't know.

The mistake that I think people make is to remember the one claim they do know, and act as though that's all there is to know. If "We will probably all die soon due to AI" is the only thing that you believe, it seems like you know all that is relevant, all that you need to know (the current trajectory is bad, a pause would be good, alignment research is good, etc). But when you add that "We have the potential to survive and flourish and live amongst the stars" then suddenly you realize there's a lot of other important questions you don't know the answer to, like what our final trajectory might look like and what key events will determine it.

You might be interested to know where I picked up this habit. Well I'll tell you. It started when I read Toni Kurz and the Insanity of Climbing Mountains by GeneSmith. See, until that point I had assumed that the story of my life would make sense. I would work on some important projects, form relationships with smart/wise/competent people, accomplish some worthy things, before dying of old age/the singularity would happen.

Then I read that and realized that the story of these people's lives made no sense at all.

I think that there are two natural options here. The first one was to ignore that observation, to blind myself to it, and not think about it. Of course the story of my life would make sense, why would I choose otherwise?

The other route that people take is to conclude that life is ultimately absurdist, where crazy things happen one after another with little sense to it in-retrospect.

As I say, I was only tempted by the first one, but the essay was a shock to my system and helped me stop blinding myself to what is described. Instead of blinding myself or falling into absurdism, I instead believe two things.

1. My life story has the potential to make a lot of sense and be something I look back on in pride.
2. Most lives do not have a reasonable story to them.

Now all that's left for me is to understand how and why different lives fall into these two different buckets, and then do the hard work to make the first one obtain rather than the latter.

To end with, here are some more beliefs-in-tension that I've come by in the course of my life. Please share your own in the comments!

1. ^{^}

   You might think that I put this quote here because I read "The Keys to the Kingdom" series as a child and loved it. However, that is not why; I put it here because a fellow Inkhaven-er was telling me about Garth Nix, I looked up quotes by him in the convo, and noticed this one was relevant for my post.

   However your assumption that I read the series as a child and loved it would be a justified-true-belief because, on clicking through the author's Wikipedia page, I suddenly remembered that I had read the series as a child and loved it! But I entirely forgot about that in the course of finding the quote and choosing to use it.

   Good luck with your future accurate-belief-acquiring endeavors.