When I was in middle school, one of our teachers gave us a “don’t do drugs” talk.

Somebody asked him whether he had ever used drugs himself. He replied something along the lines of:

I’m not going to answer that question, because it’s one that I can only lose.

Either I say yes, and you can conclude that drugs aren’t so bad since I’m fine now.

Or I say no, and you can conclude that since I haven’t tried them, I don’t know what I’m talking about.

That stuck in my mind. I couldn’t fault the logic in what he said. But something about it still felt off.

Surely it can’t be that any answer to a question makes it less likely for drugs to be bad?^[1]

Presumably it’s possible for drugs to really be bad. And if we are in a world where that is true... you need to be able to conclude that, somehow. He had concluded that somehow.

There was also the question of, if any answer should update us against believing that drugs are bad, how does telling us that help? If he gives us the logic of why we’d update against him anyway, shouldn’t that cause us to update already?

Even though he never stated this explicitly, I think I heard his comment as communicating something like:

This is an adversarial game - you are teenagers who want to defy authority, so you’re looking for ways to disbelieve me. There are illegitimate maneuvers that you can pull to turn anything that I say against me, so I’ll refuse to play that game by not letting you make those maneuvers.

So his move was, in a sense, trying to poison the well against those questions. Not only was he refusing to answer the question, but he was implying that even if we did get an answer, two possible things we could say in response were invalid.

But why were they invalid?

What his comment ignored was that if he said yes or no, he could also give the reason why he’d concluded that drugs were bad regardless:

• If he hadn’t tried them, maybe he knew people who had, or maybe he just trusted the judgment of others who had studied the topic in more detail.
• If he had tried them, maybe he was able to keep it to a few tries but things had gone more badly to other people he knew, or had heard about.

So, if he acknowledged that something was an argument against drugs being bad, but he still thought they were bad, presumably there was a reason why he believed that regardless. And then he could have shared that reason, and we could have discussed it and come to understand why he felt that way.

Of course, he wasn’t trying to just share the evidence and let us come to our own conclusions - he was trying to convince us of a particular conclusion. And there were a number of reasons why he might think just sharing his reasons might not convince us:

• He might not have known why he believed this in the first place
• The explanation might have been something illegible like “this is just my general life experience that you kids haven’t had”
• The explanation might have gotten into an complex conversation about epistemology that he might not have thought deeply about, or that he expected to be too advanced for us

Or, the resulting discussion might just have taken too much time and he needed to teach us the rest of the curriculum too, without spending all the class hours on this topic. Maybe it was just ten minutes before the day would end and this was all the time he had for now - I don’t remember.

If he’d engaged in an honest and open conversation with us, we would probably have felt more respected, and some of us might have been more persuaded by that alone. But some others might have found his rationales less persuasive than if he just refused to engage.

I don’t know which strategy would have been better overall, though the class time consideration probably favors him pulling the kind of anticipatory coverup that he did.

Many essays have a strong case that they’re arguing for. Here’s why X is true. Here’s why Y is false. Here are all the arguments for why I’m definitely right.

Sometimes, a position that you are arguing for feels simply correct, and you have strong reasons to believe in that. In those cases, it makes sense to write that kind of essay.

But sometimes I’m writing an essay that’s arguing for a position that I think is probably right, even though my argument also has some weak spots. And then I’m still tempted to write them the way our teacher might have written an anti-drug essay - making a strong case, bringing up objections only to dismiss them.

I consider myself lucky that I seem to find this sufficiently stressful that I’ll avoid doing it.

Two recent essays of mine that people really liked are “Taking woo seriously but not literally” and “How I stopped being sure LLMs are just making up their internal experience”. In both, I was making a claim that I expected people to be skeptical of:

• In the woo post, I was suggesting that things like Tarot, chakras, energy healing, etc., might have real effects, even if not by the mechanisms that people often ascribe to them.
• In the LLM post, I was suggesting that when LLMs claim to feel something, there might be a functional sense in which that is true, rather than being pure confabulation.^[2]

In both cases, my initial impulse was to make a stronger argument than my evidence actually warranted. For the woo post, this would have produced a post where I said something like this:

“I’m going to explain why I think all of these things are compatible with mainstream science and why they do something valuable, while not violating physics as we understand them. Here’s the explanation for Tarot, here’s the explanation for chakras, here’s the explanation for energy healing. Also, there are good reasons to engage in these practices.”

That’s not the post that I actually ended up writing. What I ended up writing was closer to something like this:

“I think all of these practices are probably compatible with mainstream science and do something valuable, while not violating physics as we understand them. Here’s the explanation for Tarot. My understanding of what’s going on with chakras is more vague, but I think it’s maybe something like this. For energy healing, I have a rough sense of what might be going on, though it’s a bit hand-wavy and speculative. I suggest trying some of these practices if they seem interesting to you, though I’ve only gotten moderate benefits from them personally, and I’m still not sure how valuable they are to people in general.”

What triggered the switch was a feeling of discomfort with trying to write the stronger version. When I was trying to make the stronger claims, some fears came up:

• “I feel like I’m trying to ignore some evidence that’s stacked against me here, what if someone notices and calls me out on it?”
• “What if I later turn out to be wrong about this and then it’ll feel embarrassing to have made a confident-sounding case for something inaccurate?”

That’s when I remembered that I didn’t need to say anything beyond what the evidence warranted. I could just say “here’s what I think and why”. I could even say “my personal experience leads me to believe in this, though of course I don’t expect that by itself to necessarily persuade anyone else”. And I could say, “I believe that probably X, though I acknowledge that Y is a point against it, and I don’t currently know how to explain that”.

If I did that, I wouldn’t need to be afraid of anything anymore! Nobody could catch me ignoring relevant evidence if I just honestly reported everything I knew.

And if I did turn out to be wrong later, people would still be able to see that I’d made a good-faith effort to figure out what was going on, and that I hadn’t claimed to be infallible and definitely correct.

I’ve heard my post-writing style described as something like “epistemically virtuous”. But if being virtuous is defined as something like “doing the unpleasant but correct thing”, writing the post didn’t feel particularly unpleasant from the inside. It felt like a huge relief that I could let go of trying to stretch the truth, and just share my honest thoughts.

This feels like the same thing that makes honesty feel easier than lying, even separate from any other ethical considerations. If I tell a lie, I have to remember that I told a lie, and there will be a small ongoing background stress from the possibility of getting caught. But if I tell the truth, I don’t need to memorize anything to ensure that I can keep my story straight. I can just be honest and then not think about it any further. If I turn out to be wrong, I can say “well, that’s what I believed at the time” with a clean conscience.

Likewise, once I stopped trying to make an overly strong case in my post, my mind could just stop tracking the consequences of that potentially backfiring.

Something similar happened with the LLM post as the woo one. The final version of the LLM article has a paragraph near the end, which explains that if you find my overall position unclear, it’s because I’m also unsure myself and can see arguments going both ways - LLMs might have functional feelings, or they might just confabulate, and it’s not always clear what the distinction even is.

I also had an impulse to try to make a much stronger case in that post, and to make an argument for why LLMs definitely have something like functional experiences. But it seemed hard, because there were many ways to interpret the evidence. And whenever I was about to say “this is how things definitely are”, an alternative interpretation would pop up in my mind, and then I would need to find a way to argue that alternative interpretation away, and then it started feeling like a stretch…

Again, I just reminded myself that I don’t need to do that. The finished post is structured as a series of case studies that were interesting to me. Instead of having to think how each study would support my case, my writing process could be closer to something like… laying out each case study in turn, thinking about what it might or might not imply, and just putting that out there.

I think this wasn’t just better for me personally, but the end result was also genuinely better writing. By sharing my model and the evidence behind it, my readers could better integrate my claims with whatever evidence and models they have. It was hopefully obvious from the posts that I was doing my best to figure out what was actually going on, rather than arguing from the bottom line, which I expect made many readers more receptive to what I was saying.

The cool thing about this technique - if you can call this a technique - is that while it delivers humility when you need it’s warranted, it shouldn’t create excessive humility when you ought to express something with confidence! It just involves reporting your true belief. So if you are uncertain about something, you report that; if you are very certain about something and feel like all the evidence is in favor of your position, you report that.

Alas, the extent to which a person can get this thing to work probably depends on what kinds of things their environment rewards them for. Some places will reward authors for confident claims in support of their pet agendas, even if those claims later turned out to be untrue. Meanwhile, my two posts were cross-posted to LessWrong, which is eager to challenge unsupported claims and at least talks a lot about holding people to their predictive track records (though I’m not sure if it really happens that much in practice).

Probably a lot of my hesitation about making overly strong claims came from the expectation that they would be challenged, and I would have been less hesitant to make the claims in an environment where I thought I’d get away with it. (This is one of the reasons why the popular meme of “just believe in yourself and ignore what everyone else thinks” is wrong - a bit of neuroticism about being caught in a false claim is healthy, you just don’t want to overdo it to the point where you’ll make no claims at all.)

This is similar to the tradeoff our teacher was facing. He could acknowledge all the uncertainties in his position and open them up to discussion, possibly convincing some of us. Or he could attempt to close off some objections and get us to treat them as illegitimate, in the hopes that we’d accept that.

Likewise, if I write a post where I openly share all of my true reasons for believing in something - including the illegible ones, like “I have personal experience of energy phenomena that makes me think there’s something here, but for which I can’t supply any objective evidence” - then some people are going to find that unpersuasive.

But the thing is, my goal isn’t to be maximally persuasive. My goal is to communicate the reasons for my belief in a way that lets both myself and others improve their models. If I share a piece of evidence that other people have reason to be skeptical of, then that evidence might genuinely be misleading me. If I’m misinterpreting it or not taking something important into account, being honest about the evidence lets others notice and help me by pointing out what mistake I’m making.

One cue that I’m making a mistake is if I find myself looking for ways to persuade people of a thing, that have nothing to do with why I believe in it. An example of this kind of a move might be if I had heard of a theorem that someone claimed to support my beliefs, and I cited that theorem even though I didn’t understand it and it was irrelevant to my beliefs.

This one is a bit subtle, since often I do explain things by constructing an argument that I expect to be more persuasive to my audience, and which isn’t a direct cause of my beliefs. In the woo post, I made an analogy between Tarot and chess. I suggested that just as the rules of chess are in a sense arbitrary but optimized to make for interesting play, the meanings of Tarot cards are in a sense arbitrary but optimized to provide good material for introspection.

The distinction I would draw is that when I’m constructing this kind of an argument, it’s not something unconnected with my reasons to believe X, but something logically connected to them. I hadn’t thought about the chess-Tarot connection before someone said that Tarot seems arbitrary to them… but once I heard this objection raised, I noticed that Tarot is both arbitrary and optimized for a purpose, and looked for another instance of a thing that is both arbitrary and optimized to use as an example. In a sense, I was finding a way to translate my thinking into a domain I expected my audience to be more familiar with.

Meanwhile, a theorem I don’t understand can’t be an example of a principle that I’m using in my reasoning, because none of my reasoning connects to it.

I’ve noticed that the same thing also works when it comes to real-time conversations where I feel an urge to really defend a particular position. I say something, someone else challenges it, I feel a temptation to defend it… and then it’s once more a relief if I can just drop the againstness and go into a mode of “Hmm, that’s an interesting point, what does it imply to my position? Let me try to think about it”. Even if I don’t have a fear of losing the argument, it feels like stretching the truth and trying to argue things away is intrinsically stressful, and dropping that brings a feeling of lightness instead.

Also, there’s a certain mischievousness to being in a heated argument where I’m sure my interlocutor is expecting me to argue back, and definitely not to say something like “hmm if your argument is true, then that would imply X, let me go research whether X might be true, oh I found an argument that does lend some support to your point”. There’s something slightly delightful in getting to do that.

Our teacher would probably have gotten a talking-to from the principal if he had said “good point, maybe you should do drugs”, though.

1. ^{^}

   Today I’d say that that violates conservation of expected evidence, though I didn’t know the term at the time.

2. ^{^}

   This was later corroborated by Anthropic’s paper on functional emotions.