I think there's some kind of relatively binary culture split on this issue.
Many people (including me and apparently you) are like "duh, of course peoples' self-models are wrong all the time, so of course we need to treat them like any other claims people make, i.e. use ordinary reasoning and evidence to determine whether they're true". (And more generally, beyond rationalist circles, many people just treat claims about peoples' self-models the same way they treat any other claims.)
Many other people feel like peoples' self-models are sacred and inviolable, and those people get SUPER triggered if person A claims that person B is wrong about person B's own internal motivations, emotions, memories, etc. You can see an anonymous example on LessWrong here, for instance.
I have had big fights about this before with people close to me, and I have a post somewhere in the drafts pile attempting to cross the communication gap from each side.
It certainly doesn't make conversation with me impossible. When someone tells me that my understanding of my beliefs and motives should be rejected, I don't take my ball and go home in a huff, muttering that they've made further conversation nearly impossible. Rather, I respond the same way I do to any other disagreement: by supporting my view with reasoning and evidence. Why would the rules be different just because the topic of discussion happens to be my own beliefs and motives?
Man, I was right there with you going "Yes! That! Exactly!" right up until "I respond [...] by supporting my view with reasoning and evidence". There is a very important step before that! In cases where I am wrong about my own beliefs or motives, it is very difficult to notice that myself; other people pointing it out is one of the very few ways to notice. So when someone suggests I am wrong about my beliefs or motives, is extra super-duper important to step back, check the evidence, and consider whether they might be right!
And of course they could turn out right or wrong, I need to use reasoning and evidence to figure that out just like I do for everything else. The rules are indeed not any different just because the topic of discussion happens to be my own internal mental states and mechanisms.
indeed, the very existence of persistent disagreements (when you'd expect honest rational agents to simply update on each other's evidence and converge on the correct probability) suggests that hidden motives are common.
Or perhaps its a case of trapped priors. Ontology differences mean you don't interpret the same piece of evidence the same way.
The correctness of my understanding of my beliefs and motives is something I can defend to skeptics in front of third parties (successfully or otherwise); it's not something that needs to be assumed as a prerequisite for even talking at all.
I think you miss how costly it is to do so, especially when your interlocutor is already dismissing your claims about yourself. To use your earlier exampole about buying cookies from your girlfriend: if you happened to not produce artifacts of your thinking when deciding where to buy cookies, the only thing you have is claims about your internal reasoning. Which is opaque to your interlocutor who just got through dismissing your claims about your internal reasoning!
In light of these observations, I think Piper's apparently well-meaning advice is more naturally understood as a negotiating threat
That's the highest-order hypothesis? Feels like taking an adversarial stance unnecessarily. It can easily be the case the conditions to continue the conversation are too costly. That's not a threat.
if you happened to not produce artifacts of your thinking when deciding where to buy cookies, the only thing you have is claims about your internal reasoning
Sure. So, there was actually a bit of unflagged dramatic irony in the (fictional, despite being told in the first person) cookie story—
Suppose "I" didn't make any spreadsheets, but when questioned, I say (as in the post) that my internal reasoning considered the "obvious objective criteria" of "price, quality, Anzac biscuit availability, and owner's prior familiarity with the salon's reading list."
It might occur to you to ask followup questions: "Wait, why specifically Australian cookies? And what does it matter if the dessert vendor has done the reading? It kind of seems like you specifically chose those criteria in order to give your girlfriend the business." If I don't have a surprisingly compelling response to these obvious questions, that's also a sort of evidence about my intent.
You've said X and done Y, these seem in conflict to me, what's the deal? = OK (but read the room)
My first impression of you carries more weight than your lifetime of self-assessment, and I feel comfortable casually asserting this = not 100% guaranteed to be wrong, but danger danger!
I am so deeply embedded in my mind-palace that I can't differentiate my model of reality from reality itself, therefore anyone who disagrees with me is so obviously wrong that they can't possibly not know this on some level and are therefore lying if they claim otherwise = very bad!
(but read the room)
Sorry, I don't understand this part. Why read the room? What information is conveyed by reading the room, specifically?
Um, to motivate the question, I'm a little worried that injunctions to "read the room" might be a bid to comply with local status hierarchies, such that people with enough power among the ingroup can get away with asking "these seem in conflict to me, what's the deal?", while other people asking the same question are adversarially construed as doing the "My first impression carries more weight than your lifetime" or "I am so deeply embedded in my mind-palace" things.
But that's just my non-confident paranoid worry; I'm definitely not claiming that's what you meant!
Kelsey Piper replies to Richard Ngo on Twitter:
I categorically reject the idea that rejecting someone's understanding of their beliefs and motives inherently makes conversation with them nearly impossible.
It certainly doesn't make conversation with me impossible. When someone tells me that my understanding of my beliefs and motives should be rejected, I don't take my ball and go home in a huff, muttering that they've made further conversation nearly impossible. Rather, I respond the same way I do to any other disagreement: by supporting my view with reasoning and evidence. Why would the rules be different just because the topic of discussion happens to be my own beliefs and motives?
Why It's Possible to Communicate with People Who Think You're Self-Deceived
One reason why someone might think it's different is for fear that accusations of self-deception are unfalsifiable. I have privileged introspective access to my beliefs and motivations. On what possible basis could someone else tell me I'm wrong? And what evidence could I possibly offer to defend the accuracy of my self-perception? If the accusation or its denial can be made just as well independently of any third-party-accessible evidence, then that inherently does make conversation nearly impossible. What would there be to talk about?
However, I reject the premise that the accusation or its denial can be made just as well independently of any third-party-accessible evidence. I may have privileged introspective access to my beliefs and motivations, but beliefs and motivations aren't epiphenomenal, either. We form beliefs about other entities' beliefs and motivations because such beliefs let us predict their behavior. An animal that goes to the stream to drink is presumed to be motivated by thirst; the same animal going to the stream despite it having dried up in the summer heat is presumed to have (falsely) believed that the water would still be there. We don't need to know what it feels like to be the animal in order to speak sensibly of such mental states.
In the same way, people don't need to know what it feels like to be me in order to speak sensibly about rejecting my own understanding of my beliefs and motives. They could claim that my stated beliefs and motives are inconsistent with my behavior (which they can see) while remaining agnostic about how it feels to be me (which they don't have access to). Such a claim probably wouldn't be based on nothing at all; indeed, the very existence of persistent disagreements (when you'd expect honest rational agents to simply update on each other's evidence and converge on the correct probability) suggests that hidden motives are common.
If I buy cookies for the salon from a bakery owned by my girlfriend in Australia (at a substantial markup for international shipping), it kind of makes sense for other salon members to wonder if my choice might have been biased by favoritism towards my beloved rather than reflecting true optimization towards the goal of the salon having the best desserts.
At the same time, my belief that I wasn't swayed by favoritism is something I can provide reasoning and evidence for: for example, I could try showing the other salon members the notes and spreadsheets I compiled while comparison shopping, which justify my belief that my girlfriend just happens to be the optimal dessert vendor for the salon according to the obvious objective criteria—price, quality, Anzac biscuit availability, and owner's prior familiarity with the salon's reading list.
I might not convince them, but at least we have something to talk about: it's not a deadlock between an unfalsifiable accusation of self-deception and an unfalsifiable denial, because my catering decision (and therefore the beliefs and motives that led to it) was tangled up with empirical realities of the world. For example, if my comparison shopping was thorough and credibly documents the many ways in which my girlfriend's bakery is just the best, that makes my story about only wanting the best cookies for the salon more credible; if I didn't really shop around much, that makes my story less credible. The correctness of my understanding of my beliefs and motives is something I can defend to skeptics in front of third parties (successfully or otherwise); it's not something that needs to be assumed as a prerequisite for even talking at all.
Why the Self-Understanding of Those Who Think It's Impossible Should Be Rejected
But if I'm right about that, it raises the question: why would someone incorrectly believe that self-deception accusations make conversation impossible? It could just be a random, unmotivated mistake: maybe someone found the unfalsifiability objection superficially compelling, didn't really think it through, and would change their minds as soon as the error was pointed out to them. If someone incorrectly says that 123 + 68 = 181 (when in fact it's 191), it's almost certainly just because they forgot to carry the one; there's no reason for someone to want to believe the wrong answer to that question.
In contrast, there is a clear reason why someone would want the wrong answer about whether self-deception accusations make communication impossible. If I feel vaguely guilty about something and am at some level afraid of being found out, then I have an incentive to shut down conversations that might lead to me being found out, by only agreeing to talk to people who already trust me and won't press me on any uncomfortable questions. (And those who know themselves to be innocent and want to distinguish themselves from those who have something to hide have an incentive to welcome the uncomfortable questions.)
Notice how Piper writes about Ngo calling liberals self-deceived: warning him that the "move" of adopting that "frame" sacrifices his ability to communicate with liberals, and that while he might find the benefit worth the cost, from her perspective, "it's a really serious loss."
But positing self-deception isn't just a "move" that one could decline to make while still transmitting the same information; the "frame" is the expression of a hypothesis that could be true or false, which Ngo believes to be true. (Piper characterizes Ngo as claiming that he "should make up beliefs [he thinks liberals] have", but Ngo says his distrust of liberals' self-identifications is based on what he sees as their failure to "reason very explicitly about how much they've capitulated to wokeness"; reasoning from behavioral evidence aspires to be more disciplined than just "mak[ing] up" things.)
Furthermore, as I've discussed above, the cost of not being able to communicate with people you're accusing of self-deception isn't a logical or physical inevitability; it's a choice imposed by the people who don't want to talk to those who don't already trust them.
In light of these observations, I think Piper's apparently well-meaning advice is more naturally understood as a negotiating threat: don't criticize us in this way, or we'll walk. Even if you privately think us self-deceived, you must not reveal that you think that, or we'll destroy the surplus of you being able to interact with us.
If Piper doesn't think of herself as making a negotiating threat, I guess that implies that I'm rejecting her self-understanding? But, you know, I don't think I'm doing it maliciously. I'm just applying my knowledge of game theory, human psychology, &c. to the words I see before me and observing, hey, my maximum a posteriori (point estimate) interpretation of this behavior is that it looks like a veiled negotiating threat intended to suppress a class of criticism; that's an interesting phenomenon to analyze in a blog post.
But if you think I'm kidding myself about that, then I'm happy to talk about it.