Background

Duncan Sabien wrote a list of proposed "basics of  rational discourse" guidelines. Zack M Davis disagrees with (his interpretation of) one of the guidelines. I think the question is interesting and don't feel that those two posts and their comments resolve it. (Spoiler: I largely agree with Duncan on this.)

So, Duncan says that we should

aim for convergence on truth, and behave as if your interlocutors are also aiming for convergence on truth

and that we should care whether an interlocutor is

present in good faith and genuinely trying to cooperate.

Zack, on the other hand, holds (as I understand him) that

• we should aim for truth and not particularly care about convergence
• in cases where we have something valuable to contribute, it should be precisely because there is divergence between what others think and our view of the truth
• a good frame for thinking about this is that debates are like prediction markets
• it shouldn't matter much whether someone else is "in good faith" or "trying to cooperate"; the merits and defects of their arguments are whatever they are, regardless of intentions, and we should be responding to those
• talk of "cooperation" and "defection" implies a Prisoner's Dilemma situation, and attempted rational discourse doesn't generally have the properties of a Prisoner's Dilemma.

I will argue

• that we should aim for truth rather than for convergence as such
• that I think Zack has misunderstood what Duncan means by aiming for convergence
• that what Duncan means by "aiming for convergence" is a reasonable goal
• that debates are importantly unlike prediction markets (and even prediction markets don't necessarily have quite the incentives Zack assumes)
• that there are specific forms of not-aiming-for-convergence that are more common than Zack's "aiming for divergence"
  • and they are usually harmful when they occur
  • and when they do, it often becomes necessary to adopt modes of discourse that attend explicitly to status, politeness, intentions, etc., which makes everything worse
  • and this failure mode is made less likely if (as Duncan proposes) we avoid those forms of not-aiming-for-convergence and assume until we get strong contrary evidence that others are also avoiding them
• that while in principle we can just respond to others' arguments and ignore their (actual or perceived) intentions, in practice we can't and shouldn't
• that informal talk of cooperation and defection doesn't have to imply a PD-like situation
• and is reasonable in this context even though (I agree with Zack) what is going on isn't altogether PD-like.

So, let's begin.

Convergence

Suppose A and B are discussing something they disagree on: A thinks X and B thinks Y. Here are some goals A might have. I've given them brief but crude names for later reference. The names do not perfectly reflect the meanings.

• WIN, by making A look clever and wise and good, and making B look stupid or ignorant or crazy or evil. (One way, but not the only one, to do this is by proving that X is right and Y is wrong.)
• LEARN, ending up believing X if X is right and Y if Y is right.
• TEACH, so that B ends up believing X if X is right and Y if Y is right.
• CONVINCE, so that B ends up believing X.
• EXPOUND, so that the audience ends up believing X.
• AVOID CONFLICT, by backing off from any point at which A and B seem likely to disagree very sharply.
• AGREE, by actually ending up believing the same thing whether right or wrong.

I think that when Duncan says we should "aim for convergence on truth" he means that A should aim to LEARN and TEACH, with the hope that if A and B pool their knowledge and understanding they will likely (1) both end up with truer views, (2) end up understanding one another better, and (3) since the truth is a single thing, end up nearer to one another. 3 here is not particularly a goal but it is a likely outcome of this sort of discussion if pursued honestly and competently, and I think "aim for convergence on truth" means "aim for the truth, with the expectation of convergence".

(Why do I think Duncan means that rather than, e.g., "explicitly aim to converge"? Because he wrote this: "If two people disagree, it's tempting for them to attempt to converge with each other, but in fact the right move is for both of them to try to see more of what's true." And, next paragraph: "If you are moving closer to truth [...] then you will inevitably eventually move closer and closer to agreement with all the other agents who are also seeking truth." I think that's a little too optimistic, incidentally.)

It is possible to aim for truth without aiming for convergence even in this sense. For instance, if A is very confident that X is right but has given up hope of convincing B, he may aim to EXPOUND and secondarily to WIN, promoting truth by convincing the audience that X (which A believes) is true and also making them less likely to be (as A sees it) taken in by B. Or A may not care what anyone else ends up thinking and aim only to LEARN, which will produce convergence if it happens that B is right and A is wrong but will likely do little to help B if A was right from the start.

So, Duncan thinks that the combination of LEARN and TEACH is a good goal. I don't think he's advocating AGREE as a goal. My sense is that he may favour a bit of AVOID CONFLICT where Zack largely doesn't. But what does Zack want?

Prediction markets and divergence

Well, Zack suggests that a debate is like a prediction market; you should only get into one if you think you have some knowledge or understanding that the people already there lack; you should therefore expect to diverge from others' views, to try to prove them wrong, because that's how you "win status and esteem"; to whatever extent debate converges on truth, it's by incentivizing people to contribute new knowledge or understanding in order to win status and esteem.

In terms of the goals I mentioned above, this amounts to supposing that everyone is trying to WIN, hopefully with constraints of honesty, which they do partly by trying to CONVINCE and EXPOUND. We may hope that everyone will LEARN but in Zack's presentation that doesn't seem to be present as a motive at all.

(Which is odd, because later on -- responding to Duncan's talk of cooperation and defection -- Zack instead considers someone who is simply aiming unilaterally at truth, trying to LEARN regardless of what happens to anyone else.)

Accordingly, Zack thinks it's nonsense to try to "aim at convergence", because you're only going to be able to WIN if you start out disagreeing and argue vigorously for your dissenting opinion. And because the only way for your contribution to be net-valuable is if the others are wrong and you therefore move them away from the previous consensus.

I dislike this whole framing, and I am not persuaded that any of it is incompatible with what (as I understand it) Duncan means by "aiming at convergence on truth".

When I engage in debate somewhere like LW I am not (I hope) primarily trying to gain status and esteem, and I am not (I hope) trying to WIN as such. I am hoping to LEARN and TEACH and sometimes EXPOUND; if I see that I disagree with Zack, then I don't much care about my "esteem and status" versus Zack's, but I do think it's possible that I understand something Zack doesn't, in which case maybe he'll learn something useful from me, or that Zack understands something I don't, in which case maybe I'll learn something useful from him, and that even if we fail to learn from one another whoever's reading may get something useful from the discussion. This is all rather unlike Zack's account of prediction-market-like debate.

It does all still require, as Zack suggests, that I think I know or understand something better than Zack. If I don't, I likely won't be engaging Zack in debate. So, "on average", I'm hoping that engaging Zack in debate will mean that Zack moves towards my position; if the prior state of things is that Zack and Yak and Xach are all agreeing with one another, when I enter the debate I am hoping to move them all away from the prior consensus.

But none of that means I'm not hoping for convergence! If Zack moves towards my position, then we have converged at least a bit. If instead Zack points out where I'm wrong and I move towards his position, then we have converged at least a bit. The only way we don't converge is if neither of us moves the other at all, or if one of us makes arguments so terrible that the other's contrary position gets more firmly entrenched, and I am emphatically not hoping for either of those outcomes.

And, even more so, none of it means that I'm not doing what I think Duncan means by "aiming for convergence on truth". I am hoping that Zack, or I, or more likely both, will become a bit less wrong by discussing this matter on which we have different opinions. If that happens then we will both be nearer the truth, and we will both be nearer one another.

Most of this applies to prediction markets too: it doesn't depend on what maybe Zack thinks is my naïve view of the motives of people engaging in debate. If I enter a prediction market expecting to make money, then (as Zack says) I am expecting to make the market move away from its previous consensus, but that doesn't mean I'm not anticipating convergence. I expect that the market will move towards me when I enter, because that's what markets do. If the market moves less towards me than I expected, or if it moves away from me again, then I should move towards the market because that's evidence that the market's opinion is stronger than I thought. It's true that in the prediction-market context I'm not particularly aiming for convergence, or at least I might not be, because maybe my main goal there is to make money, but I'm still expecting convergence, and for basically the same reasons as in a debate.

And, to reiterate, a discussion is not a prediction market, and I may prefer an outcome where everyone's beliefs exactly match the truth even though at that point I no longer have any prospect of WINning. (I might prefer that even in a prediction market, if it's on some important topic: better for us all to arrive at the correct view of how to win the war / stop the plague / deflect the asteroid than for me to keep making money while our cities are getting bombed, our hospitals are overflowing with the dead, and the asteroid hurtles closer to earth.)

I confess that I am confused by some of Zack's arguments in this area, and I wonder whether I am fundamentally missing his point. He says things like this:

I would prefer to correctly diverge from the market—to get something right that the market is getting wrong, and make lots of money in the future when my predictions come true.

but this isn't about diverging-the-opposite-of-converging at all, it's using "diverge" to mean "differ at the outset", and once Zack starts participating in the market the effect of what he does will be to reduce the divergence between his position and the market's. In other words, to produce convergence, hopefully "on truth". Converging means getting closer, not starting out already in agreement.

Disagreement and disrespect

Zack says that "disagreement is disrespect":

the very fact that you're disagreeing with someone implies that you think there's something wrong with their epistemic process

Well, yes and no.

Suppose I think I have a proof of some proposition and I hear that Terry Tao has proved its negation. I check over my proof and I can't find an error. At this point, I think it's reasonable to say that TT and I have a disagreement. But TT is a better mathematician than I am, and my likely epistemic state might be "70% I've made a mistake that I'm currently unable to see, 25% TT has made a mistake, 5% something else like misunderstanding what it is TT claims to have proved". It would not be accurate to say that I think there's something wrong with TT's epistemic process, in this situation. I'm seriously considering the possibility that there might be, but I'm also seriously considering the possibility that the error is mine.

But suppose I have checked my own proof _very_ carefully (and also checked any other work it depends on). Maybe at this point I'm 95% confident that the error is TT's. Does that mean I think that TT isn't "aiming for convergence on truth"? Heck no. It means I think he is honestly and skilfully trying to get at the truth, just as I am, and I think that in this case he has uncharacteristically made a mistake. If I'm going to discuss the matter with him, I would do well to assume he's basically honest and competent even though I think he's slipped up somewhere along the line. (And to be aware that it's still possible that the mistake is my own.)

I think this sort of thing is the rule rather than the exception. If I engage in a debate, then presumably in some sense I think I'm right and the others are wrong. But I may well think there's an excellent chance I'm wrong, and in the case where I'm right the others' problems needn't be "something wrong with their epistemic process" in any sense that requires disrespect, or that requires me not to think they're aiming for truth just as much as I am. And in that case, I can perfectly reasonably anticipate convergence and (in so far as it happens by us all getting nearer the truth) hope for it.

Similarly, Zack says:

So why is the advice "behave as if your interlocutors are also aiming for convergence on truth", rather than "seek out conversations where you don't think your interlocutors are aiming to converge on truth, because those are exactly the conversations where you have something substantive to say instead of already having converged"?

and it seems to me that he's conflating "not aiming to converge on truth" with "being wrong". You can aim to converge on truth and none the less be wrong, and I think that's the usual case.

Good faith, bad faith, how to win big

Duncan talks about cooperation and defection and the like. Zack responds that such language implies a Prisoner's Dilemma and (attempted) rational discourse is not a Prisoner's Dilemma: each participant should be trying to get things right unilaterally and can't possibly benefit from getting things deliberately wrong.

It seems like Zack has forgotten his own prediction-market metaphor here. If a debate is like a prediction market, where everyone is aiming for "status and esteem", then it can be quite PD-like.

Suppose Zack and I disagree on some difficult question about, say, theoretical physics. The cooperate move for either of us is to do the calculations, check them carefully, and report on the result whether it matches our prior position or not. The defect move is to give some handwavy argument, optimized for persuasiveness rather than correctness. Our audience knows (and knows it knows) less about theoretical physics than we do.

If we both cooperate, then we presumably end up each knowing the result of the calculation, having one another's calculation to cross-check, and agreeing with some confidence on the correct answer. That's pretty good.

If we both defect, then neither of us learns anything, our opinions remain unchanged, and our audience sees roughly equally plausible handwaving from both sides. The status quo persists. That's not great, but we'll cope.

If one cooperates and the other defects, the cooperator has done a ton of work and may have learned something useful, but the audience will be convinced by the handwaving (which they think they kinda follow) and not by the calculation (which they definitely don't). All the "status and esteem", which Zack proposed was the actual goal here, goes to the defector.

Is this really a Prisoner's Dilemma? Maaaaybe. It's not clear whether CD really gives the cooperator a worse outcome than DD, for instance. But I think it's PD-like enough for "cooperate" and "defect" to be reasonable labels. (Those labels do not require a strictly PD-type situation; one can make sense of them even when the inequalities between outcomes that define a Prisoner's Dilemma do not all hold.)

I think this is the sort of thing Duncan is gesturing towards when he talks of cooperation and defection in would-be rationalist discourse. And, informally, of course "cooperation" simply means "working together": what the parties in a discussion do if they are aiming to help one another understand and share their knowledge. Both of which are things you can do, and both of which are things you may be less inclined to do if your aim is "status and esteem" rather than "converging on truth". (And both of which, I think, are things we could always use more of.)

Zack considers the following possible exchange:

A. Hey, sorry for the weirdly blunt request, but: I get the sense that you're not treating me as a cooperative partner in this conversation. Is, uh. Us that true?
B. You don't need to apologize for being blunt! Let me be equally blunt. The sense you're getting is accurate: no, I am not treating you as a cooperative partner in this conversation. I think your arguments are bad, and I feel very motivated to explain the obvious counterarguments to you in public, partially for the education of third parties, and partially to raise my status at the expense of yours.

Zack says (and, in comments, Duncan agrees) that B's response is rude but also in good faith. I agree too. B is admitting to being uncooperative but not necessarily defecting in a PD-like sense. On the other hand, if B doesn't really care that much whether his arguments are actually better than A's but has seen a nice opportunity to raise his status at A's expense, then B is defecting as well as being uncooperative, and I think that's unfortunately common when people aren't treating one another as cooperative conversation partners; see below.

And I say (though I don't know how Duncan would feel about this) that while B's comment is in good faith, the behaviour he is owning up to is (not only rude but) less than maximally helpful, unless A really is as big a bozo as B apparently thinks. That is: if A is merely wrong and not an idiot, it's likely that A and B could have a more productive conversation if A were willing to treat B as a cooperative partner.

(My impression is that Zack is more willing than Duncan thinks he should be to decide that other people are bozos and not worth trying to engage constructively with, as opposed to dunking on them for "status and esteem".)

Not aiming for convergence, in practice

What does it usually look like in a conversation, where one party is not "aiming for convergence on truth"? I don't think it usually looks like Zack's optimistic picture, where they are just aiming unilaterally to arrive at the truth and not caring much about convergence. Much more of the time, it looks like that party engaging in status games, aiming to WIN rather than LEARN or TEACH. Picking arguments for persuasiveness more than for correctness. Cherry-picking evidence and not admitting they're doing so. Attacking the other party's character, who they associate with, what they look like, rather than their evidence and arguments. Posturing to make themselves look good to the audience. Etc., etc., etc.

I don't claim that these things are always bad. (Maybe the discussion is in a deliberately-designed adversarial system like a law court, where the hope is that by having two people skilfully arguing just for their own position the truth will become clear to disinterested third parties. Maybe you're arguing about something with urgent policy implications, and the audience is full of powerful people, and you are very very confident that you're right and the other person's wrong.) But in something like an LW discussion, I think they usually are bad.

They're bad for three reasons. One is that these behaviours are not optimized for arriving at truth. The second is that, if the other party is like most of us, they make it psychologically harder for them to keep aiming at truth rather than at WINning. And the third is that, if the other party notices that the discussion is at risk of turning into a status-fight, they may have to waste time and energy pointing out the status-fight manoeuvres and objecting to them, rather than dealing with the actual issues.

The default outcome of any disagreement between human beings is for it to degenerate into this sort of adversarial status-fighting nonsense. Our discussion-norms should try to discourage that.

Ah, says Imaginary Zack, but if someone does that sort of thing you can just ignore it and keep attending to the actual issues, rather than either switching into status-fight mode yourself or diverting the conversation to how the other guy is being unreasonable.

In principle, perhaps. With a debate between (and an audience of) spherical rationalists in a perfect vacuum, perhaps. Here in the real world, though, (1) status-fighting nonsense is often effective and (2) the impulse to respond to status-fighting nonsense with more status-fighting nonsense is very strong and it can happen before you've noticed it's happening.

I think this is the purpose of Duncan's proposed guideline 5. Don't engage in that sort of adversarial behaviour where you want to win while the other party loses; aim at truth in a way that, if you are both aiming at truth, will get you both there. And don't assume that the other party is being adversarial, unless you have to, because if you assume that then you'll almost certainly start doing the same yourself; starting out with a presumption of good faith will make actual good faith more likely.

Disrespect, again

But what if the other party is a bozo? Do we really have to keep Assuming Good Faith and pretending that they're aiming at the truth in a way that should lead to convergence?

Arguing with bozos is often a waste of time. The best course of action in this situation may be to walk away. (Your audience will likely figure out what's going on, and if they don't then maybe you've mis-evaluated who's the bozo.) But if for whatever reason you have to continue a debate with someone you think is a bozo -- by all means, go ahead, drop the assumption of good faith, accept that there's going to be no convergence because the other guy is a bozo who has nothing to teach and is unable to learn, play to the gallery, etc., etc., etc. But know that that's what you're doing and that this is not the normal course of events. (If it is the normal course of events, then I suggest that either everyone around is just too stupid for you and you should consider going elsewhere, or else you have again mis-evaluated who's the bozo.)

Norms and guidelines have exceptions. The situation where your discussion partner is invincibly ignorant or confused should be an exceptional one; if not, the problem isn't with the norms, it's with the people.