Imagine being a student of physics, and coming across a blog post proposing a list of guidelines for "physicist motors"—motor designs informed by the knowledge of physicists, unlike ordinary motors.
Even if most of the things on the list seemed like sensible advice to keep in mind when designing a motor, the framing would seem very odd. The laws of physics describe how energy can be converted into work. To the extent that any motor accomplishes anything, it happens within the laws of physics. There are theoretical ideals describing how motors need to work in principle, like the Carnot engine, but you can't actually build an ideal Carnot engine; real-world electric motors or diesel motors or jet engines all have their own idiosyncratic lore depending on the application and the materials at hand; an engineer who worked on one, might not the be best person to work on another. You might appeal to principles of physics to explain why some particular motor is inefficient or poorly-designed, but you would not speak of physicist motors as if that were a distinct category of thing—and if someone did, you might quietly begin to doubt how much they really knew about physics.
As a student of rationality, I feel the same way about guidelines for "rationalist discourse." The laws of probability and decision theory describe how information can be converted into optimization power. To the extent that any discourse accomplishes anything, it happens within the laws of rationality.
Rob Bensinger proposes "Elements of Rationalist Discourse" as a companion to Duncan Sabien's earlier "Basics of Rationalist Discourse". Most of the things on both lists are, indeed, sensible advice that one might do well to keep in mind when arguing with people, but as Bensinger notes, "Probably this new version also won't match 'the basics' as other people perceive them."
But there's a reason for that: a list of guidelines has the wrong type signature for being "the basics". The actual basics are the principles of rationality one would appeal to explain which guidelines are a good idea: principles like how evidence is the systematic correlation between possible states of your observations and possible states of reality, how you need evidence to locate the correct hypothesis in the space of possibilities, how the quality of your conclusion can only be improved by arguments that have the power to change that conclusion.
Contemplating these basics, it should be clear that there's just not going to be anything like a unique style of "rationalist discourse", any more than there is a unique "physicist motor." There are theoretical ideals describing how discourse needs to work in principle, like Bayesian reasoners with common priors exchanging probability estimates, but you can't actually build an ideal Bayesian reasoner. Rather, different discourse algorithms (the collective analogue of "cognitive algorithm") leverage the laws of rationality to convert information into optimization in somewhat different ways, depending on the application and the population of interlocutors at hand, much as electric motors and jet engines both leverage the laws of physics to convert energy into work without being identical to each other, and with each requiring their own engineering sub-specialty to design.
Or to use another classic metaphor, there's also just not going to be a unique martial art. Boxing and karate and ju-jitsu all have their own idiosyncratic lore adapted to different combat circumstances, and a master of one would easily defeat a novice of the other. One might appeal to the laws of physics and the properties of the human body to explain why some particular martial arts school was not teaching their students to fight effectively. But if some particular karate master were to brand their own lessons as the "basics" or "elements" of "martialist fighting", you might quietly begin to doubt how much actual fighting they had done: either all fighting is "martialist" fighting, or "martialist" fighting isn't actually necessary for beating someone up.
One historically important form of discourse algorithm is debate, and its close variant the adversarial court system. It works by separating interlocutors into two groups: one that searches for arguments in favor of a belief, and another that searches for arguments against the belief. Then anyone listening to the debate can consider all the arguments to help them decide whether or not to adopt the belief. (In the court variant of debate, a designated "judge" or "jury" announces a "verdict" for or against the belief, which is added to the court's shared map, where it can be referred to in subsequent debates, or "cases.")
The enduring success and legacy of the debate algorithm can be attributed to how it circumvents a critical design flaw in individual human reasoning, the tendency to "rationalize"—to preferentially search for new arguments for an already-determined conclusion.
(At least, "design flaw" is one way of looking at it—a more complete discussion would consider how individual human reasoning capabilities co-evolved with the debate algorithm—and, as I'll briefly discuss later, this "bug" for the purposes of reasoning is actually a "feature" for the purposes of deception.)
As a consequence of rationalization, once a conclusion has been reached, even prematurely, further invocations of the biased argument-search process are likely to further entrench the conclusion, even when strong counterarguments exist (in regions of argument-space neglected by the biased search). The debate algorithm solves this sticky-conclusion bug by distributing a search for arguments and counterarguments among multiple humans, ironing out falsehoods by pitting two biased search processes against each other. (For readers more familiar with artificial than human intelligence, generative adversarial networks work on a similar principle.)
For all its successes, the debate algorithm also suffers from many glaring flaws. For one example, the benefits of improved conclusions mostly accrue to third parties who haven't already entrenched on a conclusion; debate participants themselves are rarely seen changing their minds. For another, just the choice of what position to debate has a distortionary effect even on the audience; if it takes more bits to locate a hypothesis for consideration than to convincingly confirm or refute it, then most of the relevant cognition has already happened by the time people are arguing for or against it. Debate is also inefficient: for example, if the "defense" in the court variant happens to find evidence or arguments that would benefit the "prosecution", the defense has no incentive to report it to the court, and there's no guarantee that the prosecution will independently find it themselves.
Really, the whole idea is so galaxy-brained that it's amazing it works at all. There's only one reality, so correct information-processing should result in everyone agreeing on the best, most-informed belief-state. This is formalized in Aumann's famous agreement theorem, but even without studying the proofs, the result is obvious. A generalization to a more realistic setting without instantaneous communication gives the result that disagreements should be unpredictable: after Bob the Bayesian tells Carol the Coherent Reasoner his belief, Bob's expectation of the difference between his belief and Carol's new belief should be zero. (That is, Carol might still disagree, but Bob shouldn't be able to predict whether it's in the same direction as before, or whether Carol now holds a more extreme position on what adherents to the debate algorithm would call "Bob's side.")
That being the normative math, why does the human world's enduringly dominant discourse algorithm take for granted the ubiquity of, not just disagreements, but predictable disagreements? Isn't that crazy?
Yes. It is crazy. One might hope to do better by developing some sort of training or discipline that would allow discussions between practitioners of such "rational arts" to depart from the harnessed insanity of the debate algorithm with its stubbornly stable "sides", and instead mirror the side-less Bayesian ideal, the free flow of all available evidence channeling interlocutors to an unknown destination.
Back in late 'aughts, an attempt to articulate what such a discipline might look like was published on a blog called Overcoming Bias. (You probably haven't heard of it.) It's been well over a decade since then. How is that going?
Eliezer Yudkowsky laments:
In the end, a lot of what people got out of all that writing I did, was not the deep object-level principles I was trying to point to—they did not really get Bayesianism as thermodynamics, say, they did not become able to see Bayesian structures any time somebody sees a thing and changes their belief. What they got instead was something much more meta and general, a vague spirit of how to reason and argue, because that was what they'd spent a lot of time being exposed to over and over and over again in lots of blog posts.
"A vague spirit of how to reason and argue" seems like an apt description of what "Basics of Rationalist Discourse" and "Elements of Rationalist Discourse" are attempting to codify—but with no explicit instruction on which guidelines arise from deep object-level principles of normative reasoning, and which from mere taste, politeness, or adaptation to local circumstances, it's unclear whether students of 2020s-era "rationalism" are poised to significantly outperform the traditional debate algorithm—and it seems alarmingly possible to do worse, if the collaborative aspects of modern "rationalist" discourse allow participants to introduce errors that a designated adversary under the debate algorithm would have been incentivized to correct, and most "rationalist" practitioners don't have a deep theoretical understanding of why debate works as well as it does.
Looking at Bensinger's "Elements", there's a clear-enough connection between the first eight points (plus three sub-points) and the laws of normative reasoning. Truth-Seeking, Non-Deception, and Reality-Minding, trivial. Non-Violence, because violence doesn't distinguish between truth and falsehood. Localizability, in that I can affirm the validity of an argument that A would imply B, while simultaneously denying A. Alternative-Minding, because decisionmaking under uncertainty requires living in many possible worlds. And so on. (Lawful justifications for the elements of Reducibility and Purpose-Minding left as an exercise to the reader.)
But then we get this:
- Goodwill. Reward others' good epistemic conduct (e.g., updating) more than most people naturally do. Err on the side of carrots over sticks, forgiveness over punishment, and civility over incivility, unless someone has explicitly set aside a weirder or more rough-and-tumble space.
I can believe that these are good ideas for having a pleasant conversation. But separately from whether "Err on the side of forgiveness over punishment" is a good idea, it's hard to see how it belongs on the same list as things like "Try not to 'win' arguments using [...] tools that work similarly well whether you're right or wrong" and "[A]sk yourself what Bayesian evidence you have that you're not in those alternative worlds".
The difference is this. If your discourse algorithm lets people "win" arguments with tools that work equally well whether they're right or wrong, then your discourse gets the wrong answer (unless, by coincidence, the people who are best at winning are also the best at getting the right answer). If the interlocutors in your discourse don't ask themselves what Bayesian evidence they have that they're not in alternative worlds, then your discourse gets the wrong answer (if you happen to live in an alternative world).
If your discourse algorithm errs on the side of sticks over carrots (perhaps, emphasizing punishing others' bad epistemic conduct more than most people naturally do), then ... what? How, specifically, are rough-and-tumble spaces less "rational", more prone to getting the wrong answer, such that a list of "Elements of Rationalist Discourse" has the authority to designate them as non-default?
I'm not saying that goodwill is bad, particularly. I totally believe that goodwill is a necessary part of many discourse algorithms that produce maps that reflect the territory, much like how kicking is a necessary part of many martial arts (but not boxing). It just seems like a bizarre thing to put in a list of guidelines for "rationalist discourse".
It's as if guidelines for designing "physicist motors" had a point saying, "Use more pistons than most engineers naturally do." It's not that pistons are bad, particularly. Lots of engine designs use pistons! It's just, the pistons are there specifically to convert force from expanding gas into rotational motion. I'm pretty pessimistic about the value of attempts to teach junior engineers to mimic the surface features of successful engines without teaching them how engines work, even if the former seems easier.
The example given for "[r]eward[ing] others' good epistemic conduct" is "updating". If your list of "Elements of Rationalist Discourse" is just trying to apply a toolbox of directional nudges to improve the median political discussion on social media (where everyone is yelling and no one is thinking), then sure, directionally nudging people to directionally nudge people to look like they're updating probably is a directional improvement. It still seems awfully unambitious, compared to trying to teach the criteria by which we can tell it's an improvement. In some contexts (in-person interactions with someone I like or respect), I think I have the opposite problem, of being disposed to agree with the person I'm currently talking to, in a way that shortcuts the slow work of grappling with their arguments and doesn't stick after I'm not talking to them anymore; I look as if I'm "updating", but I haven't actually learned. Someone who thought "rationalist discourse" entailed "[r]eward[ing] others' good epistemic conduct (e.g., updating) more than most people naturally do" and sought to act on me accordingly would be making that problem worse.
A footnote on the "Goodwill" element elaborates:
Note that this doesn't require assuming everyone you talk to is honest or has good intentions.
It does have some overlap with the rule of thumb "as a very strong but defeasible default, carry on object-level discourse as if you were role-playing being on the same side as the people who disagree with you".
But this seems to contradict the element of Non-Deception. If you're not actually on the same side as the people who disagree with you, why would you (as a very strong but defeasible default) role-play otherwise?
Other intellectual communities have a name for the behavior of role-playing being on the same side as people you disagree with: they call it "concern trolling", and they think it's a bad thing. Why is that? Are they just less rational than "us", the "rationalists"?
Here's what I think is going on. There's another aspect to the historical dominance of the debate algorithm. The tendency to rationalize new arguments for a fixed conclusion is only a bug if one's goal is to improve the conclusion. If the fixed conclusion was adopted for other reasons—notably, because one would benefit from other people believing it—then generating new arguments might help persuade those others. If persuading others is the real goal, then rationalization is not irrational; it's just dishonest. (And if one's concept of "honesty" is limited to not consciously making false statements, it might not even be dishonest.) Society benefits from using the debate algorithm to improve shared maps, but most individual debaters are mostly focused on getting their preferred beliefs onto the shared map.
That's why people don't like concern trolls. If my faction is trying to get Society to adopt beliefs that benefit our faction onto the shared map, someone who comes to us role-playing being on our side, but who is actually trying to stop us from adding our beliefs to the shared map just because they think our beliefs don't reflect the territory, isn't a friend; they're a double agent, an enemy pretending to be a friend, which is worse than the honest enemy we expect to face before the judge in the debate hall.
This vision of factions warring to make Society's shared map benefit themselves is pretty bleak. It's tempting to think the whole mess could be fixed by starting a new faction—the "rationalists"—that is solely dedicated to making Society's shared map reflect the territory: a culture of clear thinking, clear communication, and collaborative truth-seeking.
I don't think it's that simple. You do have interests, and if you can fool yourself into thinking that you don't, your competitors are unlikely to fall for it. Even if your claim to only want Society's shared map to reflect the territory were true—which it isn't—anyone could just say that.
I don't immediately have solutions on hand. Just an intuition that, if there is any way of fixing this mess, it's going to involve clarifying conflicts rather than obfuscating them—looking for Pareto improvements, rather than pretending that everyone has the same utility function. That if something called "rationalism" is to have any value whatsoever, it's as the field of study that can do things like explain why it makes sense that people don't like concern trolling. Not as its own faction with its own weird internal social norms that call for concern trolling as a very strong but defeasible default.
But don't take my word for it.
"Physicist motors" makes little sense because that position won out so completely that the alternative is not readily available when we think about "motor design". But this was not always so! For a long time, wind mills and water wheels were based on intuition.
But in fact one can apply math and physics and take a "physicist motors" approach to motor design, which we see appearing in the 18th and 19th centuries. We see huge improvements in the efficiency of things like water wheels, the invention of gas thermodynamics, steam engines, and so on, playing a major role in the industrial revolution.
The difference is that motor performance is an easy target to measure and understand, and very closely related to what we actually care about (low Goodhart susceptibility). There are a bunch of parameters -- cost, efficiency, energy source, size, and so on -- but the number of parameters is fairly tractable. So it was very easy for the "physicist motor designers" to produce better motors, convince their customers the motors were better, and win out in the marketplace. (And no need for them to convince anyone who had contrary financial incentives.)
But "discourse" is a much more complex target, ... (read more)
As someone working on designing better electric motors, I can tell you that "What exactly is this metric I'm trying to optimize for?" is a huge part of the job. I can get 30% more torque by increasing magnet strength, but it increases copper loss by 50%. Is that more better? I can drastically reduce vibration by skewing the stator but it will cost me a couple percent torque. Is that better or worse? There are a ton of things to trade between, and even if your end application is fairly well specified it's generally not specified well enough to remove all significant ambiguity in which choices are better.
It's true that there are some motor designs that are just better at everything (or everything one might "reasonably" care about), but that's true for discourse as well. For example, if you are literally just shrieking at each other, whatever you're trying to accomplish you can almost certainly accomplish it better by using words -- even if you're still going to scream those words.
The general rule is that if you suck relative to the any nebulosity in where on the pareto frontier you want to be, then there are "objective" gains to be made. In motor, simultaneous improvements in efficie... (read more)
I think a disanalogy here is that all motors do in fact follow the laws of physics (and convert electricity into rotation, otherwise we wouldn’t call it a motor). Whereas not all discourse systematically leads people towards true beliefs. So rationalist discourse is a strict subset of discourse in a way that physicist motors is not a strict subset of motors.
In general, I agree that we should be open to the possibility that there exist types of discourse that systematically lead people towards true beliefs, but that look very different from “rationalist discourse” as described by Duncan & Rob. That said, I think I’m less impressed by the truth-finding properties of debates / trials than you are. Like, in both legal trials and high-school debate, the pop-culture stereotype is that the side with a better lawyer / debater will win, not the side that is “correct”. But I don’t really know.
I also agree that it’s worth distinguishing “things that seem empirically to lead to truth-finding for normal people in practice” versus “indisputable timeless laws of truth-finding”.
I was reading “Reward others' good epistemic conduct (e.g., updating) more than most people naturally do.” as like “I... (read more)
It looks like this post is resting on defining "rationalist" as "one who studies the laws of rationality", as opposed "someone training to think rationally", but, hasn't really acknowledged that it's using this definition (when I think Duncan and Robby's posts seem pointed more at the latter definition)
(Actually, looking more, I think this post sort of equivocates between the two, without noting that it's done so).
I'm not 100% sure I've parsed this right, but, this looks at first glance like the sort of language trick that you (Zack) are often (rightfully) annoyed at.
(I think it's a reasonable conversational move to point out someone else's definition of a word isn't the only valid definition, and pointing out their frame isn't the only valid frame. But if you're doing that it's better to do that explicitly)
Seems great to me! I share your intuition that Goodwill seems a bit odd to include. I think it's right to push back on proposed norms like these and talk about how justified they are, and I hope my list can be the start of a conversation like that rather than the end.
I do have an intuition that Goodwill, or something similar to Goodwill, plays an important role in the vast majority of human discourse that reliably produces truth. But I'm not sure why; if I knew very crisply what was going on here, maybe I could reduce it to other rules that are simpler and more universal.
In your view, is there an important difference between frame control, and the author having a particular frame that they use in a particular essay?
I'm proud of this blog post. I think it's a good blog post that clearly explains my ideas in a way that's engaging to read. If someone wants to talk about my motivations for writing this post and why I chose the analogies I did, I'm happy to have that discussion in the comment section, like we're doing now.
But it seems to me that a blog post that talked about my objections to Bensinger's Goodwill element, without first explaining the "motors" and "martial arts" analogies as illustrations of how I'm thinking about the topic, would be worse than this post, primarily because it would be less faithful to how I'm thinking about the topic, but also because it would just be less interesting to read.
If someone thinks my choice of analogies (or "frames"; I'm not sure if there's a specific definition of "frame" I'm supposed to be familar with in this context) is misleading for some specific reason, they're welcome to argue that in the comment section. So far, you have not persuaded me that I should have made any different writing choices.... (read more)
Inflation of "rationality" needs more specific anchors to combat it. As it stands, any purpose that looks good for someone (especially if it's actually quite good) stands a risk of getting enshrined into a "principle of rationality", such that following that principle advances the purpose, while dismissing the principle starts sounding "irrational", a norm violation if there is one in a garden of rationality, worth discouraging.
I think Scott's asymmetric weapons framing gestures at the concept/problem more robustly, while Eliezer's cognitive algorithms framing gives practical course-correcting advice:
At the moment, LW has accumulated enough anti-epistemology directed at passing good and sensible things for rationality that a post like this gets rejected on the level of general impression. I think a post focused on explaining the problem with unreflectively rejecting posts like this, or on stratifying meaningful senses of "rationality" as distinct from all things good and true, without simultaneously ... (read more)
In my mind the Goodwill norm has a straightforward justification: Absent goodwill, most people are prone to view disagreement as some sort of personal hostility, similar to an insult. This encourages us to view their arguments as soldiers, rather than as exchange of evidence. Which leads to a mind-killing effect, i.e. it makes us irrational.
To be sure, I think that some groups of people, particularly those on the autism spectrum, do not have a lot of this "hostility bias". So the Goodwill norm is not very applicable on platforms where many of those people are. Goodwill is likely a lot more important on Twitter than on Hacker News or Less Wrong.
In general, norms which counter the effect of common biases seem to be no less about rationality than norms which have to do more directly with probability or decision theory.
Certainly not. Recalibrating one's intuitions to better reflect reality is an admirable aim, and one in which we should all be engaged. However, as far as norms of discourse go, there is more to the matter than that: different people will unavoidably have differences of intuition regarding their interlocutor's goodwill, with certain individuals quicker to draw the line than others. How best to participate in (object-level) discourse in spite of these differences of (meta-level) opinion, without having to arbitrate that meta-level disagreement from scratch each time, is its own, separate question.
One of the consequences of being the type of agent that errs at all, is that estimating the precise magnitude of your error, and hence the precise size of corrective factor to apply, is unlikely to be possible.
This does not, however, mean that we are left in the dark, with no recourse but to ... (read more)
Ideally, I would arrive at my workplace exactly when my shift starts (zero error, zero loss). But if I'm ten minutes late, I get in trouble with my boss (small error, large loss), and if I'm ten minutes early, I read a magazine in the breakroom (small error, small loss). Therefore, I should "err on the side of" leaving early.
That is, the "err on the side of" idiom arises from the conflation of different but related optimization problems. The correct solution to the worker's full problem (taking into account the asymmetrical costs of arriving early or late) is an incorrect solution to the "being (exactly) on time" problem.
Yes. If my comments are too mean, I might start an unpleasant and unproductive flame war (small error, large loss). If my comments are too nice, they might be slightly less clear than a less nice comment, but nothing dramatically bad like a flame war happens (small error, small loss). Therefore I (arguably) should "err on the side of carrots over sticks."
If "Elements of Rationalist Discourse"'s Goodwill item had explicitly laid out the logic of asymmetric costs rather than taking "err on the side of" as a primitive, I'd still be skeptical, but this post's discussion of it wouldn't be written the same way (and it's possible that I might not have bothered to write the post at all).
Of course this is true, but that doesn’t actually mean that there isn’t, in fact, a cost differential; it only means that claims of such constitute weaker evidence in favor than they would in the absence of such an incentive.
And there are good reasons to believe that the cost differential exists. We may presumably discount (alleged) evidence from introspection, as it’s unreliable for two reasons (unreliability of introspection in the presence of incentives for self-deception; unreliability of reports of introspection, in the presence of incentives for deception). But that’s not all we’ve got. For example, in the linked comment, you write:... (read more)
I'll go on record as a counterexample here; I very much want politeness norms to be enforced here, and in my personal life I will pay great costs in order to preserve or create my freedom to be blunt. The requirement for me to be cautious of how I say things here is such a significant cost that I post here far less than I otherwise would. The cost is seriously non-insignificant.
The reason I don't bitch about it is that I recognize that it's necessary. Changing norms to allow people to be relatively more inconsiderate wouldn't actually make things better. It's not just that "pandering to idiots" calls for a euphemism, it's that it probably calls for a mindset that is not so dismissive to people if they're going to be in or close enough to your audience to be offended. Like, actually taking them into consideration and figuring out how to bridge that gap. It's costly. It's also necessary, and often pays off.
I would like to be able to say "Zack, you stupid twat" without having to worry about getting attacked for doing so, but until I've proven to you that I respect you enough that it's to be taken as an affectionate insult between friends.... phrasing things that way wouldn't actually ... (read more)
(Worth noting that I used to spend a great deal of effort and energy on putting up with the headache of wading through Zack's style, for the occasional worth-it nugget of insight; the moment when that constant expenditure of effort became clearly not worth it anymore was when Zack started off a(n also-otherwise-flawed) critique by just asserting "This is insane."
Even if it had, in fact, been insane, Zack would've been more effective if he'd been willing to bother with even the tiniest of softenings (e.g. "this sounds insane to me," which, in addition to being socially smoother is also literally more true, as a reflection of the actual state of affairs).
As it was, though, he was just so loudly + overconfidently + rudely wrong that it was enough to kill my last remaining willingness-to-tolerate his consistent lack-of-epistemic-hygiene-masquerading-as-a-preference-for-directness.)
Would it help if I apologized? I do, actually, regret that comment. (As you correctly point out here, it wasn't effective; it didn't achieve my goals at all.)
The reason I was reluctant to apologize earlier is because I want to be clear that the honest apology that I can offer has to be relatively narrowly-scoped: I can sincerely apologize specifically for that particular pointlessly rude blog comment, and I can sincerely make an effort to conform to your preferred norms when I'm writing a reply to you specifically (because I know that you specifically don't like the punchy-attacky style I often use), but I'm not thereby agreeing to change my commenting behavior when talking to people who aren't you, and I'm not thereby agreeing that your concept of epistemic hygiene is the correct one.
I'm worried that a narrowly-scoped apology will be perceived as insufficient, but I think being explicit about scope is important, because fake apologies don't help anyone: I only want to say "I'm sorry; I won't do it again" about the specific things that I'm actually sorry for and actually won't do again.
So—if it helps—I hereby apologize for my comment of 4 December 2021 on an earlier draft of "Basic... (read more)
Softening like this is one of those annoying things i wish we could do away with because it's smurf naming. Saying that something is insane is literally a claim that I think it's insane, and it's only because of naive epistemology that we think some other meaning is possible.
I only started adding softening because Duncan wouldn't shut up about the lack of smurfs in my comments.
I think I should just add my own data point here, which is that Zack and I have been on polar opposites sites of a pretty emotional debate before, and I had zero complaints about their conduct. In fact ever since then, I think I'm more likely to click on a post if I see that Zack wrote it.
I don't see us all as in clear agreement; I think we're at least somewhat in nominal agreement but I have found Zack to be ... I don't mean this as a contentless insult, I mean it as a literal attempt-to-model ... irrationally fixated on being anti-polite, and desperately fending off attempts to validate or encode any kind of standard or minimum bar of politeness.
By "irrationally" I mean that he seems to me to do so by irresistible reflex, with substantial compulsion/motive force, even when the resulting outcome is unambiguously contra his explicitly stated goals or principles.
To put things in Zack's terminology, you could say that he's (apparently) got some kind of self-reinforcing algorithmic intent to be abrasive and off-putting and over-emphatic. Even where more reserved language would be genuinely truer, less misleading to the audience, and more in line with clear and precise word usage (all goals which Zack ostensibly ranks pretty high in the priority list), there's (apparently) some kind of deep psychological pressure that reliably steers him in the other direction, and makes him vehemently object to putting forth the (often pretty minimal) effort required.
Similarly, even wh... (read more)
FYI, having recently stated "man I think Duncan and Zack should be seeing themselves more as allies", I do want to note I agree pretty strongly with this characterization. I think Zack probably also agrees with the above during his more self-aware moments, but often not in the middle of a realtime discussion.
I do think Zack should see this fact about himself as a fairly major flaw according to his own standards, although it's not obvious to me that the correct priority for him should be "fixing the surface-visible-part of the flaw", and I don't know what would actually be helpful.
My reasoning for still thinking it's sad for Zack/Duncan to not see each other more as allies routes primarily through what I think 'allyship' should mean, given the practicalities of the resources available in the world. I think the people who are capable of advancing the art of rationality are weird and spiky and often come with weird baggage, and... man, sorry those are the only people around, it's a very short list, if you wanna advance the art of rationality you need to figure out some way of dealing with that (When I reflect a bit, I don't actually think Duncan should necessarily be doing... (read more)
Thanks for your thoughts. (Strong-upvoted.)
Yes, that December 2021 incident was over the line. I'm sorry. In retrospect, I wish I hadn't done that—but if I had taken a few more moments to think, I would have been able to see it without retrospect. That was really stupid of me, and it made things worse for both of us.
You're also correct to notice that the bad behavior that I don't endorse on reflection can be seen as a more extreme version of milder behavior that I do endorse on reflection. (Thus the idiom "over the line", suggesting that things that don't go over the line are OK.) I wish I had been smart enough to only do the mild version, and never overshoot into the extreme version.
Are you referring to the paragraph that begins, "If two people disagree, it's tempting for them to attempt to converge with each other [...]"? In a comment to Phil H., I explained why that paragraph didn't satisfy me. (Although, as I acknowledged to Phil, it's plausible that I should have quoted and acknowledged that parag... (read more)
It was. That's why I was (and remain) so furious with you (Edit: and also am by default highly mistrustful of your summaries of others' positions).
I think it's important to note survivorship bias here; I think there are other people who used to post on LessWrong and do not anymore, and perhaps this was because of changes in norms like this one. It also seems somewhat likely to me that Said and Zack think that there's too little legitimate criticism on LW. (I often see critical points by Zack or Said that I haven't yet seen made by others and which I agree with; are they just faster or are they counterfactual? I would guess the latter, at least some of the time.)
As well, Zack's worry is that even if the guideline is written by people who have a sense that criticism should be between 4 and 12, establishing the rule with user-chosen values (like, for example, LW has done for a lot of post moderation) will mean there's nothing stopping someone from deciding that c... (read more)
It's not that I think refusing to employ the bare minimum of social grace is a virtue. It's that I wasn't aware—in fact, am still not aware—that confirming interpretations with the original author before publishing a critical essay constitutes the bare minimum of social grace. The idea that it's somehow bad behavior for intellectuals to publish essays about other intellectuals' essays without checking with the original author first is something I've never heard before; I think unilaterally publishing critical essays is a completely normal thing that intellectuals do all the time, and I see no particular reason for self-identified "rationalist" intellectuals to behave any differently.
For an arbitrary example from our local subculture, Yudkowsky once wrote "A Reply to Francois Chollet" criticizing Chollet's essay on the purported impossibility of an intelligence explosion. Did Yudkowsky first write an email to Chollet saying ... (read more)
My objection to this sort of claim is basically the same as my objection to this, from an earlier comment of yours:
And similar to my objection in a much earlier discussion (which I can’t seem to find now, apologies) about Double Crux (I think), wherein (I am summarizing from memory) you said that you have usually been able to easily explain and apply the concept when teaching it to people in person, as a CFAR instructor; to which I asked how you could distinguish between your interlocutor/student really understanding you, vs. the social pressure of the situation (the student/teacher frame, your personal charisma, etc.) causing them, perhaps, to persuade themselves that they’ve understood, when in fact they have not.
In short, the problem is this:
If “sharing interpretive labor”, “making intellectual progress”, etc., just boils down to “agree... (read more)
That's not what I meant. I affirm Vaniver's interpretation ("Zack's worry is that [...] establishing the rule with user-chosen values [...] will mean there's nothing stopping someone from deciding that criticism has to be above 8 and below 6").
(In my culture, it's important that I say "That's not what I meant" rather than "That's a strawman", because the former is agnostic about who is "at fault". In my culture, there's a much stronger duty on writers to write clearly than there is on readers to maintain uncertainty about the author's intent; if I'm unhappy that the text I wrote led someone to jump to the wrong conclusion, I more often think that I should have written better text, rather than that the reader shouldn't have jumped.)
Another attempt to explain the concern (if Vaniver's "above 8 and below 6" remark wasn't sufficient): suppose there were a dishonest author named Mallory, who never, ever admitted she was wrong, even when she was obviously wrong. How can Less Wrong protect against Mallory polluting our shared map with bad ideas?
My preferred solution (it's not perfect, but it's the best I have) is to have a culture that values unilateral criticism and many-to-many discours... (read more)
This does not seem like it should be possible for arbitrary X and Y, and so if Zack manages to pull it off in some cases, it seems likely that those cases are precisely those in which the original post's claims were somewhat fuzzy or ill-characterized—
(not necessarily through the fault of the author! perhaps the subject matter itself is simply fuzzy and hard to characterize!)
—in which case it seems that devoting more cognitive effort (and words) to the topic might be a useful sort of thing to do, in general? I don't think one needs to resort to a hypothesis of active malice or antipathy to explain this effect; I think people writing about confusing things is generally a good thing (and if that writing ends up being highly upvoted, I'm generally suspicious of explanations like "the author is really, really good at confusing people" when "the subject itself was confusing to begin with" seems like a strictly simpler explanation).
Thanks (strong-upvoted), this is a pretty good psychoanalysis of me; I really appreciate it. I have some thoughts about it which I will explain in the remainder of this comment, but I wouldn't particularly expect you to read or reply to it unless it's interesting to you; I agree that it makes sense for you to not expend patience and effort on people you don't think are worth it.
Given that my traumatic history makes me extremely wary that attempts to validate or encode any kind of standard or minimum bar of politeness will in practice be weaponized to shut down intellectually substantive discussions, I think it makes sense for me to write critical essays in response to such attempts? It's true that someone without my traumatic history probably wouldn't have thought of the particular arguments I did. But having thought of the arguments, they seemed like a legitimate response to the text that was published.
The reason this sits okay with my conscience is because I think I apply it symmetrically. If someone else's traumatic hi... (read more)
Did you mean to link to this comment? Or another of his comments on that post…? It is not clear to me, on a skim of the comments, which specific thing that Zack wrote there might be an example of “lying by exaggeration/overconfidence” (but I could easily have missed it; there’s a good number of comments on that post).
Hmm. Certainly the first part of that is true, but I’m not convinced of the second part (“without engaging with what the original author was really talking about”). For example, you mention the post “Firming Up Not-Lying Around Its Edge-Cases Is Less Broadly Useful Than One Might Initially Think”. I found that said post expressed objections and thoughts that I had when reading Eliezer’s “Meta-Honesty” post, so it seems strange to say that Zack’s post didn’t engage with what Eliezer wrote! (Unless you take the view that what ... (read more)
Everyone sometimes issues replies that are not rebuttals, but there is an expectation that replies will meet some threshold of relevance. Injecting "your comment reminds me of the medieval poet Dante Alighieri" into a random conversation would generally be considered off-topic, even if the speaker genuinely was reminded of him. Other participants in the conversation might suspect this speaker of being obsessed with Alighieri, and they might worry that he was trying to subvert the conversation by changing it to a topic no one but him was interested in. They might think-but-be-too-polite-to-say "Dude, no one cares, stop distracting from the topic at hand".
The behaviour Raemon was trying to highlight is that you soapbox. If it is line with your values to do so, it still seems like choosing to defect rather than cooperate in the game of conversation.
Yeah, I didn't mean that I thought you two agreed in general, just on the specific thing he was commenting on. I didn't mean to insert myself into this feud and I was kinda asking how I got here, but now that I'm here we might as well have fun with it. I think I have a pretty good feel for where you're coming from, and actually agree with a lot of it. However, agreement isn't where the fun is so I'm gonna push back where I see you as screwing up and you can let me know if it doesn't fit.
These two lines stand out to me as carrying all the weight:
These two lines seem to go hand in hand in your mind, but my initial response to the two is very different.
To the latter, I simply agree that there's a failure mode there and don't fault you for being extremely wary of it. To the former though.... "I disagree that this thing should be necessary" is kinda a "Tough?". Either it's necessary or it ... (read more)
Some people want motors that are efficient, high-power or similar. Some people might instead be making a kinetic sculpture out of lego and they actually are primarily interested in whether the motor's cycle looks psychedelic and it makes a pleasing noise. Neither group are wrong.
Some people want arguments that lead efficiently to a better view of the base reality. Some people are more interested in understanding the opposing side's philosophy and how they reason about it. Some people want the argument to be engaging, fun or dramatic. Some people prioritise still being friends when the argument is over. Some people like the idea of 'winning' an argument like winning a game. None of them are wrong.
Putting a label on the people who actually want their motors to move energy efficiently (calling them "physicsts" or "engineers") and contrasting them with "artists" or something might be a useful line to draw. Similarly, "rationalist discourse" might be a poor label, but if it is was called "truth seeking discussion" or similar I think it is actually carving our a fairly specific sub-part of the possible space.
This is a good question!! Note that in the original footnote in my post, "on the same side" is a hyperlink going to a comment by Val:... (read more)
Thanks for pointing this out. (I read Val's comment while writing my post, but unfortunately neglected to add the hyperlink when pasting the text of the footnote into my draft.) I have now edited the link into my post.
I think the fact that disagreements often feel like battles between warring armies is because a lot of apparent "disagreements" are usefully modeled as disguised conflicts. That is, my theory about why predictable disagreements are so ubiquitous in human life (despite the fact that Bayesian reasoners can't forsee to disagree) is mostly conflict-theoretic rather than mistake-theoretic.
A simple example: I stole a loaf of bread. A policeman thinks I stole the bread. I claim that I didn't steal the bread. Superficially, this looks like a "disagreement" to an outside observer noticing the two of us reporting different beliefs, but what's actually going on is that I'm lying. Importantly, if I care more about no... (read more)
Whether you are building an engine for a tractor or a race car, there are certain principles and guidelines that will help you get there. Things like:
The point of the guidelines isn't to enforce a norm of making a particular type of engine. They exist to help groups of engineer make any kind of engine at all. People building engines make consistent, predictable mistakes. The guidelines are about helping people move past those mistakes so they can actually build an engine that has a chance of working.
The point of "rationalist guidelines" isn't to enforce a norm of making particular types of beliefs. They exist to help groups of people stay connected to reality at all. People make consistent, predictable mistakes. The guidelines are for helping people avoid them. Regardless of what those beliefs are.
Well, the paper says disagreement is only unpredictable between agents with the same priors, so seems like that explains at least part of this?
Reporting such evidence will make you exceptional among people who typically hold the defense position; it will no longer be fair for people to say of them "well of course the defense would say that either way". And while you may care very much about the conclusion o... (read more)
As a relatively new person to lesswrong, I agree.
The number of conversations which I've read which end in either party noticeably updating one way or the other have been relatively rare. The one point I'm not sure if I agree with is being able to predict a particular disagreement is a problem?
I suppose being able to predict the exact way in which your interlocutors will disagree is the problem? If you can foresee someone disagreeing in a particular way, and then accounting for it in your argument, and then they disagree anyway, in the exact way you tried to address, that's generally just bad faith.
(though sometimes I do skim posts, by god)
Hmm, when there is a disagreement somewhere, it is worth going back to first principles, isn't it?
If I remember correctly, Eliezer's motivations for starting the whole series of posts back on overcoming bias was "raising the sanity waterline" or something like that. Basically, realizing that you are an imperfect reasoner and striving to see your reasoning flaws and do better. This is an uphill battle, humans did not evolve to reason well at all, and different people have different classes of flaws, some are too combative, some are too accepting, the list i... (read more)
I've personally gotten the most out of people displaying epistemic technique in investigating their own problems so that I have existence proofs for all the myriad spot checks it's possible to run on one's own reasoning.
Because there's ambiguity, and there's self-fulfilling prophecies. When there's potential for self-fulfilling prophecies, there's free variable that's not a purely epistemic question; e.g. "Are we on the same side?". E.g., giving any answer to that question is in some cases implicitly deciding to add your weight to the existence of a conflict.
You role-play to add some driving force to the system--driving t... (read more)
Then it would appear that we're in a conflict over a shared resource: I want to post "Zack-type" things on Less Wrong—including long-form criticism of other posts on Less Wrong—and (assuming I'm reading your comment correctly; feel free to correct me if not) it seems like you want me to not do that.
It looks like we can't both get what we want at the same time. That's a very unfortunate situation for us to be in. If you have any suggestions for Pareto improvements, I'm listening. I'm not sure what else I can say.
A distant relative of mine (I assume, the name is rare), Dr Harold Aspden, by all accounts a well-respected and successful engineer, spent the latter part of his life advocating an 'over-unity motor'.
There are quite a lot of people who think that you can use a system of mirrors to concentrate sunlight in order to achieve temperatures higher than the surface of the sun. I myself am not sufficiently confident that this is impossible to actually be seriously surprised if someone works out a way to do it.
I think 'non-physicist motors' are a thing.
Just noting that this entire post is an overt strawman; its title and central thesis rest on the exactly backward implication that both Rob's and my posts were based on ungrounded theory when they were both built entirely out of studying and attempting to model what actually works in practice, i.e. what are the observable behaviors of people who actually-in-practice consistently and reliably produce both a) clear thinking and b) effective communication of that clear thinking, in a way that is relatively domain-agnostic. In the analogy of "physicists" vs. "... (read more)
Er, where does the OP say this…? I see no such implication. (Indeed, if anything, the OP seems to be saying that the posts in question are based on, so to speak, un-theory’d ground…)
Well… sure, you can say that. But then… anyone could say that, right? I could write a post that recommended the opposite of any given thing you recommend (e.g., “cultivating an adversarial attitude is good, while cultivating a cooperative attitude leads to worse outcomes”), and I could also claim that this recommendation was “built entirely out of studying and attempting to model what actually works in practice”. And then what would we have? Two competing claims, both backed up by exactly the same thing (i.e., nothing... (read more)
This immediately brought to mind John Nerst's erisology. I've been paying attention to it for a while, but I don't see it much here (speaking as a decade-long lurker); I wonder why.
You may be right that this one sticks out and hasn't been abstracted properly. But I do think there are truth-tracking reasons for this that are pretty general. (I think whether these reasons actually hold water is pretty dubious; rough-and-tumble spaces would very plausibly be significantly more truth-tracking than current rationalist norms; I'm ju... (read more)
"Aim for long-run mental engineering / truth-tracking information processing, not short term appearance of rule-following", or some better version, seems like an important element of truth-tracking discourse.
This place uses upvote/downvote mechanics, and authors of posts can ban commentors from writing there... which man, if you want to promote groupthink and all kinds of ingroup hidden rules and outgroup forbidden ideas, that's how you'd do it.
You can see it at work - when a post is upvoted is it because it's well-written/useful or because it's saying the groupthink? When a post is downvoted is it because it contains forbidden ideas?
When you talk about making a new faction - that is what this place is. And naming it Rationalists says something ver... (read more)