The Dark Arts As A Scaffolding Skill For Rationality

An interesting topic! Some scattered thoughts, in no particular order or organization:

Vaccination

I sometimes worry that the walled garden we have somewhat succeeded in creating within the rationalist community has left us vulnerable to the dark arts, the way that the vaccination-driven suppression of Measles leaves the USA vulnerable to the Measles coming back.

So… this shouldn’t happen, right? Taken literally? I mean, that’s the whole point of vaccines: that they make you not-vulnerable to the disease coming back. (Otherwise why would anyone bother in the first place?)

But of course what you mean (right?), or at any rate should mean, is something like: we suppressed measles with vaccines, then a bunch of people stopped vaccinating, and they got away with that for a while (because of the aforesaid vaccine-driven suppression), and then measles started coming back to those people.

That’s pretty bad! But the problem is the step where you stop vaccinating, right? It’s not like we’re worse off than before we started vaccinating! Indeed, the people who’ve continue to vaccinate are still fine (i.e., immune). But the people who stopped…

(We leave off, at least for now, discussion of stuff like “what about the few people who, for medical reasons, can’t have the vaccine”—although it’s good to remember such things, in case we need to import them into the analogy.)

Ok, so how does this map to the analogy? Well, perhaps something like: we learned how to deal with Dark Arts, then we stopped teaching Defense Against the Dark Arts, and now we have a whole bunch of people who never learned it, and never had to learn it because they weren’t exposed to it, so they got away with that for a while, but then Dark Arts started coming back!

And, yeah, that’s bad for sure—but the problem, again, is the step where we stopped vaccinating people…

So the solution is to go back to what we were doing before, i.e. to resume the study and practice that allowed us to build those walls around our garden in the first place. That way we can keep the walls in good repair, rebuild them if they get damaged, and even deal with problems that somehow sneak in past the gates.

(Of course, this is severely undermined by people who want us to stop carrying swords.)

Lying

I actually do think that lying convincingly is a rationalist skill. I think this for two reasons, which both seem obvious to me, so if you disagree upon reflection (as opposed to just “hm, hadn’t thought of that before”), I’d like to hear why.

Knowing how to lie makes you better at detecting lies by others.

Pretty standard reasoning here. Knowing how do X makes you better at detecting and fighting X, for many (most?) values of X. Certainly true for lying, in my experience.

Sometimes you should lie.

Also pretty straightforward. Sometimes lying is the right thing to do. And in such a case, it would obviously be preferable to be good at it. Like, I know the “Gestapo at the door, Jews in your attic” example is overused… but, well, there are plenty of modern analogues (I omit examples to avoid political arguments, obviously). If you’re bad at lying to the Gestapo then you and your friends get tortured and killed. If you’re good at lying to the Gestapo then you save lives (including your own) and help ensure that the good guys win.

LARPing

I don’t have much experience with LARPs, but much the same is true in D&D and similar TTRPGs, in my experience. For DMs especially, lying is absolutely a core skill. If you’re bad at it then you’re going to be worse at running a good game, period. It’s a wonderfully effective training ground for deception.

[-]Screwtape3mo60

re: Vaccination:

But of course what you mean (right?), or at any rate should mean, is something like: we suppressed measles with vaccines, then a bunch of people stopped vaccinating, and they got away with that for a while (because of the aforesaid vaccine-driven suppression), and then measles started coming back to those people.

Yeah, I was gesturing quickly at a dynamic without going into detail but this is about right. One other thing I'll note while we're spelling it out a bit more is that the Measles vaccine isn't 100% effective. Checking the CDC page quickly, it's about 97% effective. That's plenty to suppress the disease such that it can't get a chance to infect even those 3% it could still make sick if it could get to them, but you could imagine someone sick with Measles coming into a completely vaccinated population (maybe they got on a plane from some other country where it was still prevalent) and shaking hands with/sneezing around a hundred people at a conference. The population might be entirely vaccinated, but some of them get sick anyway.

Likewise, if I imagine some kind of practice making people catch and internally discard 97% of manipulation attempts, about three in a hundred still wind up falling for it.^[1]

You don't need to get to 100% coverage, though that's nice if one can do it. Moving the needle from 50% to 75% of people and effectiveness is often worth doing.

Ok, so how does this map to the analogy? Well, perhaps something like: we learned how to deal with Dark Arts, then we stopped teaching Defense Against the Dark Arts, and now we have a whole bunch of people who never learned it, and never had to learn it because they weren’t exposed to it, so they got away with that for a while, but then Dark Arts started coming back!

Broadly agreed, though it would not surprise me if "we learned how to deal with Dark Arts" is a bit more like "we selected for people who independently figured out how to deal with Dark Arts." I mean, I think e.g. the sequences contain a bunch of good Defense Against The Dark Arts material, but the percentage of the community that's read the sequences has mostly gone up over the last decade or so.

Like, one mechanism I can imagine is most people are good at Defense Against The Dark Arts, they downvote or ban the people who do Dark Artsish manipulation tactics, there's a 'golden age' for a few years as new people aren't arriving that fast and one can basically trust the old hands who can and will point out when someone's up to something, then some critical mass of newcomers forms that isn't good enough at Defense Against The Dark Arts themselves and also doesn't use the Dark Arts so they never tripped any of the defences.

I don't have a strong hypothesis as to the mechanism to be clear: epistemic status is "I sometimes worry" not "I'm sure this has happened and here's how."

(Of course, this is severely undermined by people who want us to stop carrying swords.)

-huh. You and I seem to have read that essay very differently if you think it's about trying to get us to stop carrying swords. (I'm interpreting "stop carrying swords" as something like. . . stop actively defending the perimeter of the walled garden? stop actively practicing and using Defense Against The Dark Arts? Am I close with those interpretations?)

There's a rest-of-the-world. There's (ideally!) a place-where-we're-all-good-at-DADA. Either we're primarily running off of selection effects to fill the place we're all good at it, or at some point the new people have to get taught. That seems like a pretty natural grayspace to me; a zone where I expect a bunch of people to be bad at defending themselves and also where they might not have learned when not to use Dark Arts yet. I don't want people to stay in that zone forever, I want them to graduate to the place where we're all good at it and to fail out the people who can't or won't hack it.

re: lying:

1. Knowing how to lie makes you better at detecting lies by others.
Pretty standard reasoning here. Knowing how do X makes you better at detecting and fighting X, for many (most?) values of X. Certainly true for lying, in my experience.

Seems plausibly true! And yet if I imagine somehow being able to magically become amazingly good at detecting and fighting lies while not being much better at lying, that seems fine. All else being equal I'd rather just advance the defense here.

The ability to lie well would be a side effect, not part of the goal. Maybe the best way to teach detecting lies is to teach telling them, and there isn't a route to defence that doesn't path through offence.

2. Sometimes you should lie.
Also pretty straightforward. Sometimes lying is the right thing to do. And in such a case, it would obviously be preferable to be good at it.

Ah, here I think we have a definitions question.

Sometimes lying is a very useful skill to have and it would improve your life to be able to do it well, but the same is true of firing a gun and of playing the guitar. I don't really think of those as "rationalist skills" despite the fact that if I could play like Taylor Swift I could make a lot more money than I currently do. If the Nazis come for my country, I might need to lie to them, but I also might need to shoot at them.

That makes sharpshooting a useful skill, and a rational skill to get good at. Does that make it a "rationalist" skill?

re: larping:

Seems we basically agree here.

^{^}
In a simplified model anyway. There's variations where herd immunity makes sense (maybe you fall for the manipulation, repeat it out loud, and your buddy next to you doesn't fall for it and points out the flaw) and variations where it's a bit more like a lock that randomly opens 3% of the time (if the other person gets to keep resetting and trying again, they'll get you eventually.)

[-]Said Achmiz3mo180

Re: models of vaccine effectiveness: not disagreeing with anything here in particular… at this level of detail the analogy does start breaking down, so probably best to talk non-analogically once we’re digging down into the details.

… it would not surprise me if “we learned how to deal with Dark Arts” is a bit more like “we selected for people who independently figured out how to deal with Dark Arts.”

Absolutely agreed now that you put it like that; this is a great point.

The usual example springs immediately to mind, and so the obvious follow-up thoughts go something like this:

If we select for people who’ve independently figured these things out, then we should do our best to learn from them. We should try to keep them around if at all possible; we should pay attention to their commentary on the matter as well as to their behavior; and we should preserve and promulgate their insights.

After all, selective methods are great, but we can’t rely on them totally; many people must be taught these things. And surely those who’ve figured this stuff out independently didn’t start out knowing it—how did they learn? And how can we help others learn likewise?

(Just reading the relevant writings and paying attention to things in light thereof was enough for me, for instance. Now, that latter clause does gloss over a bit of the difficulty, but still—the first part is required!)

Like, one mechanism I can imagine is most people are good at Defense Against The Dark Arts, they downvote or ban the people who do Dark Artsish manipulation tactics, there’s a ‘golden age’ for a few years as new people aren’t arriving that fast and one can basically trust the old hands who can and will point out when someone’s up to something, then some critical mass of newcomers forms that isn’t good enough at Defense Against The Dark Arts themselves and also doesn’t use the Dark Arts so they never tripped any of the defences.

I don’t have a strong hypothesis as to the mechanism to be clear: epistemic status is “I sometimes worry” not “I’m sure this has happened and here’s how.”

Hmm… I don’t think that model captures LW’s trajectory in these dimensions. I’m not ready to offer a better one quite yet, but I’ll give it some thought.

(Of course, this is severely undermined by people who want us to stop carrying swords.)

-huh. You and I seem to have read that essay very differently if you think it’s about trying to get us to stop carrying swords. (I’m interpreting “stop carrying swords” as something like. . . stop actively defending the perimeter of the walled garden? stop actively practicing and using Defense Against The Dark Arts? Am I close with those interpretations?)

See e.g. this thread and this one. And also, linked OP itself says:

The source of tragedy here is a deep cultural mismatch. Bryce lives in a world where attacks are frequent, and can come from any direction, and thus you have to have your shields up at all times and your sword loose in its scabbard.

(And also everyone knows this and expects this, and thus no one takes it personally and no one thinks you ought to be doing anything differently, what do you mean, leave the house without your sword, do you want to get yourself killed?)

Alexis, on the other hand, comes from a place where attacks are rare and unusual, and thus seeing Bryce ease their sword in its sheath feels like a threat. Alexis might not have even clocked that Bryce was carrying a sword, nobody does that around here, I thought that was a walking stick or a prop or something, are you kidding me, what the fuck is going on?

Similarly: there are kinds of conversation (and kinds of interaction, and experiences generally) that you cannot have, in a world where everyone walks around gripping the hilt of their swords.

So, I mean… yeah, the essay absolutely is about trying to get us to stop carrying swords. It says as much.

I’m interpreting “stop carrying swords” as something like. . . stop actively defending the perimeter of the walled garden? stop actively practicing and using Defense Against The Dark Arts? Am I close with those interpretations?

Well, that’s just the thing. As soon as we start thinking that it’s just the perimeter that we’ve got to defend, we’re hurtling down the steep and slippery slope toward the pit of failure. No, we have to defend against the Dark Arts that are all around us and even in ourselves! CONSTANT VIGILANCE!

Of course, CONSTANT VIGILANCE needn’t be all that great a burden. We do not, for example, need to take every tiniest signal of possible treachery, from someone who has proven themselves trustworthy, to signal that we must immediately adopt our most combative posture. It’s fine to note the signal, to slightly tense, to make a small corrective move, etc. After all, we should hardly be maximally combative with ourselves, and yet we must still be on the lookout for treachery from within—how else to do so but carefully and in measured increments? Likewise, adopting practices that preclude Dark Arts is just good sense.

But if we rule it a transgression, to be vigilant, to be on the lookout for treachery—well, now we’re simply sabotaging ourselves.

To put it another way, what is foolish is the idea that we can defend against Dark Arts by some sort of person-level inclusion test: someone passes the test? Great, now we never again have to think about whether anything they say or do is suspect. The reductio of this, I emphasize again, is the one-person scenario. You can have a garden which consists only of yourself, and even then you cannot relax! You must watch yourself for Dark Arts. You can trust yourself, but still must verify—even then!

So how can this be any less true for a group, made up of God only knows how many people, who know each other not nearly so well as you know yourself…?

re: lying:

Knowing how to lie makes you better at detecting lies by others.

Pretty standard reasoning here. Knowing how do X makes you better at detecting and fighting X, for many (most?) values of X. Certainly true for lying, in my experience.

Seems plausibly true! And yet if I imagine somehow being able to magically become amazingly good at detecting and fighting lies while not being much better at lying, that seems fine. All else being equal I’d rather just advance the defense here.

The ability to lie well would be a side effect, not part of the goal. Maybe the best way to teach detecting lies is to teach telling them, and there isn’t a route to defence that doesn’t path through offence.

Well… ok, let’s say for the moment that I accept this reasoning, but—“magically” is indeed the right word there. Yeah, if things were radically different, then… things would be different. In our actual world, knowing how to lie makes you better at detecting lies. (Or so I claim! Obviously, if this is not true, then all bets are off.) If you want to advance the defense, there’s some degree to which you can do that without ever improving your offense, but it is not the maximal degree.

(It’s sort of like saying: “if I imagine somehow being able to magically become amazingly good at baking cakes while not being much better at reading and interpreting recipes, that seems fine”. And, sure, that’s not logically impossible, but in the real world, you do in fact need to know how to read and understand a recipe in order to be really good at baking a cake.)

Does that make it a “rationalist” skill?

Mm… the principle of “just because something is a good thing does not mean that it is a ‘rationalist’ thing” is of course entirely correct. The question is just what qualifies some skill as a “rationalist” skill.^[1]

If we take “rationalist skill” to mean only such skills as are somehow “meta” or apply to the development and practice of techniques of rationality, then my reason #2 should not convince us that lying is a rationalist skill.

If we take “rationalist skill” to mean also such skills as are instrumentally-convergently useful in a near-universal set of classes of circumstances, regardless of details, with the only requirement being that interaction between non-perfectly-aligned agents is a significant aspect of the scenario, then my reason #2 should convince us that lying is a rationalist skill.

As you say, this is a matter of definitions, so we are not really disagreeing on any question of fact when we debate which of the above definitions to use. I have no strong preference here, but it does seem to me that the latter usage is at the very least not unknown, and possibly is quite common (I have never before given detailed thought to this specific question, so I’m not prepared to opine confidently on the actual prevalence).

Caveat: this whole section only applies to reason #2 of the two I gave; conditional on being convinced by reason #1, you should still think that lying is “a rationalist skill” even if reason #2 does not convince you.) ↩︎

[-]Screwtape3mo20

- Selection vs Learning

If we select for people who’ve independently figured these things out, then we should do our best to learn from them. We should try to keep them around if at all possible; we should pay attention to their commentary on the matter as well as to their behavior; and we should preserve and promulgate their insights.

Somewhat agreed. Sometimes people who are good at the thing aren't good at teaching it, or even at being in the environment in which it's taught.

To use physical combat as an example, maybe the world's best fighters are hardened criminals and hyper-vigilant soldiers. If that's the case I'd guess the move is to have instructors to study them, talk with them, and then I want the instructors to make the grayspace safer for students than criminals would allow and kinder to the teachers than hyper-vigilance and flailing limbs would combine to be.

(I'm two steps down hypotheticals here- one being that the people good at the thing aren't necessarily the best at teaching it, and second that there's a better way to teach than just throwing students into the deep end. I don't presently have citable evidence for either of those, just intuition and metis.)

If I may use fictional evidence as an example for a moment, Dr. House would be a terrible biology 101 lecturer. I'm a fan of figuring out how to teach the things and then trying to make sure they get taught.

There's a pretty important ~crux of "okay, what are you trying to do right now though?" for me. I spend a lot of my time trying to teach what I view as basic and early-intermediate skills, or trying to be a force multiplier for other people to teach those. To do that well, it helps me to have a sense of what the advanced skills are like, and for what people do with these skills out the in the world, and it costs a decent amount if someone is bad at explaining themselves or drives people to drop the class. If someone views themselves as managing an intensive care unit instead of managing a (deeply weird and distributed and non-standard) education system, they might correctly be a lot more willing to put up with various drawbacks in exchange for brilliant diagnostic insights.

(I also spend time doing operations work that has only downstream relationship to rationality skills, and time handling conflict management in a way where spotting lies is directly useful to me.)

- Swords

Okay. I agree that the text of the essay is talking about putting away swords under some circumstances. If you are interpreting it as "put away swords at all, ever" then yes. Something about the connection with repairing and maintaining the way lead me to internally complete it as "put away swords all time time, such that you never pick one up" and that's different. To the extent where that's my misunderstanding, oops, I apologize.

So far I'm in agreement with Duncan here.

Of course, CONSTANT VIGILANCE needn’t be all that great a burden.

I don't know what your internal experience of CONSTANT VIGILANCE is like. For me, in interpersonal social situations, it sucks. I dislike the feeling. I have a lot of trouble setting my mental dial high enough to reliably catch manipulative shenanigans and low enough that I don't waste hours cross checking my notes on what people say against the physical evidence that exists, letting pass the normal mistakes of normal human minds and picking up on the patterns of malefactors.

"Likewise, adopting practices that preclude Dark Arts is just good sense." If we as a collective rationalist community (or better yet a general civilization) had an agreed upon canon of techniques or language use which precluded particular Dark Arts, that'd be great. We don't. Again, I spend a lot of my time in the top of the funnel with first-timers and old-timers-who-nobody-taught. So when I spot someone doing something that seems Dark Artsy- not just at the "metaphorically carrying a sword" level but the "metaphorically slashing" level- I have a bunch of extra work to figure out whether nobody told them they shouldn't, whether they'll stop if I explain the better way to do it, whether stopping even makes sense or if we're just actually in an environment where unilaterally disarming would be a bad idea.

But if we rule it a transgression, to be vigilant, to be on the lookout for treachery—well, now we’re simply sabotaging ourselves.

I think turning the sensitivity dial all the way down, such that you wouldn't notice any amount of sword usage, is a mistake.

I think different responses to it based on context makes a lot of sense. If someone throws a punch at me on the street, I'm in fight mode. If someone who isn't the teacher throws a punch at me seemingly randomly in the dojo, I block and calmly explain that we don't do that, we throw punches at people when in the ring sparring or when told to drill. (And if the teacher does it, it kinda depends on the teacher's style and style of the dojo.) If someone steps into the ring opposite me, we bow to each other, the ref says go, and they throw a punch at me, this is fine.

You used the phrase "lookout for treachery." Yeah, throwing a punch during an agreed on spar isn't treachery, sucker punching someone when they're tying their shoelaces is. We're using the sword metaphor a lot. With rationality, I think there's moves that are actually just normal and expected in much of the world and would be disallowed in the inner space of Duncan's dojo. Like, when I buy a used car in a strange town, I basically expect the dealer to try and screw me over and not tell me about a history of engine trouble. I'm not mad at the dealer for trying it. If I was buying a used car from an old friend, I would expect them to tell me if there was engine trouble, and I'd be mad if they conveniently forgot to mention it. It's not even treachery when the dealer does it, that's basically how everybody knows buying used cars works.

To put it another way, what is foolish is the idea that we can defend against Dark Arts by some sort of person-level inclusion test: someone passes the test? Great, now we never again have to think about whether anything they say or do is suspect.

That seems to be an exaggeration. I agree that a person-level, one-time test after which you trust someone entirely would be a mistake. I would be surprised if Duncan said that you couldn't get bounced out of Master Lee's advanced classes once you were in. Are you suggesting that's actually what he's saying?

But trusting someone more because of what they've done, with less regular checks, seems fine. One doesn't give every single user full admin powers, but the I.T. staff has more power, and the Chief Technology Officer might have essentially full power in the system. And yet, if someone notices the logs are weird and it looks like the CTO is up to something, that still gets flagged.

So how can this be any less true for a group, made up of God only knows how many people, who know each other not nearly so well as you know yourself…?

I do not believe it will ever become entirely, 100% true. I do believe it can become true more often, of more people, if slow and steady work is put into the project. I think that's worth trying.

When I walk down the street, full of strangers I have never met before and will never meet again, I don't particularly worry someone is going to punch me in the face. If someone steps into my path and starts yelling at me, my odds go up a little and maybe I make sure my hands are free- but my odds aren't actually very high yet, because more people just yell about conspiracy theories or the unfairness of the world than commit assault. And then if I see them start to draw back and swing, now my odds are actually high.

We have, as a civilization, achieved a world where adults in our cities don't need to carry swords.

Thus so with manipulation and lies. I kinda trust random people on the street, though I wouldn't put too much weight on it. I basically did trust the people I worked with at my old software job- sometimes I double checked to cover their mistakes, I was aware enough that office politics existed that I think I would have noticed a string of problems, but we also had all been through the educational and acculturation processes that meant we broadly agreed on what good software was like and what behavior towards each other was appropriate. But that education and acculturation wasn't just pointless busywork! (Some of it was, but some of it wasn't!) University was a gray space where the really dumb and the really rude got corrected or weeded out- not perfectly, but usefully.

Yeah, I try and check my own code for problems. Yeah, I did code reviews and checked my coworker's code for problems. But there's finite amount of time and energy and I scrutinized incoming pull requests from the public way closer than I scrutinized my coworker's changes.

And circling back to rationality and what I said about context above: one can take in some portion of the masses that haven't learned how not to ad hominem or how to taboo their words so it isn't surprising if they try, teach them the thing and that it's expected of them in this space, if not out in the world, and then pass them on to a zone where one can just expect that everyone in here knows they aren't supposed to do that and has demonstrated that they're capable of it. That doesn't mean one can 100% know nobody will do it, but it changes the rate significantly and it would change my reaction from "ah, you have made a common error, let me show you the correct move" to "you know better than that. What's the deal?"

I would love more gray spaces.

- Rationalist Skill

Mm… the principle of “just because something is a good thing does not mean that it is a ‘rationalist’ thing” is of course entirely correct. The question is just what qualifies some skill as a “rationalist” skill.

I think this is a good question, and also it verges on "what is the full list of rationalist skills or at least a " which seems a bit big for this particular comment section. I think spotting lies and more general manipulation and deceit is pretty centrally a rationalist skill.

If we take “rationalist skill” to mean only such skills as are somehow “meta” or apply to the development and practice of techniques of rationality, then my reason #2 should not convince us that lying is a rationalist skill.

And that seems self-referential. Hockey skills are the ones that apply to the practice of techniques of hockey, chess skills are the ones that apply to the practice of techniques of chess, florgle skills are the ones that apply to the practice of techniques of florgle. If we have a solid definition of rationality (ideally one concise and generally agreed upon, but any working definition is fine for the purposes of conversation between a few people) then rationality skills are the skills which apply to rationality.

From LessWrong's About page:

Rationality has a number of definitions^[1] on LessWrong, but perhaps the most canonical is that the more rational you are, the more likely your reasoning leads you to have accurate beliefs, and by extension, allows you to make decisions that most effectively advance your goals.

(If you think that's a poor definition of rationality, that's fine, but I think that's a close-enough definition that I'm fine using it and the onus is on whoever disagrees to state what they think rationality is.)

It's again plausible to me that getting better at lying makes someone better at detecting lies (thus having more accurate beliefs.) And of course, in the right circumstance being able to lie well certainly can advance your goals. I'm not against learning to lie in the pursuit of learning to detect lies- that's practically the whole point of this essay. But I do think it's worth noting the distinction, and that the goal is learning to detect lies and other manipulations.

If we take “rationalist skill” to mean also such skills as are instrumentally-convergently useful in a near-universal set of classes of circumstances, regardless of details, with the only requirement being that interaction between non-perfectly-aligned agents is a significant aspect of the scenario, then my reason #2 should convince us that lying is a rationalist skill.

That argument proves too much I think. The ability to inflict pain and physical damage on other people is convergently useful in interactions between non-perfectly-aligned agents (and I'm Hobbesian enough to think it's near-universally important at least in potential) but that doesn't mean I think marksmanship with guns is a rationalist skill. Same with the ability to make friends, or communicate clearly and understand what other people are trying to communicate; it's useful (probably in even more circumstances than lying is) but I don't think of clear communication or making friends as a specifically rationalist skill. It's just a useful thing to be able to do.

[-]Said Achmiz3mo*110

Re: learning from the pros:

Indeed the best pros might not be the best teachers, but this is exactly why I didn’t say “we should install them as teachers”; I said “we should do our best to learn from them”. That need not require that they teach. It can indeed look like what you describe—having instructors mediate the process, etc.

On the other hand, it’s worth remembering that the actual domain that we’re talking here mostly involves words, not guns or punches or medical procedures. The way that skill manifests is mostly in words. The means by which we encounter the pros (or anyone else) mostly involves words. For my example, I linked to an old Less Wrong comment, which is made of words. This inherently biases the “best way to learn from the pros” distribution toward learning directly from the pros, without some sort of meditation.

And there’s a serious danger in taking the sort of “at some remove” approach you describe, which is that there’s really no guarantee that the instructors are… well, any good. In many ways. Maybe they don’t really understand what they themselves learn from the pros, maybe they aren’t any good at teaching, maybe they introduce distortions (either purely accidentally, or through a sort of teacher’s déformation professionnelle, or deliberately in the name of effectively teaching a wider range of student, etc.), maybe who knows what. And if the students don’t interact with the pros directly, how do they get feedback on what they’ve learned? From the same instructors? That is a very different thing from having on hand the one who has mastered the skill you are trying to learn!

What’s more, there is an element of absurdity to all of this, which is that while the model of pros, instructors, and students might make sense for many sorts of practical skills, in the domain that we’re actually discussing, the far more likely scenario is that, within a community, there are some who are pros at one thing and some who are pros at another thing, and the ideal is that we all learn from each other (with an unequal distribution of “sources of mastery”, to be sure, but nevertheless no sharp divides!). If I am a master at one thing, and you at another, then if we are equals within a community, we can both learn from each other; and others can learn from us both, at once; and we can improve together. But separate us, place a layer of instructor-interpreters between each of us and the rest of the community (including the other), and you replace an extremely high-bandwidth, bi-directional channel with a unidirectional and very narrow one. It’s hard to see how such a setup could be worth the effort.

Okay. I agree that the text of the essay is talking about putting away swords under some circumstances. If you are interpreting it as “put away swords at all, ever” then yes. Something about the connection with repairing and maintaining the way lead me to internally complete it as “put away swords all time time, such that you never pick one up” and that’s different. To the extent where that’s my misunderstanding, oops, I apologize.

The key question is whether we should put our swords away while we’re within the community. Duncan’s essay says yes. I say no, for the reasons I give.

Of course, CONSTANT VIGILANCE needn’t be all that great a burden.

I don’t know what your internal experience of CONSTANT VIGILANCE is like. For me, in interpersonal social situations, it sucks. I dislike the feeling. I have a lot of trouble setting my mental dial high enough to reliably catch manipulative shenanigans

For me it’s just the normal and completely unproblematic state of affairs. It requires no effort. If I’m not sleep-deprived or otherwise impaired, it’s passive and comes naturally.

And this is the point of having a “community” (broadly understood), right? I mean, this is why we have a comments section under every post, instead of just people shouting into the void. Different people have different skills. One of those skills is the skill of “catching manipulative shenanigans” (although I wouldn’t quite put it that way myself). So if you read a post, you don’t need to have your CONSTANT VIGILANCE dial set real high. You can scroll down to the comments and see what other people, to whom it comes more naturally, have to say. (And what about those people’s comments? Well, if they do shenanigans, then other such people can catch them.)

Division of labor, you see. You don’t need to be good at everything. From each according to his ability…

Unless, of course, you forbid those people from putting their skills to work. Then you’re back to having to do it yourself. Or just trusting that you won’t need to. Seems bad.

“Likewise, adopting practices that preclude Dark Arts is just good sense.” If we as a collective rationalist community (or better yet a general civilization) had an agreed upon canon of techniques or language use which precluded particular Dark Arts, that’d be great. We don’t.

Come now, this is a silly objection. Of course we have some disagreements about this, but it’s not like we don’t know anything about what practices guard against such threats. Of course we do! As a “general civilization” and as a “collective rationalist community”! We have the scientific method, we have the Sequences, etc. We know perfectly well that principles like “you need to provide evidence for your claims” and “you need to not write your bottom line first” and “you need to apply your abstract principles to concrete cases” and “you can’t just conclude that something is true because one guy said so” guard against a very broad class of manipulations. It is just completely untrue to say that we don’t agree on any practices that preclude Dark Arts.

With rationality, I think there’s moves that are actually just normal and expected in much of the world and would be disallowed in the inner space of Duncan’s dojo. Like, when I buy a used car in a strange town, I basically expect the dealer to try and screw me over and not tell me about a history of engine trouble. I’m not mad at the dealer for trying it. If I was buying a used car from an old friend, I would expect them to tell me if there was engine trouble, and I’d be mad if they conveniently forgot to mention it. It’s not even treachery when the dealer does it, that’s basically how everybody knows buying used cars works.

Yes, of course. And my point is, how do you keep these things from happening in the dojo^[1]? You need to remain on the lookout for it happening. You can’t just say “don’t do this” at the door and then assume that nobody’s doing it and punish anyone who makes any effort to figure out whether someone’s doing it. That’s crazy.

To put it another way, what is foolish is the idea that we can defend against Dark Arts by some sort of person-level inclusion test: someone passes the test? Great, now we never again have to think about whether anything they say or do is suspect.

That seems to be an exaggeration. I agree that a person-level, one-time test after which you trust someone entirely would be a mistake. I would be surprised if Duncan said that you couldn’t get bounced out of Master Lee’s advanced classes once you were in. Are you suggesting that’s actually what he’s saying?

Is he saying it? Of course not. Does his proposed structure directly entail it? Yep.

But trusting someone more because of what they’ve done, with less regular checks, seems fine. One doesn’t give every single user full admin powers, but the I.T. staff has more power, and the Chief Technology Officer might have essentially full power in the system. And yet, if someone notices the logs are weird and it looks like the CTO is up to something, that still gets flagged.

Nah, bad analogy.

The correct analogy would be: you’re not supposed to send proprietary company info through unencrypted email. But you trust the CTO more, so if he sends proprietary company info through unencrypted email, it’s fine.

That would be totally crazy, right? Of course the CTO shouldn’t be doing that! Indeed it’s much more important that the CTO doesn’t do that (because his emails are more likely to contain stuff that you really don’t want leaking to your competitors, because he’s more likely to be the target of attackers, etc.).

If you’re the CEO, and you call in your CTO and tell him to immediately stop sending proprietary company info through unencrypted email and what the hell was he even thinking, seriously, and he protests that it’s not like you have any particularly reason to believe that your company secrets have actually been stolen, so what’s the problem…

… then you fire the guy. Right? Or you should, anyway. His defense is not only not worth anything, it betrays a fundamental failure to understand security or… just common sense.

I’ve made this point many times before, but: rationality isn’t about being so rational that we can do dumb things and still be ok, it’s about not doing the dumb things.

We have, as a civilization, achieved a world where adults in our cities don’t need to carry swords.

Replace the word “swords” with “guns” and it should instantly become obvious that this claim is a highly controversial one, about which many political arguments might be had.

Yeah, I try and check my own code for problems. Yeah, I did code reviews and checked my coworker’s code for problems. But there’s finite amount of time and energy and I scrutinized incoming pull requests from the public way closer than I scrutinized my coworker’s changes.

Of course, but presumably this is because you write your own code with proper techniques and approaches, you reflexively avoid programming styles or design patterns (or anti-patterns) that are bad ideas, etc. In other words, your checks are built in to the process which generates the code.

Now suppose that you have a very trusted coworker of whom you know that he is as good a coder as you are. One day he submits a PR, you skim it casually, expecting to find nothing but the usual good code, but notice that it’s full of nonsense and absurdity. Well, now you have to read it more carefully, and ask your coworker what the heck is up, etc.

That doesn’t mean one can 100% know nobody will do it, but it changes the rate significantly and it would change my reaction from “ah, you have made a common error, let me show you the correct move” to “you know better than that. What’s the deal?”

Sure, sure. But then, I tend to think that both of those formulations presume rather more than is necessary. Sidestep both of them by saying “that is such-and-such error”, the end. If the one says “huh? what are you talking about?”, then you follow up by saying “here is why it’s an error, and here is how to avoid it”. If the one says “normally it would be, but not in this case, for these-and-such reasons”, great. The most important thing is that you notice the error and point it out. The details of wording are inconsequential.

So how can this be any less true for a group, made up of God only knows how many people, who know each other not nearly so well as you know yourself…?

I do not believe it will ever become entirely, 100% true. I do believe it can become true more often, of more people, if slow and steady work is put into the project. I think that’s worth trying.

Totally. And the only way it will become more true is if we continually work to ensure that it remains true, by verifying, no matter how much we trust. There can be no other way.

I hate the “dojo” metaphor, by the way. I really wish we’d do away with it. ↩︎

[-]Said Achmiz3mo20

Mm… the principle of “just because something is a good thing does not mean that it is a ‘rationalist’ thing” is of course entirely correct. The question is just what qualifies some skill as a “rationalist” skill.

I think this is a good question, and also it verges on “what is the full list of rationalist skills or at least a ” which seems a bit big for this particular comment section. I think spotting lies and more general manipulation and deceit is pretty centrally a rationalist skill.

No argument there.

If we take “rationalist skill” to mean only such skills as are somehow “meta” or apply to the development and practice of techniques of rationality, then my reason #2 should not convince us that lying is a rationalist skill.

And that seems self-referential. Hockey skills are the ones that apply to the practice of techniques of hockey, chess skills are the ones that apply to the practice of techniques of chess, florgle skills are the ones that apply to the practice of techniques of florgle. If we have a solid definition of rationality (ideally one concise and generally agreed upon, but any working definition is fine for the purposes of conversation between a few people) then rationality skills are the skills which apply to rationality.

Ehh… I don’t think that this is quite right.

For one thing, the term in question is “rationalist skill” and not “rationality skill”. There’s a difference in connotation… but let’s let this pass for now.

More importantly, I think that your examples are actually all non-circular. Yes, chess skills are the ones that apply to the development (this part can’t be left out!) and practice of techniques of chess. This seems like a perfectly reasonable statement to me. You have the game of chess; you have techniques of playing the game of chess; you have skills which apply to the development and practice of those techniques. Where’s the circularity?

Take baking. You’ve got the act of baking a pie. You’ve got techniques of baking a pie, like “bake covered at a certain temperature (e.g. 425 °F) for a certain time (e.g. 25 minutes), then uncover and continue to bake at some potentially different temperature (e.g. 350 °F) for some more time (e.g. 35 more minutes)”. And you’ve got skills that apply to those techniques, like “carefully removing a sheet of aluminum foil from a hot pie dish in the oven without disturbing the pie or burning yourself”. Seems straightforward to me.

It’s again *plausible *to me that getting better at lying makes someone better at detecting lies (thus having more accurate beliefs.) And of course, in the right circumstance being able to lie well certainly can advance your goals. I’m not against learning to lie in the pursuit of learning to detect lies- that’s practically the whole point of this essay. But I do think it’s worth noting the distinction, and that the goal is learning to detect lies and other manipulations.

Well, again, these are two separate points that I’m making here. One is that getting better at lying makes you better at detecting lies. Another is that being good at lying is useful on its own. They are not really related to each other. You can accept one and not the other.

If we take “rationalist skill” to mean also such skills as are instrumentally-convergently useful in a near-universal set of classes of circumstances, regardless of details, with the only requirement being that interaction between non-perfectly-aligned agents is a significant aspect of the scenario, then my reason #2 should convince us that lying is a rationalist skill.

That argument proves too much I think. The ability to inflict pain and physical damage on other people is convergently useful in interactions between non-perfectly-aligned agents (and I’m Hobbesian enough to think it’s near-universally important at least in potential) but that doesn’t mean I think marksmanship with guns is a rationalist skill. Same with the ability to make friends, or communicate clearly and understand what other people are trying to communicate; it’s useful (probably in even more circumstances than lying is) but I don’t think of clear communication or making friends as a specifically rationalist skill. It’s just a useful thing to be able to do.

Hm… I think it proves exactly the right amount, actually?

Like, “marksmanship with guns” is an unnecessary increase in specificity; if you instead say “the ability to inflict pain and physical damage on other people is convergently useful in interactions between non-perfectly-aligned agents, and is therefore a rationalist skill” then… I think that this is just true (or rather, it fails to be a reductio—the example doesn’t undermine the appeal of the second proposed definition of “rationalist skill”).

(The skill of selecting the optimal method of inflicting pain and physical damage, given your abilities and resources, the skill of deciding what other skills to develop given some goal, etc.—these are also “rationalist skills” in the same sense! So given that you want to be able to inflict pain and physical damage on other people, the question then is how best to do so; “develop marksmanship skill with guns” is one answer, but not the only one.)

Same with the ability to make friends, or communicate clearly and understand what other people are trying to communicate; it’s useful (probably in even more circumstances than lying is) but I don’t think of clear communication or making friends as a specifically rationalist skill. It’s just a useful thing to be able to do.

Same as above.

Note, I am not advocating strongly for the second, more expansive, sort of definition of “rationalist skill”; as I said, I have no strong preference. But I do think that the second definition is basically coherent, and doesn’t lead to absurdity or proving-too-much etc. (There may be other reasons to disprefer it, of course.)

[-]JenniferRM3mo15-2

Consider: just simple basic traditional DEBATE... in its modern form!

Before overcomingbias or lesswrong existed, and before I became friends with anyone who would (eventually) end up being In The Rationality Community... I did inter-collegiate policy debate (CEDA/NDT) debate for two years in a community college.

I don't know anything similar that gives the same fusion of "adrenaline and logic" with super fast and super precise feedback and clear skill progression.

It was because of policy debate that I haunted libraries for many hours every week, doing opposition research on winning affirmative cases and putting together my own affirmative case (leveling up library science skills), and it was because of that practice that I found and read Goedel Escher Bach. Its good stuff that causes good stuff!

I loved it so much that for two more years I coached my old public High School's debate team for policy debate (they already had a very strong individual and dramatic program, like for reading poems or reproducing famous old speeches or giving one-shot informative topical speeches to persuade, but the Program Director wasn't comfy with debate debate).

My two teams of two kids (four total students) did great (both teams were in the Finals both years), and we won "Best Small School" two years straight, and then I transferred to a UC and had to stop coaching, but the rumors I heard through the grape fine is that our tiny little public high school had acquired a reputation and lots of the big schools private high schools were starting to team up to do "cooperative ops research" against us! Yay! I wish I could have coached a third year to see how that would have gone.

I like to imagine that at least some of our supernormal outcome was my coaching, but to be fair I I think my kids were like "literally geniuses"? (Like, the smartest of them maxed the SATs and later earned a PhD in Applied Math.)

So you have to subtract that out of my credit somehow if you're trying to do proper credit assignment. I don't think a single one of my four kids had an iq less than 125.

My debate experiences are one of the things I would have pointed to early in life if I was following some of the logic in Mandatory Secret Identities (MSI), which has a lot of fun and interesting ideas around status allocation in dojos (assuming debate teams don't count as rationality dojos in themselves) but that whole dojo design vibe is full of weird stuff...

Like from the name of the essay (rather than the detailed contents) it is easy to mis-use the title and imagine gwern is doing "Mandatory Secret Identities" the best of anyone by only teaching through a pseudonym?

But that's the opposite of MSI's actual internal contents!

According to the MSI essay's detailed claims, the rationality teacher's "real life" is supposed to be taken into account somehow and so gwern should have zero status or something?

But then... think gwern is awesome and any sane and good status allocation mechanism should be in favor of websites like this existing and their authors getting respect.

Its complicated! ;-)

Anyway.

I think that forensics/debate/rhetoric have a bad reputation about LW that is not justified by carefully tracing out real pedagogical outcomes.

I think they teach a lot of skills that include researching in libraries, adaptively optimizing your speech production to the true incentives and the real audience, having fun, laughing, building an identity and some pride around your ability to know things and explain things and not be tricked, and also they grow your ability to judge a debate well.

Like, in policy debate, if you get a "Flow Judge" (whose stated judging philosophy is that they take notes like the conversation was a math equation, and then whatever the bottom line is from the math equation is how they will vote, regardless of personal feelings, common sense, or who was more personable) you're happy because the round will be insanely fun and anything could happen.

And if you get a "Communication Judge" (whose stated judging philosophy is that they will take barely any notes, and will simply apply common sense and vote for whoever they think seemed more prestigious and correct and better spoken at the end) you're sad because the round will be boring and who wins will be semi-random.

And then, a little later, when you get to be the a judge yourself, you know which kind of judge you want to be (one of the cool ones who can take really really high quality substantive notes even when the speaker sounds like this).

((That video has this hilarious moment where the kid is like "I don't know why we practice talking fast while we have a pen in our mouth" and you can SEE his "I notice I am confused" reaction happen in his eyes! <3 :-D

For reference, this is a training tradition in the field of rhetoric that goes way way way back. Its a not a great sign for the epistemics of the coach, but maybe its just fun (or a puzzle bomb in the curriculum)? This kind of training for that is better supported by evidence and mechanistic priors on like... muscles in the mouth and tongue n'stuff.))

I'm going to assume I've sold you on "academic debate is worth looking into" and pivot to applications now!

If you're in High School and thinking about colleges, I think it is likely to be a net positive in your life if you explore adding Policy Debate as second or third "major" (likely un-official) on top of EE or Physics or Math or something (or possibly in parallel with Philosophy).

Part of why I re-arranged my majors so much and eventually picked up a Philosophy bachelors (there were other reasons for allocating time that way... I'm not insane) is that Philosophy majors clearly punched harder in debate.

They could speak fewer words, more slowly, and still win a round with a Flow Judge!

They could cleanly identify categories of thought and dispense with 10 arguments of 4 sentences each that shared reliance on a broken mental category (that had been named in the First Negative Constructive (1NC) because they knew it would eventually be useful!) in like three sentences in their Second Negative Rebuttal (2NR). Destroying 40 sentences of gibber-jabber in the mid game, with 10 sentences placed in the opening game, plus another 3 sentences at the end when time is precious, is a powerup!

In terms of applying this in real life, with life planning decisions...

I don't think people "pick a High School" super carefully usually? But maybe there are parents reading this who want to buy a home near a school that will help their teenage kids? In that case, this ranking of High School policy teams has numerous teams from the same school, and those schools probably have solid policy debate programs ;-)

If you're in High School and thinking about college I think it is worth an hour or five to research the Policy Debate options carefully, before you visit any schools prior to applying to them.

For example, Bakersfield College (in central southern California) and Chico State (in norther central California) both had great programs 25 years ago (despite having no amazingly great reputation overall, as schools, and being easy to get into, and cheap), and they they are still on the list of community colleges with Policy programs) so they might still be amazing and worth a look? Depending on whether their coaches solved Succession Planning, anyway?

I can't easily google to get a good ranked list of undergrad institutions, for CEDA/NDT, but this list of all the different regional circuits in the US could help. The more tournaments a school hosts, the more likely it is to have a strong program with energetic coaches.

You'll probably pick a college based on lots of other factors, but if you at least look up which ones have CEDA/NDT stuff, then, when you visit, before applying, you could probably try to visit the debate program and talk to people there to catch a vibe and make an assessment? <3

[-]Screwtape3mo20

I suspect a skilled debater ( in the modern competitive debate format) would be a useful addition to the sort of Defence Against The Dark Arts thing I'm thinking of here.

My guess is the dislike for rhetoric or competitive debate around LessWrong is that it's often used to argue for incorrect conclusions. I haven't done competitive debate ^[1] before, but my understanding is you get handed the Pro or Anti position at random and have to do your best to argue for that conclusion? If that's the case, it does seem like a case of having ones bottom line already written. . . which is fine, for the goal of practicing trying to discern the true thing when someone's trying to convince you of a false one!

And in terms of general educational outcomes- I haven't done a deep dive into the subject, but you make a good pitch!

^{^}
Sidenote, I'm using "competitive debate" to refer to the thing college debate clubs get evaluated on, if I'm using the wrong terms let me know

[-]JenniferRM3mo120

Yeah, so there are quite a few different formats (I won't even mention all of them below) and they can evolve, and they work somewhat differently here, and "being honest in your advocacy" comes up in the debates themselves quite often (but the other side is, sort of necessarily, also argued whenever the debate "goes meta about debate itself" like that).

I. "SPAR"

In High School (when I was competing in individuals events for the team I eventually returned to and coached) there was one tournament every year that had a "SPontaneous ARgument" competition.

That tournament was the best, and lots of us loved to SPAR, and that was just like the thing you're imagining.

You don't even know what the question will be. Its usually something trivial like "Peanut better is objectively better than jelly" but it could be "Invading Iraq was a mistake" or whatever.

If both people agree on who wants to be pro/con they can, otherwise its a coin flip.

Amateur parent volunteer judges with no rhetorical training then judge on... random criteria?

It is, to serious undergraduate NDT tournament at a national Finals, what your first day at an improv class is to starring in Hamlet.

II. RESOLUTIONS SELECTED BY A NATIONAL COMMITTEE

In serious policy debate the community picks a RESOLUTION and then at a tournament you take the affirmative three times, and the negative three times. When you are affirmative, you pick ANY CASE YOU WANT that is an instance of a detailed policy consistent with the RESOLUTION to advocate.

At least that's how it works nowadays. Sometimes in the past there was poor resolution selection such that every round for the entire year was the same argument and it was terrible. The trick to a good one is to hit the zeitgeist a bit, but also to leave wiggle room for crazy cases.

1928 Resolved: That a federal department of education should be created with a secretary in the president's cabinet.

1943 Resolved: That a federal world government should be established.

1966 Resolved: That the federal government should adopt a program of compulsory arbitration in labor-management disputes in basic industries.

1985 Resolved: That the federal government should provide employment for all employable United States citizens living in poverty.

1999 Resolved: That the United States should substantially change its foreign policy toward Russia.

2007 Resolved: The United States federal government should establish a policy substantially increasing the number of persons serving in one or more of the following national service programs: AmeriCorps, Citizen Corps, Senior Corps, Peace Corps, Learn and Serve America, Armed Forces.

2016 Resolved: The United States federal government should substantially curtail its domestic surveillance.

In general, teams can run cases they truly believe in, or they can run "trick cases", or they can run centrally topical cases. These are more like dimensions? In modern times there's a whole other thing called "kritiks" which I'll touch on very lightly later. Affirmative cases can often be positioned in these three or four dimensions, but often push very hard into one specific dimension.

III. Example Trick Case: Bomb Russian Booze

I remember a trick case with Russia, where the affirmative plan was to send Navy Seals into Russia, and secretly bomb literally every large alcohol distillery (also all the warehouses with stores of vodka), and get away with it (no one would know who the bombers were), and so (the claim went) nothing would change except a massive increase in the price of hard alcohol in Russia, based on massive decrease in the supply, and thus a massive lowering of their rates of alcoholism, for a massive net benefit to Russian domestic wellbeing.

And this was, obviously, INSANE. But it kept winning! :-D

They had all this published recruiting material, that young men are often exposed to by ROTC recruiters, saying that the Navy Seals were essentially made of magic and could do anything and never be caught.

And they submitted it as strong direct written evidence of the ease with which the plan could be carried off, and bloviated about how "the American government would never lie to its precious young men in its recruiting material so either this is true or else the USG is evil... is that what you're really saying when you accuse our recent and clearly cited evidence of being overstated?? do you you lack patriotism?!?"

It was hilarious. People almost always laughed the first time they heard it. And (rare) Flow Judges and (common) Coms Judges could both appreciate the "technically clean logic" and (often) the fun/patriotic/cynical showboating.

IV. Other Example Cases: Drug Legalization & Dioxins

So, basically, if you wanted to spend your optimization pressure to pick an AFF case you really really believed in (like as a political posture?) even if it had shit evidence, and only one crackpot in one insane progressive magazine with a readership of 1000 was advocating it directly (and you weren't even doing what the crackpot suggested because that was a bad plan, but you were modifying it creatively yourself)... I mean... you could do that, but it would be tougher and maybe less funny?

And funniness, in my book, counts for a lot.

((I kinda "went authentic" my second year of actually debating, trying to pass off radical domestic drug reform in the US as a "demand side" attempt at a "foreign policy" adjustment towards a country with large "illegal" opium farms and cartels and stuff. It was something someone in their early 20s might foolishly think was wise? Arguably we were Brucing a bit in choosing to run it... we sometimes actually lost on AFF, which is not great.

In my first year we just went straight up the middle with a dioxin emission fix based on mandated medical waste incinerator smokestack filters paid for by government subsidies, and enforced by fines for refusing to accept the retrofit. Dioxins are basically not emitted in the US anymore, but back then they were. We won a lot. And it wasn't even a trick case! But people didn't laugh... just nod along and agree that this boring thing was boringly correct and the negative arguments were grasping at straws. The coach made sure we didn't change since we were beginners and etc etc.))

Often, teams would pick a case, and change it a little bit between every tournament and bring polished proposals and skilled defense to tournaments at the end of the year.

The hard and creative (and more morally ambiguous) part was Negative.

V. The Desperate (Deeply Theoretical?) Bullshit Negatives Need To Ever Win

Over and over and over (usually three times every tournament) you hear this AMAZING First Affirmative Constructive (1AC) and with shining rhetoric, a clean story, tidy evidence, and true confidence in the voice of the person speaking it, and...

...you want to stand up and applaud and say "this is a great proposal that America should do"...

...but instead you have to pull shit out of your ass somehow to say NO to it...

...like maybe: "NO! uh... because... technically it is a prima facie burden of the affirmative team to propose a plan with no vagueness, and here is a formal expert in rhetoric to back us up on this claim about how debate has to technically work if it is to be properly educational, and this plan is vague because... uh... [checks list of 20 ways a plan can be vague that was given to them by the advanced team that usually gets to at least the quarter finals in tournaments] it lacks a clear funding mechanism... and you, dear wonderful Judge who is very wise and full of common sense, you wouldn't sign a blank check, would you? wouldn't that be crazy? this is a prima facie burden! and it can't be fixed in the second speech! if they didn't have it in the first speech you know they're just making shit up and you can't trust them, and therefore they should lose and there's no two ways about it, and here are five independent reasons that the basic idea, if true, constitutes valid reason to give the ballot to us, all based on just the truth that their plan was vague <and so on>".

You actually give it A LOT more structure thatn that. Break "Vagueness" into an A, B, C, D set of claims, where the four lettered claims abstractly explain why NEG has to win. Give each letter several numbered justifications... Also, you make four other half-crazy arguments, each of which attacks the AFF case from orthogonal directions from "Vagueness".

Then (on the Vagueness position) the AFF later says:

(1) you made an ad hominum in your accusation, about them not being trustworthy

(2) and yes do too they have a price number, and they just baldly assert <a made up price> with no evidence at all, and

(3 & 4 & 5) they make some other rebuttals, but they don't don't refute EVERY independent line of argument through your entire plan vagueness scaffold.

So in your counter you magnanimously grant almost everything they said, except for the totally made up price which is obviously made up. Which you reiterate is too late so shouldn't count for educational reasons, and also NEW 1 its a fake number with no evidence, and also... <three more new numbered arguments>.

And you point out that you made more and stronger arguments that they ignored, and reiterate the A B C and D of the Vagueness argument...

...using only A2, A4, B1, B5, C2, C3, and D3, D4, and D5 as voters.

You point out that you now have 2x2x2x3==24 different paths to retain the overall Vagueness Position in a form that could destroy AFF! <3

(These 24 paths to victory are a shorter and smaller version of what you originally said! You said more before... via speed reading a generic Vagueness Template that you've practiced speed reading the pro forma parts of.)

And then you remind the judge that the next time the AFF talks it won't be a constructive, and they would have made their strongest arguments, and if they make new arguments it wouldn't be fair, because you probably could have made strong creative counter-arguments to those new arguments, but you won't have the time or legal allowance for that (which might be why they would have stuck arguments in that were weak in the long run but strong initially)...

...and so anything else they make up later would violate the "educational spirit" of "the noble practice of policy debate", and so, sadly, despite the case possibly being great in real life (unless either of the two other independent anti-affirmative voting arguments goes through) the judge will have to Vote On Vagueness And Vote Negative.

And then you watch the judge's facial expressions carefully during this as they look down at their notes and write what you say... or whatever the judge does. Its the judge. The judge is god for the round. You want the ballot, right? The judge controls the ballot.

VI. Subservience To The Judge Is Good Strategy

Depending on the judge(s)'s nods or frowns, maybe in an even later (even shorter) speech, you might concede or silently drop or spend only 20 seconds half-assing the Vagueness position at the very very end?

Or you might hammer it super hard in an eloquent minute long rant explaining why this is ONE OF TWO very very strong and very very independent reasons to Vote Negative.

And you spell out a potted but plausible history of the debate so far (that everyone has notes on and it just happened but it was fast and you're trying to crystalized it in people's minds) summarizing all the rhetorical flubs of the AFF team who had a BURDEN OF PROOF! And not-at-all-fakely emoting about how the AFF burden of proof is so important... right now... (until you're on AFF in the next round, insisting that stock issues are fake bullshit in general, and Vagueness isn't even one that people normally cared about)...

In finals, there are three judges!

Based on facial expressions these three "gods" might buy different things, so there are entire lectures you'll get from the coach in the five hour van ride to yet another college for yet another tournament, on how to pick what to drop and what to emphasize so that you can try to get at least two of the three judges in the final to vote for you, even if they vote on different issues.)

And then sometimes all the prep work and all the fallbacks, and all the combinatorial optimization gives you a weaselly win as a negative?

And you win against a naively "real world very strong case" where the Affirmative just were kinda lazy about researching the cost of their plan enough? Or not?

It depends.

VII. The Bullshit, Emphasized Above As Silly, Is Actually Often Educational

However also... it overall... these practice really are, often, "educational in general"! <3

The vagueness bullshit might work in the second tournament of the season, but in the fourth tournament of the season, with the AFF running a newer and sexier version of their case, and they will have changed the wording slightly (at no time cost) to mention a price in their Official Plan!

And it will be backed by evidence they didn't mention, but now have in their laptops or box of papers, that they can quote (but only if you're fool enough to not notice the slightly different words and try vagueness on reflex, rather than from actually listening and deciding what Negative attacks to try, on the fly, in the round) and the evidence will be based on an expert they found in some library article, after haunting the library for a day.

Negative teams rarely win the same way twice against the same Affirmative team, and by the end of the year the same teams will often have been in the finals against each other two or three times, arguing the same AFF case as always (but now very strong) and "yet another crazy way to attack it" (like a kritik maybe?).

But anyway.

If you go with a stock issues frame, and try Vagueness, on NEG, that first time... (early in a season, with a strong generic theoretical counter-argument to a half-assed AFF case) you might win, especially in High School...

...in college anarchoprogleft types might try to accuse you of being logocentrically racist if you naively try to use stock issues, and anarchoprogleft college judges might agree with that? College rounds can be insane or not, in hundreds of different ways ;-)

I lost to "Kant"... twice. Never got a 3rd rematch.

I lost to "deep eco feminism"... twice and then beat it the third time and had "Cara" (who was in her late 20s, and finally getting a degree to make something of her life, and was intimidatingly confident, and made finals in like half the tournaments I saw) ask for a copy of my citations and evidence at the end, which made me felt super proud. Then she stopped running "deep eco feminism"! Which I thought was high integrity of her!

Anyway. If things are working well overall, Winning As Negative gets harder and harder and harder over the course of the season.

Because there is feedback!

Because it is at least half real, and every 90 minute debate at a tournament gives the AFF yet another real life, adrenaline filled, actual redteamed, OODA loop's worth of information on how to make their advocacy, that they get to choose and refine (within the arguably vague bounds of the annual resolution that is stable all year) stronger against arbitrary attacks :-)

VIII. Thoughts On The Flexibility Of The Ontology Of Traditional Debate Norms

Mostly here I'm talking about specifically undergrad level "CEDA/NDT debate" and its echo at the High School level which is called "Policy Debate".

Anyone can make up anything they want, in terms of a group of people who meet to argue and vote and award prizes and whatever.

"Debate" in a fully general form, is nearly infinitely open and flexible. It doesn't have to be two people vs two people. It doesn't have to have a yearly resolution (in High School sometimes they change it halfway through the year).

It doesn't have to have five minute constructive speeches with two minutes of cross-examination at the beginning. It doesn't have to have ballots that say either AFF or NEG. It doesn't have to have "speaker points" over and above that to help break ties in how the seeding process for finals should work.

Debates have been happening for millennia and you can do them however it seems correct to you to do them. They are deeply "existential" rather than "essential" when the judges can write anything they want on a ballot, and your job is to get people to grant you ethos, and feel "<YourName> wins" is true of what just happened, and express it somehow. If people are truly good, it can work. If people are bad, then... maybe the methods can save it? Maybe?

The debates exist... that existence includes a competitively collaborative search for the meaning of the competitive practice, with winners and losers, internal to the practice...

The form that exists now agrees, mostly, that it should be able to regenerate itself from inside its own logic, but the forms we have are deeply deeply tested and not everyone will understand all of why things happen and particular way. And these forms are part of continuous traditions going back to the 1920s that are still culturally resonating, which I think is kinda cool <3

[-]Shankar Sivarajan3mo20

If you watch some modern competive debate, you might have a better understanding of why it's disliked. The Cross Examination Debate Association's National Championship is the most prestigious of the US college debate tournaments, and the 2014 finals was widely circulated as an illustrative example of the current style.

[-]Screwtape3mo20

I'd be interested in hearing more about the different kinds of judges? I just googled Flow and Communication Judge for debate and got a bit about diagramming the flow of a debate.

(You can also say "go read X website" or "the consensus is pretty good, random googling will get you basically the right answer." Reading up on debate rules has been added to my backlist but you strike me as the kind of person who might enjoy explaining it :) )

[-]JenniferRM3mo20

"Com judges" might not exist any more (or might be called something different)?

I think the current meta focuses on "K" vs "not-K"? (Roughly, the "K" people only want to debate philosophy, and they sort of abuse the Policy Debate platform (using the logic of policy debate) to try to undermine doing policy debate within a Policy Debate tournament because maybe "better policy outcomes" would happen "in real life" if people stopped having policy debates and debated philosophy instead.)

Coms judges probably do exist, but maybe not "by that name" anymore because they are somewhat timeless? They come in two archetypal flavors:

(1) the default you'd expect of a private school PMC (Professional / Managerial Class) parents who volunteer to judge when their kid's coach asks for parent volunteers to enable the coach to run a tournament, where those parents will act and judge like normies, and will predictably reward "the appearance of articulate prestige" in terms of PMC cultural standards and...

(2) church lady debate coaches who think the naive reactions of those PMC parents are essentially correct and timeless and wants to consciously "teach to that test", and who also personally "judge to the theory that teaches to that test" when they are judging... in a high church way that connects back to latin phrases like "post hoc ergo prompter hoc" and "status quo" and and "ad hominum" and certain ways of organizing policy proposals based on "prima facie stock issues" of some kind (the lynchpin of all of them being the stock issue of "Solvency" where the plan had better at least pretend to be positively likely to positively work to fix some problem, and the AFF has to prove this or else they are a bad AFF). Some of them get very defensive, and sort of refuse to flow, and will pantomime "ripping up the ballot" if a debater starts to talk really fast.

Sankar's comment is indicative of the lay public's attitude, that church lady com judges respect.

If you watch some modern competive debate, you might have a better understanding of why it's disliked. The Cross Examination Debate Association's National Championship is the most prestigious of the US college debate tournaments, and the 2014 finals was widely circulated as an illustrative example of the current style.

((

Arguably, however, the NDT year end tournament of the merged CEDA/NDT circuit system is more prestigious that the CEDA year end tournament? And former NDT regional circuits tend to be in the South, and are more rhetorically traditional, and less "post-modern" (which some peopledislike for separate reasons). The distinction here isn't about flow vs not-flow, or stock-issues vs not-stock-issues, but rather K (post modern? philosophic? anarchist? woke?) vs not-K (modernist? policy-centric? archist? conservative?).

))

Flow Judges... flow!

And rely on it heavily to decide their ballot. (There are subtypes. I will not enumerate them <3)

Here is an example image (sauce here) of a flow for ONE (or two???) position(s) (maybe an entire case?!), that was discussed substantively by skillfully-intellectually-organized speakers all of whom remembered and addressed each other's previous points coherently in a skillful way, to make flowing easy (often maybe using verbally numbered arguments based on everyone taking similar notes in similar ways), in seven sequential speeches (which we know, because there are seven columns):

We have sub-issues going horizontally across the page.

Interpreting this a bit... "H" with a circle at the top left in black ink probably stands for "Harms", which often shows up in the first affirmative constructive (1AC) proving the stock issue of "Harms". Something has to be wrong with the status quo. If the status quo ain't broke, it shouldn't be fixed. The AFF has to make people want change before they propose change. All of this is latent in "H" in a circle... and it has something to do with something being "untraceable". Everyone heard this in the round of course. That's why they can get away with so much shorthand. They might have a whole logic loop in working or audio memory and be able to know what is being talked about if someone says "on the traceability argument in Harms, our response to their response saying <blah blah> is... NEW X".

Whatever the arguments are, it involved a cited substantive claim, published in 2010, by "Bachi" (whoever that is).

((Google scholar offers no insight when I search for something, fwiw. This is bad. A KEY FUNCTION of these notes should be to enable people to do opposition research on key ideas "out in the literature" and get a better picture of reality thereby, and win debates about reality thereby. You want to be able to aumantically hear an argument, and come back later at a new tournament magically knowing the backstory of what was claimed as a posterior, and via scholarship, this can happen! The citation and keywords, for later research, are like the NUMBER ONE THING at least one of the two people on your two person team should be capturing.)

Back in the image, where is says "Brower '08" that means someone with the last name of Brower was quoted ver batim based on something published in 2008. The substance of what was said is going to be in everyone's working memory, but the gist of it is "The economy going down somehow causes something to do with satellites" which we get from "econ V -> satellite".

The red ink is almost certainly the negative rebuttals, which attacked various subarguments line by line with the first such speech not "reasoning" that much, but mostly just reading four pieces of (probably case specific) counter-evidence. (When that happens to you, as the AFF, your stomach often sinks, because you might drop an AFF round, even though you picked the ground upon which to fight, which is a huge advantage.)

The blue and black ink are probably all affirmative speeches. The blue was likely written very fast, in the middle of the round, while speeches were happening.

The black ink was probably "pre-flowed" and written in advance of the round by the affirmative team, to save time, so as not to waste seconds during the round by taking notes on a set speech they already optimized the shit out of (and might know by heart).

This might be a sloppy/weird flow? It has the flavor of something fake.

Normally you'd put each issue on its own piece of paper or file, or tab in your spreadsheet on a laptop, because you don't know how much it might explode. This is "less real". Good for pedagogy maybe?

See the "2 Blackouts '10" in black ink near the bottom?

To the right of that is an "S" with a circle in red ink, which usually means "Solvency" which is a common stock issue that is different from "Harms" (and should be on different paper??). Also, on the far left in the same row there's red ink that simply says "no solvency" which suggests that the entire horizontal sequence of debate is "about Solvency"... and if this is true then it suggests that the negative had the last word on the subject (and the affirmative might have dropped it)!

If the affirmative really did drop Solvency, then they basically lost the debate.

Therefore a flow judge would probably give the ballot to the negative (unless some other piece of paper also exists, about what standards to use for a ballet, and somehow that debate resolved in "Solvency doesn't matter anymore for <reasons>"). A Flow Judge would give NEG the ballot even if the flow judge was a flat earther who doesn't believe in satellites but does believe aliens will rapture us all before anything being talked about in the round actually ever happened... you can be a Flow Judge AND be crazy... but being a flow judge immunizes you some from having wrong beliefs so long as you track syntax and are in a half-honest environment ;-)

So we see here, from the example flow: NEG probably won in some "objective" sense.

And a flow judge would notice and give NEG the ballot?

And this is why debaters often prefer to be judged by flow judges... it puts the speakers legibly and clearly in control of their own destiny within a round, tightening the OODA loop and the lessons it can teach :-)

[-]JenniferRM3mo80

Minor counter-point here...

Let's talk about the controlled setting of a dojo for a moment, because I think that’s actually a very important qualifier.
If I punch you in the nose while you’re standing at the bus stop or while you’re walking through the halls of your high school, I’m a jerk and this is not how martial arts dojos work. It is true that the bus stop or your high school are the places you will want the self defence skills!

I had a co-worker who was in a kinda half-made-up-seeming dojo for "ninjitsu", and in one of his belt tests the test was whether or not the teacher could "touch his back with a symbolic dagger" (it was a long time ago and my memory is fuzzy, but I think maybe symbolic poisoning of food or drink was also valid?) at any time, waking or sleeping, in any place, public or private, for an entire multi-day window. (He failed the first test, and was hypothetically interested in trying a second time when he and the sensei both had the time for it.)

There were rules to keep it from hurting any third parties or looking too weird, but it sounded (1) super fun and (2) very expensive in terms of the time cost for the person being tested and the person testing (if they actually make multiple attempts or multiple attempts per day).

I have no idea how to adapt this to real life exactly, but it was so exceptional and fun-sounding that it seems worth a mention here ;-)

[-]JenniferRM3mo80

A possible answer to this question is: sort of YES! (But the practice is I know about from experience... does not involve a pair of people, where the liar is known, and the thing they're lying about is clear. It is a group practice with some fraction of liars.)

Has any rationalist practice ever involved pairing up to practice being lied to and responding to lies?

There's a wiki entry on Paranoid Debating which might have been mentioned for the first time ever in Eliezer's 2009 "Selecting Rationalist Groups" as part of the Sequences, and the highest relevant substantive post is a writeup about it where a group in London playing the game ~11 years ago.

Its not perfect. Its kind of like playing Avalon Mafia mixed with a game about Fermi estimates. Knowing what I now know (like, for example, having played Avalon) I think I could design a similar game that was more fun and more focused on giving faster feedback on specific subskills.

[-]Screwtape3mo40

Knowing what I now know (like, for example, having played Avalon) I think I could design a similar game that was more fun and more focused on giving faster feedback on specific subskills.

I would be interested to hear more thoughts on this!

[-]JenniferRM3mo20

Huh. I feel like this could be a whole subreddit or wiki on its own? So open ended! Its not like there's a uniquely correct response to it.

For myself, lately, the thing I've gotten the most mileage out of is "p-beauty contest played by, 2, 3, 4, or maybe 5 players".

In a two player game, you simply bid zero if you can do math and want to win.

That's all there is to it.

But it turns out a lot of kinda smart people need that math lesson!

Once you have three players who have all had that math lesson... then it gets interesting because it turns out a LOT of people hate the idea of generating a real "common knowledge certificate of mutual awareness of mathematical rationality recognizing mutual awareness of mathematical rationality"... or something?

So people will throw games! Or wax philosophical about their objections for 30 minutes before playing... or all kinds of stuff.

In my entire life, I have only seen "four humans all bid zero in the very first round" once.

[-][anonymous]3mo70

Has any rationalist practice ever involved pairing up to practice being lied to and responding to lies?

Yes, actually (at least in theory; I'm not aware of Turntrout's proposal getting a ton of traction in practice, but maybe it has).

[-]Screwtape3mo60

I've run at least one event like that, based on Olli's writeup! I wound up adding in 'traitors' (akin to the bad guys in Mafia) who tried to distract and misdirect the rest of the group.

It's not a bad start and I expect I'll run more like it, though it's a bit drier than my current ideal version.

Thanks for linking it though- if I hadn't seen it, this would have been something I'd want brought to my attention!

[-]Said Achmiz3mo50

See also Gwern’s “Fake Journal Club: Teaching Critical Reading”. (This is not quite the same sort of thing being described in the OP, but it’s in the same spirit.)

[-]Kaj_Sotala3mo60

My recent thought that talking to LLMs might help train resistance to manipulation by humans seems relevant here.

I’ve seen people express views like “LLMs are causing some people to have psychotic episodes, better not to talk with any LLMs so my brain doesn’t get hacked”. I suspect this is not the best approach. If an LLM might “hack your brain”, that’s because you have existing vulnerabilities that a human might exploit, too.
The alternative perspective is that LLMs are a relatively safe ground to practice with, safer than many humans. You can write one kind of message, then go back and edit that message, doing controlled experiments on how it reacts to changes in vibes. You can take messages it has written late in the conversation and see how it reacts if you ask it critique its own writing in a fresh conversation. Or you can stay in the same conversation and completely change your own stance, and witness when it turns around to match and when it pushes back.
And despite all of that, you might still get pulled in. I’ve had the experience that, despite thinking I roughly know how LLMs work and what to look out for, I’ve on occasion still ended up trusting their answers more than I should have. But experiencing that and noticing it is better than avoiding LLMs entirely and never having the chance to notice it!
You don’t train your emotional brain with logic, you train it by giving it the chance to experientially notice contradictions in its expectations. Whatever emotional rules determine my extent of intuitive trust in an LLM arise from the emotional rules that determine my intuitive trust in a human. And it feels to me like the lessons I learned from my interactions with the LLM do generalize to how I feel about trusting humans.
With an LLM, you have a degree of control over the conversation that you’d never have with any human. Even if you get sucked in, the LLM is not acting strategically to exploit you, it’s just operating as an unreflective stimulus-response machine. A potential abuser who noticed that you were stepping out of their frame might react by intensifying their assault and trying to force you back into it. An LLM that’s particularly strongly within a particular frame might still push back on attempts to change it, but none of that will carry over to the next conversation you have with it. And it might also happily go “yes boss!” and change tune right away in the same conversation.

A conversation with an appropriately-prompted LLM would have many of the dojo-like properties, such as only happening when you explicitly choose to engage with it. Maybe you could prompt an LLM with something like "you are a fervent believer of X, convince the user of the truth of X" and then later prompt it with "you are a fervent believer of not-X, convince the user of the truth of not-X". This could help reduce the risk of being persuaded by untrue things, since you would experience the persuasion attempts going in both directions.

Analogous to first aid, you could also afterward run any conversations you had through another LLM instance prompted with something like "point out any fallacies and incorrect claims in what the persuader-LLM said in this conversation".

[-]Screwtape3mo20

I could see LLM conversation being decent repeated practice for this, and I can envision putting an LLM in a packaged format in a much more accessible way. (A reviewer for this essay asked me if I had considered making a Meetup in a Box version of practice for this, and the main reason I hadn't was I'd need to put a trustworthy manipulator in the box!)

I'm a bit leery of the LLM not being good at calling a halt and patching one back up afterwards, but maybe this is achievable with the right prompting.

[-]jimmy3mo61

This reminds me of the what is probably my favorite Overcoming Bias post, "My Favorite Liar".

[-]River3mo30

I don't know much about LARPing. I do regularly attend a Blood in the Clocktower game at a rationalist house. Do you think that is more similar to Werewolf or LARPing?

[-]Screwtape3mo20

Werewolf, but it's moving a step in the LARP direction.

I've only played Blood on the Clocktower once or twice, and this gradient isn't something I'm confident is definitely there just one I suspect matters. Also, "LARP" is a pretty broad category and if I had better subgenre words to describe the ones I'm talking about I'd use 'em.

Some tactics I've seen employed in LARPs that I think wouldn't work as well in Blood on the Clocktower:

Threatening a third party into supporting your story.
Being really persistently annoying when pushing forward a course of action. Just do not let up. Bring up your preferred policy at every opportunity
Get a group to combine resources to solve some real common problem, then embezzle or misappropriate resources.
Create a false consensus. "Everyone agrees on this except you. Can you move past this already?"

If you have ~30 people, a half-dozen rooms, and a ruleset that people know they don't know all of (imagine playing Blood on the Clocktower again, but for this round the Storyteller doesn't have to give the players the list of roles) there's a lot more angles by which someone can pull epistemic shenanigans.

If I was going to move Blood in the Clocktower a step towards being better training ground for this. . . I think the main thing I'd change is voting in a big circle where you can all see each other. Cornering someone one on one (or better yet, with you bringing confederates so it's 2:1- I'm impressed this works as well as it does for conversation!) and pressuring them into casting their vote now, that's the obvious change. A bigger deal is that trying epistemic bullshit in front of an audience that's all looking for the malfeasance is a riskier play.

But BotC has a decent amount to recommend to it. There is more modeling of what world you're in given various claims. The chance to wander and chat with each other independently allows for some subterfuge.

[-]Kabir Kumar3mo3-3

instead of becoming good at lying, you can get good at comedy instead. same skillset, less Dark

LESSWRONG
LW

LESSWRONG
LW

82

The Dark Arts As A Scaffolding Skill For Rationality

82

82

Vaccination

Lying

LARPing

I.

II.

III.