Epistemic status: Exploratory. I can't tell if this is really prevalent or if I'm just annoyed at how often it happens around me.

I.

Did you know you can just buy blackbelts? It's true! Go online and take a look, they're about ten dollars. 

Think about that. The black belt is a symbol of skill in many martial arts, and while the exact degree of skill it implies varies, thousands of people around the world who have studied for years still don't have one. Many an ambitious or dedicated student has entered the dojo vowing to work hard and someday get a black belt, or spoken in impressed and awed terms of the senior students who have theirs. And that can be yours for five minutes of Amazon shopping!

Of course this is nonsense. Nobody thinks it's the fabric that's the important part of the black belt. That's absurd, practically a reductio ad absurdum, a Goodhart's Law gone past plausibility into parody. And yet I keep running into people who seem to try things just as silly.

I run Calibration Trivia sometimes. It's like pub trivia, where you try to answer questions about miscellaneous details of the world, except you also try to give an answer for how confident you are that your answer is correct. It's easy to explain, it's easy to run, and if you do it regularly you can start training some calibration and a good felt sense of what it's like to be uncertain. In the martial art of rationality, calibration trivia may be our basic punching drill. 

And every single time I run it^[1] I get some clever fellow who points out that they could answer "I don't know" or "somfadsoifm", put 0.0001% chance they're right, and wind up with the best calibration score in the room.

That person is totally right! And yet this would be pointless and everyone knows it.^[2]

It's like going to the gym with a car jack and claiming you can lift five hundred pounds because you put the jack under the dumbell. It's not even like playing calibration trivia like this would fool the rest of the room into thinking you were impressive: I explain at the start that I'll show both the number of correct answers and the Brier scores side by side. Having your name next to  0 out of 30 correct answers and 99.99% calibrated means you don't know any of the answers.

II.

I'm not complaining about people doing weird munchkin moves to get the things they care about in ways that ignore parts of the normal process. 

Copying and pasting code you found online (or more realistically these days, asking an LLM) isn't buying a blackbelt. Often you're not trying to appreciate the sublime beauty of software engineering, you're just trying to get that script to work and upload those files. 

Talking a big game about some virtue — donating to charity, being honest, tolerating different views from yours — in order to reap the social benefits of being a virtuous person, then not actually practicing the virtue, isn't buying a blackbelt. There's a thing you want that you have a chance of getting: the acknowledgement and adoration of your peers. 

Using glitches or cheats in a videogame in order to win isn't buying the blackbelt if you want to see the cut scenes of the story with less work, or you just like watching the explosions when you blow up the entire enemy army with a single button press. Age of Empires II, a videogame I loved growing up, was mostly about medieval armies fighting with swords and arrows but had a cheat to give you a sports car with a machine gun. I loved driving it around blowing things up. Still do once in a great while. I am in some ways a very simple man: fire is pretty.

Even publishing a blank paper and calling it The Unsuccessful Self Treatment of a Case of Writer's Block isn't buying a blackbelt. Sure, it's obviously not advancing the repository of all human knowledge, but it's funny and it made people laugh.

People can even disagree about what goal we're pursuing! Teachers whose goal in assigning homework is to impart an appreciation for the sublime beauty of software engineering and students whose goal in doing homework is to finish in time to prep their D&D campaign later that night have different goals. The student isn't buying a blackbelt, they're just not on the same page with the teacher here. 

The problem I'm pointing at is when the person doing it has mistaken "I found an edge case" with "I have achieved the goal."

III.

People in the rationalist community contains people who are occasionally proud of their cleverness in finding some loophole, regardless of whether exploiting the loophole would actually get them what they want.

This is annoying to me. It's annoying both because I get tired of explaining the pointlessness of it every time I run Calibration Trivia, sometimes to the same person again, and because the people doing it are misdirecting their energy. They aren't actually getting the thing they want, just a poor pica version of it. I'm spending energy getting them back on track and they're spending energy getting told no.

My best advice I've come up with for these circumstances is to think one move further ahead. What are you about to do, and what do you think will happen next? If you're trying to get some kind of social acclaim, do you think other people will admire what you're doing? If you're playing a board game to win do you think the judge is going to accept whatever weird loophole you're talking about, or rule against you as soon as your opponents call the judge over? 

(And if you're going to accuse someone of lying because they said something imprecise or idiomatic, and you plan to make a big deal of this and raise a stink over the untrustworthy nature of the other person, do you think observers are going to think you're in the right once they look at what both people said? Or at least that's what I want to say, but that particular strategy has proven surprisingly effective in my observation! Not a perfect long term strategy, to be clear, but it has more legs than I'd have thought possible when I was young and innocent in the halcyon days of 2020.)

The surface level behavior of spotting gaps can be useful in certain stages of some projects. QA testers are a beloved part of a good software team, and they're engaged  in this kind of thing all the time. But QA testers know why they're doing it.

Please don't take this as saying I generally don't want you to point out an important gap in something I'm working on. 

But maybe let this be one more drop in the ocean, trying to raise the sanity waterline in one very small way?

1. ^{^}

   Is this literally true and it's every time? I can't prove that. What I can prove is that I've got a bunch of index cards with my notes from Calibration Trivia tests that say variants of 'yep, someone tried the calibrated for wrong answers thing again, it was ____ this time.'

2. ^{^}

   Maybe they're trying to helpfully point out a fix to the scoring rules? This is true in some cases. I'm pretty sure Maia in that comment is trying to helpfully point out a complaint people had about the activity, and she's musing on ways to fix it. In a couple of local cases it really didn't seem like the local person at my meetup was trying to be helpful, since they had a habit of raising objections the majority of the times we tried any kind of rationalist practice in a dismissive way.