Seeing you write about this problem, in such harsh terms as "formerly-known-as-rationality community" and "effects are iffier and getting worse", is surprising in a good way.
Maybe talking clearly could help against these effects. The American talking style has been getting more oblique lately, and it's especially bad on LW, maybe due to all the mind practices. I feel this, I guess that, I'd like to understand better... For contrast, read DeMille's interview after he quit dianetics. It's such a refreshingly direct style, like he spent years mired in oblique talk and mind practices then got fed up and flipped to the opposite, total clarity. I'd love to see more of that here.
To me, doing things because they are important seems to invite this kind of self-deception (and other problems as well), while doing things because they are interesting seems to invite many good outcomes. Don't know if other people have the same experience, though.
Maybe the most actionable path is to try influencing employers. For example, Google doesn't require college degrees for some jobs, but there's still a difference between not requiring degrees (but still preferring candidates with degrees) and ignoring degrees. One phrase to watch for is "degree or equivalent work experience". I'd be happier if Google's hiring process officially ignored degrees, and the same for other tech companies. But I don't know if anyone's campaigning for that kind of thing.
Your question is pretty abstract, so my answer will be abstract too: if you're ready to own the negative consequences of your action should they arise, then take the action, otherwise don't. Whatever arguments you read here, they won't be a valid excuse for you later, it's on you.
So we can continuously raise the required stakes for each wager, until either (1) the market approximately clears
I think the resulting odds won't reflect the probability of anything, because they depend a lot on whether Alice or Bob is more risk-tolerant (=rich).
Also, it seems to me that your scheme works best for yes/no questions. For anything more complicated, Alice and Bob can cooperate to mislead Judy, which is especially scary in case of AIs. I'm not sure how to fix that problem: it seems to require a way for a non-expert to check the work of a malicious expert, not just adjudicate between two experts.
Yeah, I made a pointlessly longer calculation and got the same answer. (And by varying the prior from 0.5 to other values, you can get any other answer.)
One of Eliezer's favorite writing tools is framing things as a two-sided conflict: atheism vs religion, MWI vs Copenhagen, Bayes vs frequentism, and even when presenting his views about AI he was always riffing off the absurdity of opposing views. That's fun to read and makes the reader care about the thing. I think it worked on us for basically the same reason that criticism-as-entertainment works.
I'm not Scott, but I think he's arguing against using force to violate rights (kill people or shut them up). He's not against using force to defend rights (when someone tries to kill you or shut you up).
I think I see another drawback of these kinds of techniques: when someone criticizes your thing, your first thought is "let's analyze why the person said that", rather than "wait, is my thing bad?" It's worrying that the thing you're defending happens to teach that kind of mental move.
For some reason it's not as annoying to me when you do it. But still, in most cases I'd prefer to learn the actual evidence that someone saw, rather than their posterior beliefs or even their likelihood ratios (as your conversation with Hal Finney here shows very nicely). And when sharing evidence you don't have to qualify it as much, you can just say what you saw.