Crossposting Eliezer's comments.

Thread between Eliezer and Vitalik

Eliezer Yudkowsky replying on Twitter:

This applies way beyond mere ethics, though! As a kid I trained myself by trying to rationalize ridiculous factual propositions, and then for whatever argument style or thought process reached the false conclusion, I learned to myself: "Don't think that way."

Vitalik Buterin:

Indeed, but I think ethics (in a broad sense) is the domain where the selection pressure to make really powerful galaxy brain arguments is the strongest.

Outside of ethics, perhaps self-control failures? eg. the various "[substance] is actually good for me" stories you often hear. Though you can model these as being analogous, they're just about one sub-agent in your mind trying to trick the others (as opposed to one person trying to trick other people).

Eliezer Yudkowsky:

Harder training domain, not so much because you're more tempted to fool yourself, as because it's not clear-cut which propositions are false. I'd tell a kid to start by training on facts and make sure they're good at that before they try training on ethics.

Vitalik Buterin: Agree!

Eliezer Yudkowsky:

Eg:
Argue: That a saltshaker is Jesus Christ.
Rationalization: Gurer jrer n ybg bs ngbzf va Wrfhf'f obql naq fbzr bs gurz ner ab qbhog va guvf fnygfunxre, tvira Nibtnqeb'f ahzore naq gur ibyhzr bs Rnegu'f fhesnpr.
Lesson: Look at what your thoughts did there; never do that.

Eliezer Yudkowsky used ROT13 to hide an example Rationalization.
After thinking about it yourself, hover over the box below, to reveal his example.

Related writing by Eliezer

Skill: Anti-rationalization (1): Prevent your mind from selectively searching for support of only one side of an argument.

Subskill: Notice the process of selectively searching for support, and halt it. Chains into "Ask whether, not why", or into trying to unbind the need to rationalize. Level 1, notice the active process after performing a conscious check whether it's running; Level 2, detect it perceptually and automatically; Level 3, never run this process.

Exercise material: List of suggested conclusions to rationalize. I'm not quite sure what the prerequisites here would be; one of my own childhood epiphanies was finding that I could argue reasons why there ought to be a teapot in the asteroid belt, and thinking to myself, "Well, I better not do that, then." I suspect the primary desideratum would be more that the topic offers plenty of opportunity to come up with clever rationalizations, rather than that the topic offers motivation to come up with clever rationalizations.

Exercise A: Pick a conclusion from the list. Come up with a clever argument why it is true. Notice what it feels like to do this. Then don't do it.

(This is distinct from noticing the feeling of being tempted to rationalize, a separate subskill.)

Exercise B: In the middle of coming up with clever arguments why something is true, stop and chain into the Litany of Tarski, or some other remedy. Variant: Say out loud "Ew!" or "Oops!" during the stop part.

Exercise C: For a conclusion on the list, argue that it is true. Then "Ask whether, not why" - try to figure out whether it is true. Notice the difference between these two processes. Requires a question whose answer is less than immediately obvious, but which someone can find info about by searching their memory and/or the Internet.

https://www.lesswrong.com/posts/f4CZNEHirweN3XEjs/teachable-rationality-skills?commentId=F837zHgkrTA2rqusn

I learned at a very young age not to be impressed by the sleight-of-hand that assembles a clever-sounding argument for why there might be a teapot in the asteroid belt, even or especially when it was my own brain generating the argument; that was where my journey began, with the search for a kind of thinking that couldn't argue impressively for any conclusion.

https://www.reddit.com/r/rational/comments/dk4hh8/comment/f4po393

When I was a kid I trained myself by coming up with the best arguments I could for propositions I knew to be false, then contemplating the arguments until they no longer seemed persuasive. I picked ridiculous targets.

https://x.com/ESYudkowsky/status/1900621719225987188

See Also: Against Devil's Advocacy

AI risk is not common knowledge. There are many people who do not believe there's any risk. I really wish people who make arguments of the following form:

From the inside view, this seems reasonable. From the outside view, however, you basically end up with this:

<meme of NPCs all thinking, EVEN THOUGH I'M IN AN ORGANIZATION THAT'S ADVANCING CAPABILITIES FASTER THAN SAFETY AT A 30:1 RATIO, I'M THE GOOD GUY ON THE INSIDE, IF IT WEREN'T FOR ME IT WOULD BE 40:1

would acknowledge this fact. It is simply not true that everyone in frontier labs thinks this way. You can ask them! They'll tell you!

It would be nice if we were just in a prisoner's dilemma type co-ordination problem. But when someone is publicly saying "hitting DEFECT has no downsides whatsoever, I plan on doing that as much as possible" you need to take this into consideration.