Compare the following two arguments.
- E. is described by the following axioms
- Therefore under E. The square of the longest side of a right angle triangle is equal to the sum of the squares of the remaining two sides.
- All men are mortal.
- Socrates is a man.
- Therefore Socrates is mortal.
Naively, the second argument seems tautological, whereas with the first it's much harder to tell. Of course in reality the first argument is the tautology and the second argument is the more dubious one. The phrase "immortal man" doesn't seem contradictory, and how do we know Socrates is a man? He could be an android doppleganger. And the first argument's conclusion completely follows from euclid's axioms. Euclid and 100s of other mathematicians have proved that it does.
The arguments that change what we think, and the arguments that would change what a logically omniscient bayesian superman wielding Solomonoff's Lightsaber thinks are not very tightly correlated. In fact, there's a whole catalogue dedicated to finding out types of arguments we're persuaded by that a bayesian superman wouldn't be, the logical fallacies. But it's not just argument structure that causes us to lose our way, another factor is how well the argument is written.
People are more persuaded by essays from Eliezer Yudkowsky, Bertrand Russell, Paul Graham or George Orwell than they would be from a forum post by an average 13 year old atheist, even if they both make the exact same point. This threw me for a bit of a loop, until I realized that Eliezer was pitching to bayesian supermen as well as us mortals. How well stylish writing correlates to the truth compared to unstylish writing is nowhere near how much we are persuaded by stylish writing compared to unstylish writing.
There's also the dreaded intuition pump, I think the reason it's so maligned is because it makes things much more persuasive without making them any more sound. A well chosen metaphor can do more to the human mind than a thousand pages of logic. Of course, we *want* intuition pumps for things that are actually true, because we want people to persuaded of things that are true and more importantly, we want them to be able to reason about things that are true. A good metaphor can enable this reasoning far more effectively than a list of axioms.
The problem lies in both directions, we aren't always persuaded by cogent arguments, and we are sometime persuaded by crappy arguments that are delivered well. I put it to Less Wrong readers, how can we reduce the gap between what we are persuaded by, and what a bayesian superman is persuaded by?