Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the "valley of bad rationality".
From Incremental Progress and the Valley:
...Ah. Well, here's the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.
The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.