Epistemic status: Some babble, help me prune.
My thoughts on the basic divide between rationalist and post-rationalists, lawful thinkers and toolbox thinkers.
Rat thinks: "I'm on board with The Great Reductionist Project, and everything can in theory be formalized."
Post-Rat hears: "I personally am going to reduce love/justice/mercy and the reduction is going to be perfect and work great."
Post-Rat thinks: "You aren't going to succeed in time / in a manner that will be useful for doing anything that matters in your life."
Rat hears: "It's fundamentally impossible to reduce love/justice/mercy and no formalism of anything will do any good."
Another way I see the difference is that the post-rats look at Newcomb's problem and say "Those causal rationalist losers! Just one-box! I don't care what your decision theory says, tell your self whatever story you need in order to just one-box!" The post-rats rally against people who are doing things like two-boxing because "it's optimal".
The most indignant rationalists are the one's who took the effort to create whole new formal decision theories that can one-box, and don't like that the post-rats think they'd be foolish enough to two-box just because a decision theory recommends it. While I think this gets the basic idea across, this example is also cheating. Rats can point to formalism that do one-box, and in LW circles there even seem to be people who have worked the rationality of one-boxing deep into their minds.
Hypothesis: All the best rationalists are post-rationalists, they also happen to care enough about AI Safety that they continue to work diligently on formalism.
In light of reading through Raemon's shortform feed, I'm making my own. Here will be smaller ideas that are on my mind.