TL;DR. LessWrong's decision-theory debates (Newcomb, FDT vs CDT, counterfactual muggings) are almost entirely about what we suppose when we consider a candidate action or policy. There is a second, older, semi-orthogonal, but not fully orthogonal question: how to score a gamble once you know the possible outcomes. The predominant (and mostly implicit) answer to that was "take the expected utility". This post treats the two questions, plus a question about how a choice made before receiving information should relate to choices made afterward, as separate axes, and maps every decision theory you have heard of (and several nobody has built) into the resulting grid. Interestingly, the axes are provably entangled: theorems old and new show that there exist different restrictions on what places in that "decision-theoretic space" are inhabitable. I think there is a structure, maybe a deep and consequential structure, inside this map of decision theories which shows what possible combinations across the axes are coherent and fruitful. If we study it, we may understand the entire set of all possible coherent decision theories, something akin to the "metatheory of decision theories". It may be useful to know the entire set.

This is both a self-educational note and a research post. I have tried to state the scope of every result. While I am not certain about every conclusion about the "space of decision theories," I am confident that thinking in terms of the proposed space is useful. In the most conservative case, it is a nice map of decision theories. In a more optimistic case, the space has non-trivial structure whose study may lead to new results. Note also that this is a rather technical post. I tried to make it as simple as I could, but at some point, you lose precision, and I didn't want to go further that route, because it is meant not only as an educational post, but also as a research one. That said, I think with some additional effort it may become more clear. In particular, I added many footnotes that clarify terms (without putting a lot of theory in the body itself), and I think more footnotes can be added. Most of the footnotes are AI-generated, prompted by me to focus on addressing some common cases of confusion. Please let me know if something should be clarified. Also, as the post sometimes goes quite technical and is rather long, feel free to skip som...

Decision theory and metatheory

Decision theory and metatheory