Fundamentally, finding a good mathematical definition of decision theory that encompasses all the phenomena people care about is a big open problem.
My take: Counterfactuals are Confusing because of an Ontological Shift:
"In our naive ontology, when we are faced with a decision, we conceive of ourselves as having free will in the sense of there being multiple choices that we could actually take. These choices are conceived of as actual and we when think about the notion of the "best possible choice" we see ourselves as comparing actual possible ways that the world could be. However, we when start investigating the nature of the universe, we realise that it is essentially deterministic and hence that our naive ontology doesn't make sense. This forces us to ask what it means to make the "best possible choice" in a deterministic ontology where we can't literally take a choice other than the one that we make. This means that we have to try to find something in our new ontology that roughly maps to our old one."
We expect a straightforward answer to "What is a decision theory as a mathematical object?", since we automatically tend to assume our ontology is consistent, but if this isn't the case and we actually have to repair our ontology, it's unsurprising that we end up with different kinds of objects.
I think the most fundamental thing might be taking in a sequences of bits (or distribution over sequences if you think it's important to be analog) and outputting bits (or, again, distributions) that happen to control actions.
All this talk about taking causal models as an input is merely a useful abstraction of what happens when we do sequence prediction in our causal universe, and it might always be possible to find some plausible excuse to violate this abstraction.
If we want a space of all decision theories, what mathematical objects does it contain? For example, if a decision theory is a function, what are its domain and codomain?
The only approach I'm familiar with is to view expected utility maximizing decision theories as ways of building counterfactuals (section 5 in the FDT paper). A decision theory could then be described as a function that takes in a state and an action and spits out a distribution over world states that result from counterfactually taking action in state .
But EDT, CDT and FDT require different amounts and kinds of structure in the description of the state they take as input (pure probability distributions, causal models and logical models respectively), so this approach only works if there is some kind of structure that is sufficient for all decision theories we might come up with at some point.