It might be informative to note that the ratio of the square of a circle's area to its radius, or of its circumference to its diameter, is only necessarily what it is in Euclidean space because of certain axioms and postulates; in Hyperbolic space it is variable. This means it is logically possible for the number defined in this way to vary but with implications concerning other properties of geometry. In this sense, a (large) disagreement about
the value of π could be cordoned off by information about the curvature of spacetime. On the other hand, if you were to define π in a way which included all of the axioms and postulates of Euclidean geometry then imagining it to have a different value would become logically impossible and I don't know how to make a decision here beyond deriving a contradiction. If you actually believed this but were otherwise logical, you could believe anything because of the principle of explosion.
When I attempt to imagine myself operating in a universe in which π has slightly different digits, and conclude it wouldn't affect my decisions , beliefs or behaviors very much, I think what I am doing is not actually conditioning on π having those digits directly, but rather making the observation that my model of the universe is of a kind of 'equivalence class' of all the different universes in which the first few digits of π are those which I know, which I can infer has certain properties which all of them share from the knowledge that
π is between 3 and 4. As a mind, I consider myself to be living in this 'equivalence class' of universes in logical space when I can't distinguish between them by observation. If I had to consider a situation in which π(in the Euclidean sense) was equal to 2, I would necessarily "expand" in mathematical/logical space to a much broader type of universe where my system of axioms would not be sufficiently powerful to prove theorems like that
3<π<4, which would be a weirdly constrained place to live (and of course I can't actually do this).
Conversely, if I was a much more computationally powerful and intelligent but similarly inflexible mind, I wouldn't be able to imagine even the 2^2^2nd digit of π being anything other than what it is.
In the classic Prisoner's Dilemma (https://www.lesswrong.com/w/prisoner-s-dilemma), there are two agents with the same beliefs and decision theory, but with different values. To get the best available outcome, they have to help each other out (even if they don't intrinsically care about the other's values); and they have to do so even though, if the one does not help the other, there's no way for the other to respond with a punishment afterward.
A classic line of reasoning, from the perspective of one of the prisoners, goes something like this: I and my collaborator each only cares about himself. So it seems logical that we will defect against each other. However, there's some kind of symmetry at play here. If you abstract away the details of which specific prisoner I am, really I'm in exactly the same situation as my collaborator. So it's almost as though our decisions are logically bound to each other: Either my reasoning leads to me defecting, and therefore his reasoning also leads to him defecting, or else likewise I cooperate and he cooperates. We will make "the same choice" as each other, i.e. the symmetric / conjugate choice.
In this line of reasoning, what is shared between the two agents, and what is different? What's different is that I am here in this room, so I can respond Defect or Cooperate on behalf of this agent, and I care about how long this agent gets thrown in prison; and you likewise, mutatis mutandis. We what we share is our beliefs about the situation, our knowledge about how each other thinks, and our general procedure for making decisions. We both seem to act against our values (unshared), but it makes sense from a shared pespective that's behind a veil of ignorance about values.
In the Epistemic Prisoner's Dilemma (https://www.lesswrong.com/w/epistemic-prisoners-dilemma), the two agents have the same values (as well as the same decision theory). But they have different beliefs. According to my belief that the village is sick with malaria, we should treat them with malaria medication; you think it's bird flu, so you want to treat for bird flu. There's no time to explain our evidence to each other and argue and update. To cooperate and get the best outcome according to both of our beliefs, I should give up 5,000 malaria treatments (which I think would help) to gain 10,000 bird flu treatments (which I think would not help), and you should give up 5,000 bird flu treatments to gain 10,000 malaria treatments. We both seem to act against our beliefs, but it makes sense from a shared perspective that's behind a veil of ignorance about which beliefs we have (or which evidence we saw).
From the perspective of Functional Decision Theory (https://www.lesswrong.com/w/functional-decision-theory), the reasoning can go something like this: "I don't know exactly what all the subjunctive-dependent-consequences of my decisions are. However, clearly if there is another agent who is identical to me, then if I make a decision, I make that decision also for that other agent (indeed, I couldn't even tell which one I am). Further, if there is another agent who is identical to me except in some way that is completely separated from the general procedure I use to make decisions, then clearly if I make a decision, I make that decision for that other agent but conjugated through the differences—i.e. I also make the decisions for that other agent in situations that differ only in ways that my general decision procedure are (/ought to be) invariant to."
In the above cases, there is a clear way to "cordon off" differences between two agents. Prima facie (though maybe not actually), caring about this person or that person should not affect the general decision procedure you use. Likewise, having control over this or that agent's outputs shouldn't change your general procedure, and having seen this or that evidence shouldn't change it. When you can do this "cordoning off", the FDT reasoning (which is only a "spherical cow fragment" of what FDT would want to do) above should go through.
However, there's a whole spectrum of how possible it is to "cordon off" some difference between two agents, separated off from their shared identical decision procedure. On the "yes, 100% doable" end, you have two agents that start off identical, but then each see a different piece of information about the world. The information only says something like "this disease is probably malaria", and doesn't have other implications for the agents. They have the same prior beliefs, ways of updating on evidence, values, and ways of going from values to actions. So they can easily ask "What would I have done / how would I have thought, if I had seen what the other agent had seen?".
On the other end of the spectrum... I don't know exactly, I don't understand what this looks like. But the vague idea is: Imagine two agents who have eternal, core differences in their decision-procedures, which have many practical applications, and where it doesn't make sense to ask "what if the other agent is just like me, but with some tweak", because the difference between you and the other agent is so big that "If I were like that, I just would not be me anymore.".
What this does or could look like, I don't know. (See "Are there cognitive realms?".) An example might be CDT / Son-of-CDT. I think you can quasi-timelessly coordinate with Son-of-CDT—as long as you only need the agent to take actions that come after the moment when CDT self-modified into Son-of-CDT, and it only needs you to take actions that come after you've read the modified source code; or something like that. But you cannot timelessly coordinate with that agent before the magic self-modification moment. And probably there's some successful and reflectively stable Son-of-CDT like this, so we can't just say "well sure, other people can be irrational".
There are intermediate points on this spectrum which contain rich and confusing decision-theory questions. There are cases where the difference between the two agents is not so extreme that coordination is off the table (e.g. between an FDTer and Son-of-CDT, regarding actions before the magic moment), but the difference is not easy to "cordon off" from the decision theory. There are "metaepistemic prisoner's dilemmas", where the two agents have some different way of processing evidence or arguments. (It's not obvious that there is a canonical way for bounded agents to do this, especially about questions that are intrinsically too computationally complex to answer, e.g. many self-referential questions.) There are "meta-decision-theoretic PDs", where two agents have stumbled into or reasoned themselves into different decision procedures.
For example, how do you coordinate with a frequentist? Do you know how to answer the question "If I decide this way, what does that imply for how I would have decided if I were me-but-a-frequentist?"?
For example, how do you cooperate with someone who has a different idea of fairness from you?
For example, how do you coordinate with someone who is not updateless? (Imagine they are somehow reflectively stable.) (Are you updateless? Which things are you updateless about? Do you sometimes change your decision theory itself, based on evidence / arguments / thinking? How can you (and others) coordinate with yourself across that change?)
These questions are not necessarily meta-flavored. For example, in the Logical Counterfactual Mugging, Omega flips a logical coin by asking "What is the Graham's-numberth digit of π?". That seems easy enough to cordon off. I feel like I know how to answer the question "Even though I know that the answer is 6, how would I think and act if the answer were 5?". Since that logical question is not a consequence of, and doesn't have consequences for, my decision procedure, I know what that hypothetical means. But, what if it's the third digit of π? Do I know how to imagine that world? How would I think in that world? If the third digit were 9 rather than 1, that would totally change a bunch of computations, wouldn't it? And therefore it would change many of my decisions? (There might be a clean way to analyze this, but I don't know it.) (Since π should be pretty objective, i.e. all agents point to the same π, this question might actually boil down to questions like "How likely is it that I'm being simulated and tricked to think that the third digit of π is 1?".)
These are theoretically interesting questions, because they poke at the edge of decision-theory concepts. What's the boundary / boundaries between decision theory and other aspects of an agent like beliefs, belief-formation, and values? What is core to my decision theory, and what isn't? What are some sets of minimal conditions sufficient for me to be able to coordinate with another agent? There could be a different "core of my decision-theory" or minimal conditions for cooperation depending on context.
They're also practical questions. Human disagreements, conflicts, threats, bargaining, and coordination involve meta-agentic PDs. I'm frequentist, you're Bayesian; I think in term of duty to God, you think in terms of strategy-consequentialism; I rely on ethics, you rely on intuition and emotion; I reprogram myself in order to deeply pretend I believe X by actually believing X, you make truth-seeking triumph over everything else in yourself; I'm trying to find symmetries between us so that we can coordinate, you are trying to get the drop on me; and so on.
The setting of open-source game theory (https://arxiv.org/abs/1401.5577, https://arxiv.org/abs/2208.07006) can describe many of these situations, because it's very flexible—agents are programs. Löbian cooperation is a kind of coordination that works in a rich set of cases.
But for the most part these problems are unsolved.
Thanks to Mikhail Samin for a related conversation.