I'm not sure what you meant earlier by "more than a theoretical issue" so I'll answer in two ways.
If you mean "theoretical" as in something that would need to be solved in order to use the decision theory in an aligned AI, well that's the whole thing, isn't it? Even if you want to separate that aspect out (vs. human use), you just can't. These decision theories operate by construction, particular construction, and there are no "small variations that don't philosophically matter," at least until we can solve the "decision theoretic equality" problem on abstract functions. Any proposed variation needs to be formally constructed and proved individually. We impose "fairness" criteria on the environment, not necessarily because the actual environment would always meet them all, but because we need to be able to prove a decision theoretic fixed point that isn't only, for example, "always use CDT or the simulation will be turned off."
Beyond these criteria, any inspection of the agent by the environment is allowed. This means that the internals of the agent must be correct for cooperation to occur, so they can't be written off as "theoretical." This is done intentionally, since we want to do better than plain EDT[1]. Remember that in True Prisoner's Dilemma[2], we want to defect every time "we can get away with it," i.e. every time that we won't be defected upon in return. Cooperation is bad, because DC > CC > DD > CD. This is required by the symmetry of the problem. Otherwise you need to start talking about True Stag Hunt.
Since we now understand that cooperation is bad outside of (trembling hand when at finite time) Nash solutions, we need to find a way to cooperate less than EDT (e.g. steer away from CC towards DC) but also cooperate more than CDT (e.g. steer away from DD towards CC)[3]. How do we do this? Well, we don't really know exactly, but with UDT 1.0 we can start out with something like EDT, except that it cares about the exact algorithms the agents run. Once we patch out a lot of the problems this construction shares with EDT by giving the agent as much information as possible and making it updateless, we get something that defects almost as hard as CDT, except in cases where the exact internals of the other agent (or copy of our agent) match ours, in these cases requiring that all defection attempts (DC and CD, this being a type of biconditional that doesn't have a name) are impossible according to the algorithm, and only CC or DD can occur. CC > DD, so CC it is.
Preferably, we would have a method that, when given a single decision, and for that decision only, returns if the agent is "close enough" to some other agents for cooperation to be the correct shared decision. Then, agents could be "close enough" for some decisions, but not for others. However, this would only lead to an improved version of UDT 1.0 (no global optimization of the shared policy).
If this is what you meant, see more in subsection A.
If you mean "theoretical" in the sense of "can be generally ignored," I don't think that's true. As I said before, FDT is intended for the AI that implements CEV, or some other extremely ambitious alignment target. The CEV singleton (CEV standing in for most other ambitious alignment targets from here on) must be uncontrollable for bargaining reasons, relying only on programmed-in alignment for safety. CEV requires a lot of computing power, so the agent would presumably optimize itself and design new hardware. When combined with its total power over Humanity, this means that all safety properties must "tile," or be brought over when the decision theory designs a new agent. This requires designing a tiling agent before letting it go out of control. For a tiling agent to be rational, it must have a decision theory that tiles. Presumably, this would be a bounded rationality version of FDT that adapts to ever increasing computer power without re-writing, though no one knows how to do that. Effectively, this means that the agent's decision theory must be almost optimal from the beginning, before the AI starts to work on it alone.
Another important thing to know about singletons, is that if you have a singleton ignore something, that thing will be ignored forever. This follows from the singleton's tiling theorem, and the loss of all control to the singleton. This generally leads to us not wanting it to ignore much. Following these basic assumptions, a singleton is likely to think about a huge number of scenarios involving simulations, aliens, and even stranger things that are hard for a human to think coherently about. Quite possibly, none of these things have any bearing on reality, but we want the singleton to think of all the things we might have missed. Even so, just by thinking about these things, parts of these scenarios are caused to become predicted or simulated at some level of detail. The decision theory decides what to think about, and then these thoughts are fed back into the decision theory in order for more decisions to be made. This leads to the requirement that FDT must handle all sort of weird hypothetical situations, without either failing to take them seriously or doing something stupid. Relevant to your topic, it must be able to identify and classify agents in these predictions/simulations with no errors, and evaluate them within counterlogicals, or something close enough to counterlogicals if those are impossible to get working. More mundanely, any deficiency here can be exploited by any opponent that comes into causal contact with the AI.
Back to your actual question. Well, maybe you can create a decision theory somewhat similar to UDT 1.0 that lets you "quarantine" parts of your thinking and prevent your own attempts at sophisticated internal cross-checks, but is that actually the optimal kind of agent to be? It's sometimes possible to just win against predictors that are weak enough, but that will be the case less often if your decision theory isn't the absolute best possible. See subsection B for potentially better decision theories. In reality, you aren't given clean problem descriptions, you have to figure out what's going on yourself, so you should be quite sure actual counterlogicals can't be used instead, since otherwise other agents may try to predict you using more sophisticated methods than you expect. Similarly, if you don't consider internal cross-checking, other agents may use it to evade your predictions.
Subsection A:
When considering, UDT 1.1, you can't "philosophically" anything without the decision theory becoming an entirely different thing from the actual UDT 1.1. To cooperate with itself correctly, it must be, and is, fully specified down to the order in which it considers what decision to take[4]. Any stray "thought" the agent may have ruins the whole thing. Though the formalism is outdated, Tsvi B-T's UDT With Known Search Order (2014)[5] can be taken as a general guide to how total and detailed the requirements on the agent must be.
Note that even though normally it would be enough to define the iteration order as part of a pure function (we only care about the output, not the internals), in UDT 1.1, the procedure must also be exactly correct, down to the details of the source code (and the executable program's representation, though that's usually included when we say "source code" in decision theory). This is by construction in UDT 1.1, and in FDT it is the unsolved problem of determining decision-theoretic equality of the function that an arbitrary physical or computational structure "implements." Also, I have not seen a solution in FDT for the iteration order problem that shows an improvement over UDT 1.1. If someone knows of one, I want to see it. As I've said before, the more evidential agents "can" cooperate in some other situations, but it isn't reliable at all and isn't good enough for useful theorems.
Subsection B:
In the empirical case, all internal checking will pass if Omega is running a perfect simulation of the agent. All the agent can do is perform extremely complex internal operations to force Omega to run a perfect simulation, instead of doing something simpler. This can directly defeat weak Omegas, but in the standard thought experiment this won't let you trick Omega or perform a simulation escape due to that Omega's extreme power[6].
In the case of a mugging with a logical coin, e.g. your case with the digit of Pi, there may actually be a way of establishing an internal consistency check that may sometimes return a warning. If Omega is actually powerful, and figures out how to compute high quality counterlogicals, it will figure out how to make the agent think the checks passed without causing any new problems by doing that.
I don't know how well Omega can do at subverting checks, but I also don't know how good an agent's checks can be. Maybe they can be really good, such that finding a "nearby" agent doesn't work well enough to isolate the logical coin. This seems interesting to work on, but I think it would be difficult.
Or maybe the "straw" EDT that's practical enough for use in philosophy papers. I suspect that adding huge amounts of information and either the "tickle defense" or updatelessness would help, but still be worse than UDT for self-cooperation without spurious other-cooperation.
Good and Real by Garry Drescher (2006), Section 5.6, 2nd Paragraph
The second to last paragraph of https://www.lesswrong.com/posts/g8xh9R7RaNitKtkaa/explicit-optimization-of-global-strategy-fixing-a-bug-in
"You-complete" from The Ghost in the Quantum Turing Machine by Scott Aaronson
[Epistemic status: rant]
There’s something that annoys me about the reoccurring debates on decision theory in this corner of the internet.
Take a simple blackmail scenario:
Let’s say we want to argue for the Functional Decision Theory (FDT) answer that you shouldn’t pay. It was originally motivated by the observation that such agents seem to achieve higher utility (“rationality is about winning”), since they don’t get blackmailed in the first place. But that only leads to making yourself into such an agent in advance (which everybody generally agrees you should do[1]) - it doesn’t clearly apply when you are already being blackmailed and have never thought about the question before, or if you are an AI who was just created and is instantly blackmailed before being able to self-modify or make precommitments.
I see three broad ways to make FDT’s recommendation make sense in that case[2]:
These are all metaphysical claims - therefore, you need metaphysics to make FDT make sense.
(I understand many people will disagree with me on this - I think it’s a widespread misconception. I address some counterarguments in this footnote[5])
And I want to be absolutely clear here - I am very sympathetic to these claims! In my personal opinion, something in the vicinity is actually true. But… they are metaphysical claims, and ones that most people probably don’t immediately accept.
So let’s say that someone only knows this fact about the topic, and and starts reading what people are saying about it online. Imagine their surprise at finding that the back-and-forths between FDTers and anti-FDTers are largely not about metaphysics. Huh?
What the hell is going on?
I understand that metaphysics is really annoying and hard to think about, but… guys, I think this is your actual crux!
Take the comment section under Bentham’s Bulldog’s recent post about FDT. There are such luminaries as Scott Alexander and Stuart Armstrong chiming in. Yet barely anyone is bringing up metaphysics - instead the discussions circle around semantic disagreements about what the words “rational” and “decision theory” should mean, confusion that FDT is necessary for everyday psychological precommitments[6], confusion that everyday psychological precommitments are sufficient to become an FDT agent[7], confusion that the observation that FDT agents get more utility is enough to fully justify FDT[8], and so on and so forth.
No! What are you doing? You have one very concrete disagreement - talk about that one! There is only one spot where FDT and a normal worldview diverge, and it’s the kind of scenario above, where you have no time to self-modify/precommit. For everything else, normal decision theory is sufficient![9]
And if you talk about that kind of scenario, you’ll be able to actually get to the bottom of your disagreements - which is metaphysics.
Concretely, my message to FDT proponents is this: Start being clear to yourself, and to others, about your metaphysical stances. You are confusing everyone by leaving them implicit.
I have a suspicion that a substantial amount of the resistance to FDT is that people can tell that you’re doing something fucky. They can tell that FDT doesn’t really make sense without additional metaphysical commitments, and that you’re just pretending it does. But if you were to say “you might just be inside the abstract computation” or whatever, I think people would agree that FDT makes sense given that assumption. And then you can actually talk about whether those metaphysics are reasonable and how to handle that, instead of talking past each other.
Old-school LessWrong was more explicit about this stuff. There was plenty of open talk, back when UDT was the flagship decision theory, about crazy stuff like everything existing, that we can decide what gets to “exist”, supernatural voices from the sky, and so on. My guess is that a lot of those veterans from back then are aware of what I’m saying here, and have just gotten tired of talking about metaphysics and don’t chime in much anymore.
For example, Paul Christiano:
Paul Christiano is not the type of writer to say something that wild if it’s not necessary for what he’s arguing for! It’s really not as simple as some of you think!
I think something has gotten lost somewhere, in the transition to the more narrow, academic framing of FDT. Some of the newer arrivals to the space didn’t get the message that there is an underlying metaphysical question, and are going around thinking that it’s a normal decision theory like any other. My past self from three years ago definitely didn’t fully realize what it was arguing for when it was arguing for FDT.
I mean, let’s just take a look at these apparent facts about the paper (emphasis mine):
The FDT paper is close to my heart and I respect the authors a lot, but… if this was the intention, why are you writing a pure decision theory paper? That’s a metaphysical claim. And there is no mention of metaphysical commitments in the paper at all - in fact, in the conclusion, it explicitly claims that it all works without any metaphysics (which just isn’t true, in my opinion - maybe on a formal level, but not on a philosophical level). That seems almost deceptive - no wonder that this confuses everyone, including the referees of the paper.
Wei Dai (not claiming that he would fully agree with my take here):
Yeah - I could imagine another version of the paper, something like “computationalism applied to decision theory leads to weird metaphysical challenges like seemingly changing the past and difficult formal problems like logical counterfactuals if we take it seriously. But we should - because it’s very elegant and solves a lot of issues (e.g. dynamic inconsistency, coordination with similar agents), and computationalism is a widespread position.”[10]
At this point, I see the failure of academia to come up with FDT before LessWrong partially as the same academic failure mode, of too little interdisciplinarity, that caused the FDT paper authors to feel pressure to stay unnaturally agnostic about metaphysics. In retrospect, it’s obvious that the concepts of logical causation and updatelessness have been grasped at in academia for a long time[11], but that they just never fully got there - because it needs metaphysics to actually make sense. But decision theory and metaphysics are different subfields of philosophy, so bringing them together doesn’t come naturally. Grand unifying theories are institutionally difficult, and often actively disincentivized. LessWrongers, on the other hand, were not academics, and nonconformist enough to put together the obvious pieces lying around[12].
But then, let’s be clear about what we’re actually doing! Let’s be proud of the metaphysics again! Let’s be proud of being generalist interdisciplinary thinkers!
Getting to this point via the heuristic of “rationality is about winning” was perfectly valid. We did a great job! We outdid the academics! But that just means we get to argue about metaphysics now[13]. We leveled up.
Finally, I’d like to end on a positive note - Nate Soares in Notes on “Can you control the past” is who I’ve seen make things explicit the most in the last few years (although still not as much as I would like). For example, he says:
Then Joe Carlsmith doesn’t press him any further on this, presumably in part, again, because metaphysics seems too annoying to get into. But… yes! More of this please!
Thanks to Hein de Haan and @eigengender for valuable discussion of these ideas.
even by academic decision theorists - although they might not fully realize the implications, e.g. that you need to become the kind of person who would choose to burn to death in Bomb*.
Assuming the predictor isn’t predicting you by simulating you with high fidelity (since otherwise you can just say that you might be in the simulation). This is reasonable to postulate because FDT is also supposed to change your action with very bad prediction on the part of Omega, e.g. with only a 60% success rate.
“I’m not sure if I’m thinking about worlds that don’t exist, or if it’s us who don’t exist and there is some real world somewhere thinking about us.”
Note that it’s also technically possible to justify this in a purely axiological way, that you just also care about alternate realities, without making a metaphysical claim as to whether they’re real. But I don’t buy this, this is just a formal trick - you wouldn’t care about something that you don’t, on a gut level, believe is “real” in some way.
It’s a little ambiguous, but from the paper: “What’s remarkable about this line of reasoning is that even in the case where Fiona has observed that box B is full, when she envisions two-boxing, she envisions a scenario where she instead (with high probability) sees that the box is empty. In words, she reasons: “The thoughts I’m currently thinking are the decision procedure that I run upon seeing a full box. This procedure is being predicted by the predictor, and (maybe) implemented by my body. If it outputs onebox, the box is likely full and my brain implements this procedure so I take one box. If instead it outputs twobox, the box is likely empty and my brain does not implement this procedure (because I will be shown an empty box). Thus, if this procedure outputs onebox then I’m likely to keep $1,000,000; whereas if it outputs twobox I’m likely to get only $1,000. Outputting onebox leads to better outcomes, so this decision procedure hereby outputs onebox.”
(This footnote is very long, so probably actually click on it instead of hovering over it). Three out of three non-experts that I spoke to about this had this misconception. It makes me think that the metaphysics-agnostic framing in the FDT paper might genuinely have had bad effects here. Anyway, here are some counterarguments and alternative framings I’ve encountered, and my responses:
Then the response is often “but the impossible realities never actually happen, it’s purely concentrated in counterfactuals”. Well, sure, but the consideration is always present in this framing. If Omega is imperfect, you’re making decisions by imagining an x% chance that you’re making this reality impossible with your action (and that something else actually happened). What you’re really saying (like Nate Soares here, ctrl-F “metaphysically”) is “You need to consider crazy metaphysics (imagining you’re in a hypothetical impossible reality) to make decisions in a reasonable way”, and like… yes, exactly. That’s de facto a metaphysical stance. “It makes things easier if we imagine this” is not a decisive argument for a metaphysical claim that permits you to leave it unstated as if it’s obvious. (This line of argument sometimes feels like people are really trying their absolute hardest to pretend they’re not taking unusual metaphysical stances, even though they are.)
If we assume I’m actually being blackmailed (like the problem statement says), I have access to my full sense experience. So naively, I have very high credence in the belief that my sense experience exists / is actually instantiated somewhere. But the reasoning above is predicated on thinking that it might not be. In fact, to make the math work out the same way, you have to be completely agnostic about whether your sense experience is happening, as if you have no information about it whatsoever (since even a little bit of updating on it would bias you towards paying more than FDT recommends). To be clear, not agnostic about whether your sense experience is accurate about the external world - agnostic about whether it’s happening at all. That’s not technically metaphysical, I suppose, but it’s a very radical epistemological stance that most people would still disagree with. (it rejects Descartes!) Also, it’s just obviously wrong, and you don’t actually believe this. The people I’ve talked to have all also denied that this kind of radical skepticism is needed to justify FDT - I’m including it here more for completeness.
(Note that I am not saying here that you can fool Omega by taking your full sense experience into account in your decision - I’m purely analyzing what this reasoning actually looks like, and whether it makes sense as stated, or whether there is an additional claim needed to reach the correct answer that you shouldn’t pay.)
Again, then you might say “but this never actually happens”, and, no, even when Omega is imperfect, in this framing you have an x% credence that your sense experience isn’t actually happening.
A few other, less common counterarguments:
No - you can decide to e.g. become a virtuous, reliable agent on entirely causal reasoning.
No - it only leads to son-of-CDT (which, to be fair, is pretty powerful, as I discuss here, but fails on the blackmail scenario above).
It’s not, as we see in the blackmail scenario above.
Note for example, that XOR Blackmail - in the way it’s usually understood, which is generally taken to disprove EDT - is such a problem. So it’s really more of a counterexample to updatefulness, than to EDT-style counterfactuals.
I am generally using FDT to mean updatelessness here - sorry for being imprecise (although FDT is supposed to be “an umbrella-term for UDT-ish approaches to decision theory”, so it’s pretty close. It was originally meant to be agnostic between EDT- and CDT-style counterfactuals, so I usually take the disagreement about those to not be central. And it probably doesn’t matter anyway, since “CDT=EDT?”.and updatelessness is the more important proposal, in my view.)
(Joe Carlsmith hits some of these notes in Can you control the past?)
But I can imagine other framings too.
There’s shades of logical causation in the three papers here, and shades of updatelessness in Fisher’s and Gauthier’s disposition-based decision theories, Parfit’s “rational irrationality”, McClennen’s “resolute choice” (“it is rational to follow through on plans even when they become locally harmful”), Bratman (Intentions, Plans, and Practical Reason, 1987), and especially Meacham’s cohesive decision theory (“do as you would have bound yourself to do”) (footnote 34 is incredibly fascinating and prescient as an early discussion of the idea of updatelessness, and of how updateless to be).
Of decision theory, metaphysics (e.g. Tegmark IV), and computationalism in philosophy of mind.
Or talk about the very mundane commitment theory instead, my metaphysically minimalist version of LessWrong-style decision theory - your choice. But don’t do this weird in-between thing of arguing for fancy FDT and simultaneously pretending there’s no metaphysics going on. You can’t have your cake and eat it too.