I have an alternative ending:
Sometimes you just get unlucky. That doesn’t mean you get to betray your past self.
...
Yeah. There’s an honor in that. A righteousness.
...
But ... how strongly did I actually make the great commitment? I said the words and have always done my best to live by them. I thought that was enough. But now I am faced with the ultimate test. Omega knows me better than I do. Surely, "nor is it Omega nods, but we that dream." Isn't it more likely that I faltered in the strength of my great commitment, and Omega knew that? That I was guilty of just a smidgeon of virtue signalling and belief in belief in my commitment?
If Omega had made the Left box empty, it would be easy for me to take it, and everything would be consistent. That would be no test at all.
But Omega could also have put a bomb in it, the bomb I'm looking at, knowing that I was not capable of that level of perfection and that I would choose Right.
(you look again at the Right box)
This goes against everything I believe...
(you sob in despair and cry out)
I'm not worthy! I'm not worthy!
(you take the Right box. PHHOOFFF! As a penance you shave all your hair off and spend a week on your knees with tweezers picking every single grain of glitter out of the soft furnishings and the carpet.)
(If you want to immediately skip to why I believe past attempts to incorporate LessWrong-style intuitions into standard decision theory are insufficient, go to footnote 16)
Thanks to @eigengender and especially Chris Lakin and Simon Dima for valuable comments on a draft.
In this post, I will present an alternate framing[1] of LessWrong-style decision theory. I believe it should make its concrete recommendations much more palatable to people who find them absurd. Very little of the theoretical substance here is original to me, and the core claim will be obvious to many, but I don’t think anyone has cleanly framed it all like this for easy consumption.
TLDR: Framing the thorny issues of decision theory[2] as stemming from a novel, non-decision-theoretic consideration (something like “cooperating with your past self”) - instead of forcing the relevant intuitions and claims into a decision theory frame - is much less counterintuitive, and gives the exact same recommendations.
Introduction
Let’s briefly check out Will MacAskill’s “Bomb*” scenario[3]:
Functional decision theory (FDT), the main LessWrong-style decision theory, recommends taking the Left box - in the full knowledge that as a result you will slowly burn to death. Why? Because if your decision process were to output ‘Left’, then Omega would have predicted that. So there would be no bomb in the box, and you could save yourself the effort of having to clean by taking the Left box[4] - even though that would mean changing the past.
Proponents also sometimes say, by putting the same math into words differently[5], that FDT chooses to take the Left box and burn to death because it imagines that you must be in the reality where Omega made a mistake, and that you can currently affect 999,999,999,999,999,999,999,999 other versions of you and cause them to not have to clean - even though we have no reason to think these other versions exist.
Many people find this kind of recommendation, and either of these framings, extremely implausible - both MacAskill and Bentham’s Bulldog see it as a knock-out blow to FDT, and Wolfgang Schwarz, an academic decision theorist who was a referee of the FDT paper, calls it “insane”. LessWrongers, on the other hand, insist that this way of thinking actually makes sense.[6]
The debate has landed at an impasse - both sides have planted their flags and refuse to budge from their deeply held intuitions.
But I believe that there is a way out.
Many people have made the observation that LessWrong-style decision theory is trying to do something different, in some deep worldview-level way, from academic decision theory - and that this is what leads to the persistent and seemingly intractable disagreement between the two camps.
For example, Paul Christiano:
Scott Garrabrant:
Will MacAskill:
So… let’s run with this. Let’s assume that LessWrongers were wrong all along when they thought they were doing decision theory. Let’s come up with a different name for whatever it is that they’ve been doing: call it commitment theory. And let’s reexamine the topic with a fresh eye.
The great commitment
Let’s step back from the extremity of “Bomb*” for a minute - take a simple blackmail scenario:
Everyone agrees that if you can precommit (somehow force your future self) to not pay before you get into this situation, you should - since that would make it so Omega won’t blackmail you and you don’t get into this situation in the first place.
So there’s a class of situations where you would like to previously commit yourself to a choice, to get better outcomes overall. That’s interesting.[7]
So maybe we should do a kind of "generalized precommitment" for those situations, where we commit now to always doing whatever it would have been good to have precommitted to?
Yeah - it seems like, by definition, it would be good to do that. It’s not clear yet what exactly it means for something to be “good to have been precommitted to”, but… we can figure out the details later. I mean, this is tautological: It just has to be good to precommit to… the actions that would have been good to have precommitted to.
In fact, we should do this ASAP, before we end up in such a situation. So let’s do it right now, together, on three: one... two... three!
Well done! Let’s call whatever we just did, this “generalized precommitment”, the great commitment.
What does it mean concretely, that we made this “great commitment”? Well, we’ll get better outcomes if we get put into the blackmail scenario above in the future - so that’s good.
What about other cases? What about, say… Parfit’s Hitchhiker?
In the past, we obviously wouldn’t have paid, since it would lose us 1,000$ for no reason. But now, as commitment theorists, we think:
Okay, that’s interesting. It seems like this might be pretty useful!
Commitment theory
But what does this all mean? What did we just commit ourselves to, exactly?
That’s exactly what commitment theory research consists of - the study of what situations are covered, and how much, by our vague intuitive notion of the “great commitment”.
And since it seems like the great commitment is tautological and like all agents should make it, commitment theory research seems like an important part of the study of ideal decisionmaking - even if it may be distinct from decision theory as such. One might say that decision theory is concerned with what choices are rational, and commitment theory with how, precisely, we should force our future selves to be irrational.[8]
FDT
So how does this relate back to LessWrong-style decision theory?
Well, you might have guessed where this was going already - sorry for tricking you into it. When you made the great commitment earlier, you actually, for all intents and purposes… became an FDT agent[9]. Congratulations!
Yes - commitment theory gives the same concrete recommendations as FDT[10], seemingly without requiring any big metaphysical claims about changing the past or alternate realities[11], or any big normative claims about seemingly absurd choices actually being “rational”.
(The only normative claim is that you should make the great commitment ASAP, if you haven’t already - again, it’s tautologically good!)
This also finally gives us a way of reinterpreting some of the complicated technical machinery[12] that people have come up with over the years in their study of LessWrong-style “decision theory” - all this kind of stuff:
Treutlein (2023) defining a bargaining solution for acausally correlated players in a Bayesian game.
Wei Dai (2010) fiddling with the implementation of the concept of updatelessness.
In effect, it’s exactly the commitment theory research that I referred to before: trying to pin down and formalize our intuitive notion of “what would have been good to have precommitted to” - trying to figure out what exactly the great commitment covers.
A detailed example
Lastly, let’s reexamine “Bomb*” from our new perspective:
As a commitment theorist, your thoughts in this situation might now go along these lines:
Where FDT was doing weird stuff, commitment theory makes the equivalent decisionmaking seem more understandable. Of course, it remains counterintuitive - as Bomb* was designed to be maximally counterintuitive. But hopefully this makes it at least conceivable that a real person might actually take the Left box.
I do believe the great commitment is possible. Things like this are normal: for example, you might tell yourself to get up early tomorrow - and that itself makes you to do it, even if you don’t feel like it in the morning. Human brains are physical systems and, via internal physical manipulations (cognitive activity), can affect their future mental state.[15]
In my mind, it’s a psychological change, some deep realization of a new, non-decision-theoretic notion of “acting correctly” - something like “cooperating with”, being “loyal to” and “coherent with” your past self.
Whether it’s psychologically possible for you to follow such a principle as far as in Bomb*, when the stakes are life-and-death, is an empirical question. But that doesn’t change the fact that the situation is clearly covered by the great commitment, i.e. that you would want to precommit to choosing to burn to death in Bomb*, if you could.
And… you can.
Conclusion
Here’s what I think your takeaways from this post should be:
To be clear, it’s not my personally preferred framing, and in my opinion it ultimately fails for conceptual/metaphysical reasons, and LessWrong thought is much closer to correct - but I think it goes quite far, as I’ll argue in this post.
Those about updatelessness, specifically, not those about counterfactuals (which are obviously part of academic decision theory).
I’ve rewritten it to make it clearer, since the original is a bit confusing and underspecified (see this Stuart Armstrong comment), and including that Omega specifically simulates you to predict you is unnecessary. My version preserves the structure and should meet the core desideratum of its proponents (that FDT chooses to burn to death for seemingly no reason) - but maybe call it Bomb* (with a star at the end) for clarity, if you reference it.
It’s kind of like the Transparent Newcomb’s Problem, but optimized for counterintuitiveness - for example, that you are in an “impossible reality” and not in a “possible reality that you just need to make real”.
at least with extremely high probability. (also, for convenience this paragraph is taken from Will MacAskill and modified for my context - I couldn’t find a better way of explaining it than his, but I also needed to make too many small changes to make it a direct quote.)
This is a more UDT-style framing.
e.g. there are a variety of arguments in the comments on Will’s post.
This is the academic discussion on “binding” and dynamic choice.
One might also gesture to some intuition that this is central to what “being an agent” means - you only have the ability to make plans if you can lock your future self into a set of actions, if you can expect your future self to “cooperate” with your current self’s commitments. Decision theory and commitment theory seem like two sides of the coin of ideal decisionmaking - how to maximize for your preferences (an adversarial position towards the world) versus how to intentionally ignore and blind yourself to your preferences (to cooperate with the world). (This framing is inspired by Richard Ngo).
This is the famous “Non-FDT agents self-modify to be FDT agents” argument, put into practice.
This is one of the core ways that UDT was originally characterized. The clearest statement of that I’ve been able to find is Demski and Garrabrant (2019), who straight-up define it as: “UDT [recommends] that the agent do whatever would have seemed wisest before—whatever your earlier self would have committed to do”.
(and FDT is an umbrella term for “UDT-ish approaches to decision theory”. Although there are some terminological subtleties - there’s a longer comment by Rob Bensinger in that post that’s quite clarifying. But it also comes up several times in the FDT paper as a core property, just ctrl-F “committed”.)
Of course, you still need the correct counterfactuals - e.g. when deciding what to precommit to in a situation with an agent that is similar but not identical to you. But counterfactuals are the “decision theory dimension” (EDT vs CDT), not the “commitment theory dimension” (updateless vs updateful), so they are a separate issue. (But I think e.g. commitment theory EDT just works).
It’s buried pretty deep in the post, so I’ll quote it - Christiano says: “I’m not sure if I’m thinking about worlds that don’t exist, or if it’s us who don’t exist and there is some real world somewhere thinking about us.”
There’s one caveat here (this is complicated and not well-argued, feel free to skip): When agents throughout the universe realize that they should make the great commitment, they might already be in the midst of executing strategies that they might together have precommitted to not doing in advance (e.g. cooperating with other value systems in the universe too little - ECL), if they were making the great commitment from the perspective “outside of time” of the simple agent structure that is inside of them (cf Nate Soares’ discussion of being “wrapped around The Algorithm” here). This is the debate about “how far back” to be updateless in UDT - there’s an analogous debate in commitment theory about how simple of a version of yourself you imagine making the great commitment (and therefore how much what you commit to is determining outcomes all throughout the universe). So you can’t avoid weird metaphysics after all, (“outside of time” is obviously metaphysical reasoning), no matter how hard you try. That’s part of why I think the commitment theory framing is ultimately insufficient. But it should at least make clear that people shouldn’t be disagreeing about stuff like Newcomb’s problem and Bomb* (anything other than ECL, really) - you don’t even need to get into metaphysics to see the right answer there!
Not all of it - Some of it is better classified as EDT vs CDT stuff (i.e. the “decision theory dimension”), and some as updateless vs updateful stuff (i.e. the “commitment theory dimension”).
If this makes you disconnect from the scenario because your past self wouldn’t be this utilitarian, just imagine some other less bad consequence. Something like this works no matter what your preferences are - the point is just that FDT can make you choose a locally bad outcome for seemingly no reason.
Of course, if you face such an unlikely situation in real life, you should completely rethink your assumptions and seriously wonder if you’ve gone crazy. But as it’s a hypothetical decision problem, we’ve fixed the agent’s ex hypothesi beliefs about the situation he’s in - so I’ve written this snippet like you just happen to not think of that possible response.
Once again, this is the academic discussion on binding.
I want to briefly examine other attempts to do something similar, and explain why I think they fail. First, Will MacAskill’s “Global CDT”, as it’s closest in the social graph:
Even as it succeeds at replicating a lot of FDT’s recommendations, it leaves less of its conceptual attractiveness intact. Global CDT is framed as simply evaluating with CDT many different “evaluative focal points”: different personalities, dispositions, rules, and so on. For example, it gets as specific as “[the] sort of person… who cooperates in prisoner’s dilemmas”. But we’d prefer to retain the intuitive sense that the “FDT insight” is one singular thing (“act as you would have precommitted to”), and still avoid the associated metaphysical and normative claims - which we can, with commitment theory.
More importantly, I think it fails on counterfactual mugging with a deterministic coin, since the reality where the coin lands heads is not real and could never have been real, and so doesn’t get taken into account in any updateful decision theory after you learn it lands tails (even if we evaluate bigger things like dispositions or rules). (You can understand dispositions and rules as leaky human approximations of theoretical “true” updatelessness - which just evaluates anew the correct precommitment to have taken for every situation)
For attempts from academia (this is going to be uninteresting if you’re not in the weeds on this stuff, so feel free to skip, and I haven’t fully read any of these papers so take it with a grain of salt): I expect the worries above to also apply to Fisher’s and Gauthier’s disposition-based decision theories, Parfit’s “rational irrationality”, and Spohn (”Reversing 30 Years of Discussion”, 2012). McClennen’s “resolute choice” (“it is rational to follow through on plans even when they become locally harmful”) is insufficient because it fails on situations that you didn’t anticipate, like Bomb* - the thing you need to be “resolute” about is much deeper than plans. The same goes for Bratman (Intentions, Plans, and Practical Reason, 1987). Poellinger (“Unboxing the Concepts in Newcomb's Paradox”, 2013) and Hedden (”Counterfactual Decision Theory”, 2023) are about counterfactuals (the “decision theory dimension”), and not updatelessness (the “commitment theory dimension“), so they’re orthogonal to this discussion.
Meacham’s cohesive decision theory (“do as you would have bound yourself to do”, quite similar) is the closest academic relative to FDT (footnote 34 is genuinely fascinating as an early discussion of the idea of updatelessness) and has seemingly the exact same recommendations (modulo a theory of counterfactuals) - but as it also has acts as its evaluative focal point, it incurs the same intuitive drawback of calling it “rational” to choose to burn to death in Bomb*. Also, since it lacks the intuition pumps I’ve built up around it, and tries to fit it into the unnatural frame of decision theory, it ends up looking a bit unmotivated (which is maybe why it didn’t get much traction). It also doesn’t explicitly open up the question of what exactly it means for an action to “have been good to have been bound to” (which ends up being a rich field of study).
Especially because, in my opinion, you inevitably run into metaphysical questions (changing the past, alternate realities, platonism about computations) if you really try to justify why e.g. taking the Left box and burning to death in Bomb* is the “correct” decision, in the decision-theoretic sense of maximizing your utility. (Not sure how much of a hot take that is, I will justify it more in a future post). And it’s understandable that people are reluctant to wade into that.
Finally, a brief and dense sidenote (don’t worry if this doesn’t make sense to you): Commitment theory is actually slightly better than naive TDT-style FDT (which is the way many people conceive of FDT), because it performs better on counterfactual mugging with a deterministic coin. This is the sense in which FDT lacks a theory of anthropics, in the “UDT = FDT + a theory of anthropics” way of carving up the space - another way of seeing that metaphysics comes into play.
(Of course, commitment theory doesn’t fully avoid metaphysics at the end of the day either, as I argue in footnote 11)