Hi! I just wanted to mention that I really appreciate this sequence. I've been having lots of related thoughts, and it's great to see a solid theoretical grounding for them. I find the notion that bargaining can happen across lots of different domains -- different people or subagents, different states of the world, maybe different epistemic states -- particularly useful. And this particular post presents the only argument for rejecting a VNM axiom I've ever found compelling. I think there's a decent chance that this sequence will become really foundational to my thinking.

This reminds me of an example I described in this SL4 post:

After suggesting in a previous post [1] that AIs who want to cooperate with each other may find it more efficient to merge than to trade, I realized that voluntary mergers do not necessarily preserve Bayesian rationality, that is, rationality as defined by standard decision theory. In other words, two "rational" AIs may find themselves in a situation where they won't voluntarily merge into a "rational" AI, but can agree merge into an "irrational" one. This seems to suggest that we shouldn't expect AIs to be constrained by Bayesian rationality, and that we need an expanded definition of what rationality is.

Let me give a couple of examples to illustrate my point. First consider an AI with the only goal of turning the universe into paperclips, and another one with the goal of turning the universe into staples. Each AI is programmed to get 1 util if at least 60% of the accessible universe is converted into its target item, and 0 utils otherwise. Clearly they can't both reach their goals (assuming their definitions of "accessible universe" overlap sufficiently), but they are not playing a zero-sum game, since it is possible for them to both lose, if for example they start a destructive war that devastates both of them, or if they just each convert 50% of the universe.

So what should they do? In [1] I suggested that two AIs can create a third AI whose utility function is a linear combination of the utilities of the original AIs, and then hand off their assets to the new AI. But that doesn't work in this case. If they tried this, the new AI will get 1 util if at least 60% of the universe is converted to paperclips, and 1 util if at least 60% of the universe is converted to staples. In order to maximize its expected utility, it will pursue the one goal with the highest chance of success (even if it's just slightly higher than the other goal). But if these success probabilities were known before the merger, the AI whose goal has a smaller chance of success would have refused to agree to the merger. That AI should only agree if the merger allows it to have a close to 50% probability of success according to its original utility function.

The problem here is that standard decision theory does not allow a probabilistic mixture of outcomes to have a higher utility than the mixture's expected utility, so a 50/50 chance of reaching either of two goals A and B cannot have a higher utility than 100% chance of reaching A and a higher utility than 100% chance of reaching B, but that is what is needed in this case in order for both AIs to agree to the merger.

I'm confused by the "no dutch book" argument. Pre-California-lottery-resolution, we've got $CB\prec CA$, but post-California-lottery-resolution we simultaneously still have $A\prec B$ and "we refuse any offer to switch from $B$ to $A$", which makes me very uncertain what $\prec$ means here.

Is this just EDT vs UDT again, or is the post-lottery $A\prec B$ subtly distinct from the pre-lottery one,  or is "if you see yourself about to be dutch-booked, just suck it up and be sad" a generally accepted solution to otherwise being DB'd, or something else?

Figure I should put this somewhere: I recently ran into some arguments from Lara Buchak that were similar to this (podcast: https://www.preposterousuniverse.com/podcast/2022/12/12/220-lara-buchak-on-risk-and-rationality/)

These are super interesting ideas, thanks for writing the sequence!

I've been trying to think of toy models where the geometric expectation pops out -- here's a partial one, which is about conjunctivity of values:

Say our ultimate goal is to put together a puzzle (U = 1 if we can, U = 0 if not), for which we need 2 pieces. We have sub-agents A and B who care about the two pieces respectively, each of whose utility for a state is its probability estimates for finding its piece there. Then our expected utility for a state is the product of their utilities (assuming this is a one-shot game, so we need to find both pieces at once), and so our decision-making will be geometrically rational.

This easily generalizes to an N-piece puzzle. But, I don't know how to extend this interpretation to allow for unequal weighing of agents.

A preference ordering on lotteries over outcomes is called geometrically rational if there exists some probability distribution $P$ over interval valued utility functions on outcomes such that $L⪯M$ if and only if $G_{U\sim P}{E}_{O\sim L}U\left(O\right)\le G_{U\sim P}{E}_{O\sim M}U\left(O\right)$.

How does this work with Kelly betting? There, aren't the relevant utility functions going to be either linear or logarithmic in wealth?

One elephant in the room throughout my geometric rationality sequence, is that it is sometimes advocating for randomizing between actions, and so geometrically rational agents cannot possibly satisfy the Von Neumann–Morgenstern axioms.

It's not just VNM; it just doesn't even make logical sense. Probabilities are about your knowledge, not the state of the world: barring bizarre fringe cases/Cromwell's law, I can always say that whatever I'm doing has probability 1, because I'm currently doing it, meaning it's physically impossible to randomize your own actions. I can certainly have a probability other than 0 or 1 that I will do something, if this action depends on information I haven't received. But as soon as I receive all the information involved in making my decision and update on it, I can't have a 50% chance of doing something. Trying to randomize your own actions involves refusing to update on the information you have, a violation of Bayes' theorem.

The problem is they don't want to switch to Boston, they are happy moving to Atlanta.

In this world, the one that actually exists, Bob still wants to move to Boston. The fact that Bob made a promise and would now face additional costs associated with breaking the contract (i.e. upsetting Alice) doesn't change the fact that he'd be happier in Boston, it just means that the contract and the action of revealing this information changed the options available. The choices are no longer "Boston" vs. "Atlanta," they're "Boston and upset Alice" vs. "Atlanta and don't upset Alice."

Moreover, holding to this contract after the information is revealed also rejects the possibility of a Pareto improvement (equivalent to a Dutch book). Say Alice and Bob agree to randomize their choice as you say. In this case, both Alice and Bob are strictly worse off than if they had agreed on an insurance policy. A contract that has Bob more than compensate Alice for the cost of moving to Boston if the California option fails would leave both of them strictly better off.