Section II.1 of CLR’s safe Pareto improvements (SPI) agenda briefly reviews the challenges for making SPIs individually rational. Here, I’ll flesh out these challenges, taking renegotiation SPIs as a running example. The ideas here overlap a lot with DiGiovanni et al. (2024) and its distillation. But this post will hopefully be more accessible, and will consider bargaining problems with non-simultaneous commitments. I’ll also revisit some controversial arguments made in those previous writings.
My aim is to precisely yet informally answer: “Why might an agent expect opting out of SPIs to have better consequences? What are the mechanisms leading them to expect this?” The high-level mechanisms I’ll discuss apply to a variety of SPIs, not just renegotiation, though the details importantly differ between different SPIs. I’m not confident that this list of mechanisms is exhaustive. Rather, the point is to show that at least these obstacles need to be overcome in order for SPIs to be individually rational.
Summary: Consider some bargaining interaction between Alice and Bob. When would Bob prefer to agree to a given SPI with Alice? Here’s a sketch of what this post will argue:
Typically, SPIs work by keeping each agent’s demands the same as if the other agent hadn’t participated in the SPI, holding fixed their chosen bargaining commitments. But this isn’t sufficient for Bob to prefer agreeing to the SPI. Bob might expect that if he commits to refuse SPIs, he can get a better deal, by either (more):
(i) removing Alice’s incentive to choose a more demanding commitment because she expects the SPI to make bargaining failure less costly;
(ii) signaling to Alice that he’d make aggressive demands; or
(iii) refusing this particular SPI to get Alice to accept his preferred SPI (the SPI selection problem).
Problem (i) can be blocked if, at the time the agents bargain, it’s verifiable to Bob that Alice satisfies a weak form of participation independence (PI): her demands are no higher than those of the commitment she’d have chosen had Bob committed to unconditionally refuse SPIs. (more)
But even if Alice satisfies weak PI, problem (ii) remains: If Bob is still uncertain whether Alice is committed to weak PI when he chooses his own commitment, he might commit to unconditionally refuse SPIs to signal aggressiveness. (That said, this incentive appears relatively conjunctive, once we dig into the details. And it can be blocked if Alice makes her commitment to weak PI credible to Bob before he commits.) (more)
Even if problems (i) and (ii) are solved, the SPI selection problem (iii) remains. While there are some proposed solutions — in particular, DiGiovanni et al.’s (2024) “Pareto meet minimum”-based approach, and Oesterheld & Conitzer’s (2022) — I don’t think any are very robust, based on our current understanding. So research into mitigations of the SPI selection problem seems particularly important. (more)
Setup
Recap: Renegotiation SPIs
We’ll focus on SPIs implemented via renegotiation programs, and for simplicity we’ll just consider two agents. Renegotiation programs roughly work like this: By default, use whichever bargaining strategy you like — the same strategy regardless of whether the other agent also uses a renegotiation program. But, if the other agent also uses a renegotiation program, and your programs agree on some outcome that (weakly) Pareto-improves on the default outcome, take that outcome instead.
More precise definition. As discussed here, a bargaining program is a commitment to behavior in a bargaining problem conditional on the other agent’s program/commitment. Now, take any “default” program that an agent might use. The agent chooses some renegotiation function, which, conditional on the other agent’s renegotiation function, maps each possible bargaining outcome to a weak Pareto improvement. Then, they can construct a corresponding renegotiation program via the algorithm below:[1]
We’ll say that two programs “renegotiate” with each other if they each run line 6 of Algorithm 1 (or, an algorithm that matches Algorithm 1 except that line 5 includes additional conditions besides matching proposals;[2] this possibility will be important later). In that case, we call the outcome the programs go to the renegotiation outcome.
SPIs via renegotiation programs. Recall the agenda’s definition of SPIs (with new notation):
AnSPIis a transformationsuch that, for all [program pairs]in some space, the agents’ payoffs [in the given game] when they all follow programs(weakly) Pareto-dominate their payoffs when they all follow.
Then we have a straightforward way to construct SPIs, given credible conditional commitment ability: Take some renegotiation function . For any given pair of programs the agents might use, map to the pair of Algorithm 1 instances with and . This mapping is an SPI, simply because maps any given outcome to a Pareto improvement.
“Default demands” of renegotiation programs. The hard part is to argue that the agents individually prefer to use such an SPI. Going forward, we’ll look at how the agents’ preferences to renegotiate depend on the demands they expect each other to make. For convenience, we’ll let an agent’s “demands” — as a quantity — refer to the payoff they get if the counterpart concedes to their demands-in-the-qualitative-sense, e.g., “give me 60% of the pie”. Then, for a renegotiation program Alice might use and any program Bob might use, let ’s default demands be the demands that makes when the two programs run against each other. See footnote for more.[3]
Before the bargaining phase (recall the definition here), Alice commits to renegotiation, meaning, she commits to the constraint that will be a renegotiation program.
Bob might choose before or after Alice commits to renegotiation. We don’t assume Bob knows Alice is committed to renegotiation when he chooses .
Then: Under which conditions would Bob prefer to use athat ends up renegotiating with Alice?
Naïvely, we might think the following argument is enough: Against any non-renegotiating program Bob might use, Alice’s renegotiation program plays (line 10 of Algorithm 1). Whereas against a renegotiation program with , her default program runs against (line 2), i.e., it again plays . So her default demands are exactly the same whether or not Bob renegotiates.[4] And the renegotiation function will make neither agent worse off. So wouldn’t Bob always prefer to have be willing to renegotiate?
But that’s too quick. We’ve only established that Bob prefers to renegotiate holding two things fixed: Alice’s default demands against both renegotiators and non-renegotiators, and the outcome they’d renegotiate to. What if Bob is uncertain of these default demands when choosing , and expects that his choice could change them? And what if Bob insists on a different renegotiation proposal (line 4 of Algorithm 1) from Alice’s?
To crudely summarize, Bob might choose a program that doesn’t renegotiate with Alice’s if he expects any of the following (not necessarily exhaustive):[5]
If he’s committed not to renegotiate, then:
Alice will lose the incentive to make high demands that renegotiation would’ve otherwise given her; or
Alice will take the information “Bob doesn’t renegotiate” as a signal that Bob will make high demands (and hence she won’t make high demands herself).
If he’s committed to renegotiate only to outcomes that are sufficiently favorable to him, then: Alice’s program will renegotiate to a better outcome for Bob than if he’d been willing to renegotiate to less favorable outcomes.
We’ll run through different cases to see when each of these possibilities might (not) apply. (See footnote for some notes on scope.[6])
First possibility: Preempting the incentive to make higher demands
To start, suppose Bob decides whether to credibly commit to never renegotiate. We’ll call the set of programs under this constraint . Then one possible scenario is:
Bob thinks Alice’s default demands will be higher in expectation if his program is sometimes willing to renegotiate than if he commits to .[7] Specifically, he expects that both:
Alice might verify whether he’s committed to before she commits to her default demands;[8] and
If he doesn’t commit to : Alice can afford the risk of committing to high default demands (against both renegotiators and non-renegotiators). This is because she expects Bob might renegotiate, and thus the outcome of incompatible demands won’t be as bad. So, she’ll choose a default program that makes higher demands than she’d choose if Bob had committed to .
Bob therefore commits toto preempt such higher demands.
Various writings about SPIs have discussed similar problems before. But it’s important to spell this out carefully, because in my experience, looser discussions of this problem have led to confusions about the conditions necessary to avoid it.[9]
More general formulation. This first possibility encompasses any case where:
Bob makes a commitment that leads him to not renegotiate with Alice (not just ); and
Bob chooses this commitment for the following reason (at least partly): He thinks Alice will have an incentive to make higher demands because the SPI reduces her risk from higher demands, and he wants to remove this incentive, to the extent that he can do so.
When is this a problem?
Intuitively: At least as far as this first possibility goes, Bob only wants to opt out of renegotiation insofar as this makes a difference to Alice’s default demands . So, suppose Bob can commit to only renegotiate ifAlice’s demands are no higher than if Bob had committed to never renegotiate. Then, in principle, Bob no longer has the above incentive to actually commit to never renegotiate. To be a bit more precise, let:
be the set of Bob programs that renegotiate only if , where
are the demands Alice would have made, had Bob committed to , i.e., to the analogous program in . (More details on what “would have” means in Appendix III.)
Then, the option of might block this first possibility: If Bob commits to , then by construction, he only renegotiates when he can verify that Alice’s demands aren’t worse for him than under . And, Bob would plausibly believe the following about Alice: She’ll anticipate Bob’s incentive to require weak PI. So, she won’t expect violating weak PI to pay off because renegotiation makes conflict less costly, since violating weak PI would disqualify her from renegotiation anyway. (Cf. the “non-punishment assumptions” discussed in Part II.1 of the agenda.[11]) So Bob won’t prefer over for reason (2) listed in the “More general formulation” above.
However, suppose that at the time of bargaining, Alice can’t make credible that satisfies weak PI. Then even if Bob uses , he’ll still end up refusing to renegotiate with her. In practice, it might be infeasible for Alice to make weak PI credible, because this requires certifying a counterfactual.
Second possibility: Signaling aggressiveness
(This is less well-studied than the other mechanisms in this post, so take the following analysis as more uncertain.)
Above, we considered Alice’s incentive to make higher demands due to the SPI lowering the risk of such demands. Verifiable weak PI suffices for that problem. Here’s another worry: Suppose that when Bob chooses his commitment, he doesn’t yet know whether satisfies weak PI. Then from his perspective, Alice might still make higher demands if he commits to rather than (or some other commitment more stringent than ), based on what each commitment signals about him. This could give him a reason not to use , even if he could verify weak PI at the time of bargaining.
Specifically, suppose Bob expects Alice to learn whether he’s committed to or before she decides whether to commit to weak PI. Then Bob might imagine that, if he commits to , Alice will reason like this:
“What if I’d seen that Bob still wouldn’t renegotiate, even if I satisfy weak PI ()? That would be evidence that he’s irrational: he’d give up the opportunity to avoid pointless conflict, when this makes no difference to my demands.
“In that case, I’d be more worried about him rigidly committing to high demands, and commit to low demands myself (i.e., low ).
“But I know Bob is willing to sometimes renegotiate ()! So I’d rather violate weak PI and make higher demands than . Seems worth the risk that we end up fighting.”
So Bob might expect that committing to will signal that he’ll make high demands.
This isn’t to say the signal is strong evidence, or that it clearly goes in a direction that favors committing to . For example, insofar as Bob would consider it rational to make this commitment based on the argument above, then arguably, committing to can’t be strong evidence of “irrationality” in the first place! Or, committing to might just as well signal that Bob is unreasonably uncooperative, leading other agents to not trade with him. My claim is only that some agents might have an incentive to use for signaling purposes, if they’re willing to risk costly conflict.
When is this a problem?
The root of this problem is Bob’s uncertainty about whether Alice is committed to weak PI. If he decides whether to commit to afterverifying that Alice has committed to weak PI, he has noincentive to commit to for signaling purposes. This is because in this case, he knows Alice would never demand less against than against , i.e., she wouldn’t respond favorably to the signal. But Bob might still prefer to commit to if he decides his commitment before verifying Alice’s commitment to weak PI. The analysis of when and why he might do so, in light of this post, is left as an exercise to the reader.
Third possibility: The SPI selection problem
So far, we’ve implicitly only considered a single SPI, namely, the mapping from to programs of the form in Algorithm 1 where:
and are the agents’ respective default programs;
line 5 of the algorithm also checks whether the other agent’s program satisfies weak PI; and
the agents’ renegotiation functions return matching proposals (line 5).
But, as originally discussed in Oesterheld & Conitzer (2022, Sec. 6), there can be a bargaining problem over multiple SPIs: the “SPI selection problem”. Specifically, Alice’s and Bob’s renegotiation functions might propose different outcomes, and their programs won’t necessarily agree on a proposal. The mechanistic explanation for this is very similar to the explanation for miscoordination in the agents’ original bargaining problem: Suppose Bob’s program conditions its proposed renegotiation outcome on incompatibility with Alice’s, i.e., backs down when their proposals don’t match. Then Alice’s program can insist on an outcome that’s better for her (and worse for Bob) risk-free. (And vice versa.) See also the table below.
There are a variety of commitments Bob could make to certain demands, which Alice would prefer to agree to rather than have bargaining fail — if she hasn’t committed otherwise.
There are a variety of commitments Bob could make to certain renegotiation outcomes, which Alice would prefer to agree to rather than forgo the benefit of renegotiation — if she hasn’t committed otherwise.
But Alice might commit to not agree to those demands, because she expects Bob might be willing to back down to lower demands in response to her commitment.
But Alice might commit to not agree to those renegotiation outcomes, because she expects Bob might be willing to back down to a renegotiation outcome that’s better for her in response to her commitment.
→ If Bob is committed to not back down, bargaining fails.
→ If Bob is committed to not back down, Bob doesn’t renegotiate.
When is this a problem?
It’s an open question whether, and how, the SPI selection problem can be solved or at least mitigated. Here’s a (non-exhaustive) survey of responses to the problem.
Alice verifiably commits “first”
Of course, if Alice commits to only accept a particular renegotiation outcome, and Bob verifies this before making an incompatible commitment, there’s no SPI selection problem. But the whole motivation for SPIs in the first place is that Alice and Bob might blindly commit to incompatible demands. So “just commit first” doesn’t seem to engage with the problem.[12]
Set-valued renegotiation and the “Pareto meet minimum”
Suppose, as in DiGiovanni et al. (2024), that each agent’s program can propose a set of renegotiation outcomes (see Algorithm 2 in Appendix IV). And suppose:
Alice and Bob can at least agree on a function that selects an outcome from the intersection of their proposed sets (a “selection function”). So their programs renegotiate to that selected outcome, if the intersection is nonempty.
The agents’ beliefs satisfy assumptions similar to those we invoked to argue that Bob might as well use rather than .
They choose programs simultaneously.
Then, we can argue the following: Under the programs Alice and Bob individually prefer, they’ll at least renegotiate to an outcome no worse for either agent than the Pareto meet minimum (PMM), i.e., the payoff pair where each agent’s payoff is that of their least-preferred Pareto-efficient outcome. See Appendix IV for a (mostly) informal, corrected version of the paper’s argument for this result. Very briefly, the intuition is: Even though Bob has an incentive to insist on the best renegotiation outcome for himself, the PMM isn’t better for Alice than that outcome (by definition). So if he offers the PMM as well, Alice has no incentive to refuse his preferred outcome in order to get the PMM. (And the same holds switching the roles of Alice and Bob.)
But since writing that paper, I’m now significantly more worried that its argument for assumption (1) above doesn’t work (thanks to Nathaniel Sauerberg for raising this issue). I unpack this more in Appendix V, but roughly: The paper argues that agents can get the same outcome under any selection function by jointly switching their renegotiation sets. But we need each agent to expect the same outcomes by individually switching their set, and this seems to require strong assumptions.
[P]layers can profit from the use of SPIs even without addressing this SPI selection problem. To do so, a player picks an instruction that is very compliant (‘dove-ish’) w.r.t. what SPI is chosen, e.g., one that simply goes with whatever SPI the other players demand as long as that SPI cannot further be safely Pareto-improved upon.
In the context of renegotiation, such a “dove-ish” program would be a renegotiation program that matches whichever proposal Bob’s renegotiation function insists on, conditional on Bob satisfying weak PI.
To assess this idea, it’s helpful to separate two claims: 1) “Alice individually prefers to use a dove-ish program over any other renegotiation program.” 2) “Alice individually prefers to use a dove-ish program over a non-renegotiation program. And using a dove-ish program would significantly mitigate conflict, without being equivalent to just conceding to Bob’s demands.”
(1) seems false, for precisely the reasons given in the table above. So in that sense, dove-ish programs aren’t a solution to SPI selection.
But this need not imply that (2) is false. In particular, if Alice uses a dove-ish program, this would mitigate conflict without her needing to concede more than if she had never renegotiated.
Concluding thoughts on SPI selection
SPI selection currently seems like a serious conceptual challenge for SPIs. For example, it’s plausible that in order for the PMM argument to work, we’d need to basically assume away a fundamental bargaining problem (over selection functions). Another approach might be to argue that certain ways of resolving SPI selection are focal points. But I also find this unsatisfying, because the motivation for SPIs is that agents might fail to coordinate on a focal point in the first-order bargaining problem. (Though, coordinating on an SPI might still be easier in degree, even if the qualitative problem is the same as in first-order bargaining.)
However, I’m cautiously optimistic about a few directions for progress here.
First, even if DiGiovanni et al.’s argument as a whole fails, the argument that the PMM avoids perverse incentives does still seem sound. That is, the properties of the PMM itself are doing some work in the argument. So my intuition is that something like the PMM result might be provable under weaker assumptions than “the agents solve a bargaining problem that’s just as hard as the original problem”.
Second (h/t Nathaniel Sauerberg): Consider SPIs implemented via delegating to an agent with different goals, but with the twist that this delegate can itself implement another delegation-based SPI. Then the delegate themselves faces an SPI selection problem, not just the original agent. So if SPI selection breaks down, the downsides might fall on the delegate’s goals rather than the original agent’s. (There’s a lot more to say on this idea, which I leave to future work, but that’s the basic sketch.)
Lastly, a promising approach is to have “our” AIs err somewhat on the side of caution at the level of choosing an SPI, while still allowing them to be more or less assertive in first-order bargaining. Dove-ish renegotiation is one instance of this. While it’s possible that we’ll conclude this strategy is too conservative, it seems well-motivated under severe uncertainty about what kinds of SPIs others might accept.
Acknowledgments
Thanks to Caspar Oesterheld, Matt Hampton, Paul Knott, Jesse Clifton, Cadence James, Neil Crawford, Julia Karbing, Nathaniel Sauerberg, Tristan Cook, and Claude for comments.
Appendix I: Notation table
Notation
Meaning
,
The programs of Alice and Bob, respectively.
Agent ’s default program: the bargaining program that ’s renegotiation program runs unless it renegotiates (see Algorithm 1).
Agent ’s renegotiation function, which, conditional on the other agent’s renegotiation function, maps each possible bargaining outcome to a weak Pareto improvement.
The demands that ’s default program makes against , where is a renegotiation program.
The set of Bob’s programs that never renegotiate.
The demands Alice would have made had Bob committed to the program in analogous to .
The set of Bob’s programs that renegotiate only if — i.e., only if satisfies weak participation independence (PI).
Appendix II: How the possibilities in this post map onto the agenda’s Risks 1–3
Recall the distinction between the analysis of Alice’s incentive to inflate demands and her possible responses to signaling.
Possibility / Case
Risks covered
Preempting inflated demands
Risks 1 & 2, incentive type.
Alice can make weak PI verifiable at bargaining time
Risk 1, incentive type: resolved — weak PI is the agenda’s “natural solution to Risk 1”.
Risk 2, incentive type: (arguably) resolved — since Bob doesn’t renegotiate if weak PI is violated, opting-out counterparts don’t have an incentive to bargain harder due to expecting renegotiation to make conflict less costly (cf. the non-punishment assumptions).
Alice cannot make weak PI verifiable at bargaining time
Risks 1 & 2, incentive type: live — Bob can’t verify weak PI, so he might worry that Alice’s demands will be inflated and refuse to renegotiate.
Signaling aggressiveness
Risks 1 & 2, signaling type.
Bob verifies that Alice is committed to weak PI before deciding whether to commit not to renegotiate
Risk 1, signaling type: resolved — an Alice already committed to weak PI can’t respond to the signal. (This only pertains to Risk 1, because Alice is presumably only motivated to commit to weak PI if she wants to participate in the SPI.)
Bob verifies that Alice is committed to weak PI after deciding whether to commit not to renegotiate
Risks 1 & 2, signaling type: live — since Bob doesn’t know whether Alice has committed to weak PI, he may expect a signaling benefit (whether or not she’d participate in the SPI).
SPI selection
Risk 3.
General case
Risk 3: live — there’s currently no general, robust solution to SPI selection.
Appendix III: More on the PI counterfactual
What does it mean to check the counterfactual condition in the definition of weak PI? It’s a bit of an open question how to make counterfactuals precise in general. But here’s one sketch, which I think is fairly standard:
We can think of each possible world as a diagram of variables that stand in causal relations to each other.
When there’s an arrow from variable to , we say is the “parent” of .
When there’s some sequence of arrows from to , we say is “causally downstream” of .
For each variable, some causal laws map the values of its parent variables to that variable’s value. Parentless variables just directly have their own values.
Let be the actual world.
Suppose, to start, that Bob follows causal decision theory.
Whenever Bob looks at a counterfactual like “if [agent] had [done ]”, roughly, he looks at the possible world constructed as follows:
The variable corresponding to [agent]’s actions is set to the value .
For every variable that isn’t causally downstream of :
That variable in takes the same value as in .
For every variable that is causally downstream of :
That variable in takes whichever value is output by applying to the values of its parent variables.
(More in-the-weeds, not essential to follow unless you’ll do research on SPIs involving agents with acausal decision theories:) What if Bob follows an acausal decision theory? This is where things get trickier to specify, but:
Suppose Bob has some notion of logical parents and downstreamness, distinct from the causal notion, capturing intuitions like “the decision made by a simulation of me is logically downstream of my decision”. We augment the diagram with arrows encoding these relations.
And we extend the causal laws to “logi-causal” laws . For each variable , maps the values of all of ’s causal and logical parents to ’s value. If has no logical parents, this mapping is just the original laws .
Given this, would be constructed just like above, but substituting “logically or causally downstream” for “causally downstream”, and substituting for .
Appendix IV: Revised definition of “PMP-extension” in DiGiovanni et al. (2024), and informal argument for the PMM bound
This appendix presents an updated version of the section “Pareto meet projections (PMP)” from this post, to address the problem discussed in this comment. I’m not highly confident this updated version avoids that problem either (even setting aside the selection function problem). Importantly, the following analysis assumes the agents choose programs simultaneously.
First, to recap from the post:
For any game, consider the set of outcomes where each player’s payoff is at least as good as their least-preferred Pareto-efficient outcome — Rabin (1994) calls this set the Pareto meet. … The Pareto meet minimum (PMM) is the Pareto-worst payoff tuple in the Pareto meet. [See the star symbols in Fig. 5 below.]
…
For a given player and any feasible payoff , we define the PMP of for player , denoted , as follows: maps any outcome to the set of Pareto improvements on such that, first, each player’s payoff is at least the PMM, and second, the payoff of the counterpart is not increased except up to the PMM.[13] This definition is perhaps best understood visually:
Figure 5: Representation of the PMP of four different outcomes (filled circles). Blue continuous line: PMP for Alice, red dashed line: PMP for Bob. [The x-axis is Alice’s payoff, y-axis is Bob’s.]
…
Selection function. Recall that if the agreement set (intersection of both [players’] renegotiation sets) contains several outcomes, a selection function is used to select one of them. For our PMM bound result, we require that the selection function select a Pareto-efficient point among points in the agreement set, and satisfy the following transitivity property: If outcomes are added to the agreement set that make all players weakly better off than [under the outcome selected from the original agreement set], the selected renegotiation outcome should be weakly better for all players.
Now consider conditional set-valued renegotiation (CSR) programs, defined as follows, where “select” is a transitive selection function:
We’ll define modifications to CSR programs that add the PMP, as follows:
Take any pair of CSR programs with set-valued renegotiation functions , whose default programs would play some outcome (line 2 of Algorithm 2). We call the outcome that would be obtained by that pair the default renegotiation outcome (either line 6 or line 8, depending on the renegotiation sets). (See empty circles in Fig. 6 below.)
Now take any set-valued renegotiation function . The PMP-extended, denoted , is the function that returns, for a given input :
, if the other agent’s renegotiation function for some ;
, otherwise.
(We call the “base” renegotiation function of . If some is not PMP-extended, then the base of is itself.)
The PMP-extension of a program is the new program with the same default program, but whose renegotiation function is the PMP-extended version of the original program’s. Fig. 6 below illustrates this graphically: On both panels, the blue and dotted red lines are Alice’s and Bob’s PMPs, respectively.
Figure 6: Two possible renegotiation procedures, corresponding to two different renegotiation sets for Bob. Alice adds her PMP (blue line) of the outcome that would otherwise be achieved (empty circle) to her set of renegotiation outcomes. Adding her PMP conditional on Bob doing so never makes Alice worse off, and makes Bob better off in the case of the left plot. Symmetrically, Bob is never worse off adding his PMP (red dashed line).
How do PMP-extensions help the agents achieve the PMM or better?
Suppose Alice considers whether to (i) use some base renegotiation function or (ii) use its PMP-extended version. As illustrated by Fig. 6, if we hold fixed the default renegotiation outcome (empty circle) between (i) and (ii), it’s clear that (ii) is weakly better for her (arrow).[14] The default renegotiation outcome is indeed fixed ifwe assume Bob’s base function responds the same to Alice’s extended function as to her base, and vice versa.
Therefore, Alice PMP-extending her renegotiation function never makes her worse off under that assumption. If we set aside signaling incentives, this assumption seems justified because the addition of the PMP is conditional:
On Bob’s side:
On one hand, if Bob’s renegotiation function is itself PMP-extended, the assumption holds by construction. Bob’s base function only takes as input Alice’s base, so it can’t respond differently to Alice’s PMP-extended function. (This is analogous to how, when renegotiation programs evaluate the default outcome, the default programs only take each other as inputs; see line 2 of Algorithm 2.)
On the other hand, suppose Bob’s renegotiation function isn’t PMP-extended, and his function responds differently to Alice’s PMP-extended function. Then her renegotiation set against him would not include the PMP. So, he’d be disqualifying himself from any benefit of responding differently in the first place (same argument as above, again setting aside signaling).
On Alice’s side: The argument immediately above applies just as well here, with Alice’s and Bob’s roles swapped. That is, Alice has no incentive to use a base that responds differently to Bob’s PMP-extended function than to his base.
And why only add the PMP, rather than a larger set? Suppose Alice also added some outcome strictly better for Bob than her PMP — hence, strictly worse for Alice than her most-preferred outcome in her PMP-extension . Then, if Bob’s renegotiation set also included , the selection function might choose , leaving Alice worse off than if she got .
By a symmetric argument, Bob is never worse off PMP-extending his renegotiation function (adding the red dotted line in Fig. 6), under the same assumption as above. And if both Bob and Alice add their PMP, they are guaranteed the PMM (left plot of Fig. 6) or Pareto-better (right plot of Fig. 6).
Appendix V: More on the selection function problem
For simplicity, let’s consider a setting with two possible outcomes the agents might include in their renegotiation sets: , which is better for Alice, and , which is better for Bob. And suppose that if both outcomes are in the intersection of their sets, selects , while selects .
For any outcome attainable under , that outcome is attainable under , via the agents jointly switching their renegotiation sets.
So, Alice shouldn’t expect different outcomes under different selection functions. Hence she might as well agree to use .
But that argument needs an extra step, like: “Suppose Alice will likely get under . If she agrees to and individually switches her set, then she is still likely to get .” And the worry is that this step is implausible, because it seems harder for Alice to get under than under . Specifically, under , she needs to drop from her set. Bob might respond to that by dropping .
Why might Bob’s set-valued renegotiation function drop in response to Alice’s dropping ? I’m not confident in this, but here’s one possibility (h/t Opus 5.5):
By hypothesis, Bob likely includes in his set under . Why would he do this? One reason is, he might think Alice is more likely to exclude from her set if he excludes than if he includes it.[15]
So, it’s plausible that Bob might follow that kind of policy himself under , i.e., be more likely to exclude from his set if she excludes than if she includes it.
This isn’t to say we should expect agents to reason symmetrically in bargaining problems, in general. I’m only making the more modest claim that in order for DiGiovanni et al.’s argument to go through, we’d need to expect Bob not to adopt the kind of policy he expects from Alice.
This algorithm is similar to DiGiovanni et al.’s (2024) “conditional set-valued renegotiation”, but without the “set-valued” part. The “conditional” part will be sufficient to convey the points in the main text of this post. ↩︎
Specifically, line 5 would be of the following form, for some conditions and : “if my_proposal = their_proposal and and then”. ↩︎
More formally: Let be the payoff Alice receives from taking some bargaining action if Bob concedes to the qualitative demand specified by . Then if is not a renegotiation program (see line 10 of Algorithm 1), or if is a renegotiation program (line 2). (Payoffs in this sense need not be interpreted as “utility” in the precise sense from expected utility theory. In general, the analysis in this post doesn’t assume the agents are (explicit) EU maximizers. Rather, just represents how Alice ranks outcomes in some broadly consequentialist way.) ↩︎
I’ve deliberately used a different breakdown from the agenda’s “Risks 1-3”, to make it easier to see Bob’s motivations for ignoring SPIs. (More in Appendix II.) The breakdown isn’t exhaustive because, e.g., in principle Bob might expect benefits (from opting out of renegotiation) other than lowering other agents’ demands or getting a better renegotiation outcome, and this post’s three mechanisms might not exhaust “lowering other agents’ demands or getting a better renegotiation outcome”. ↩︎
First, I’m setting aside the question of whether/when Alice herself would prefer to commit to renegotiation. Second, for simplicity, I’ll assume the agents involved follow a causal, updateful decision theory. But the analysis broadly goes through for acausal and/or updateless decision theories if, instead of saying (e.g.) “Alice verifies Bob’s commitment before deciding what to commit to”, we say something like “Alice doesn’t condition her choice of commitment on any predictions that Bob might consider (decision-relevantly) dependent on his choice of commitment”. ↩︎
Where the “expectation” is with respect to Bob’s uncertainty over both Alice’s choice of program and his own choice of program (which isn’t fully determined by whether he commits to ). As in “Taboo ‘equilibrium’”, this need not refer to literal expected values; Bob’s beliefs might be more vague than that. ↩︎
Or at least, she might gain some information about whether he’s made this commitment. ↩︎
For example, we’ll see below that we don’t need to invoke any references to Alice making higher demands than if “SPIs weren’t a thing”. Relatedly, in some discussions, people define “SPIs” such that they avoid this problem. Consider, e.g., a perspective on which the point of an SPI is to make the agents better off when it’s “exogenously” introduced to them, and they respond to this information. We might define an “exogenous” SPI as follows: a transformation such that, for any way of choosing a program pair in depending on whether or not the agents know the transformation is available, the agents’ payoffs under the programs they choose knowing the transformation is available (weakly) Pareto-dominate their payoffs under the problems they choose not knowing it’s available. But the (“endogenous”) definition of SPIs we’re using in this post doesn’t immediately avoid the given problem. ↩︎
As opposed to PI per se, which holds only under the equality. ↩︎
Note: The argument here doesn’t require these specific assumptions as stated in DiGiovanni et al. (2024); it only requires the more modest assumption stated in the above argument. ↩︎
That said, one possibility is that Alice can make her first-order demands verifiable to Bob before he makes any incompatible commitments, but she doesn’t want to risk doing so, because she isn’t very confident Bob won’t in fact make incompatible commitments first. In this case, she might still prefer to commit early to a particular renegotiation outcome. While this, too, is risky if it turns out Bob is committed to an incompatible renegotiation outcome, she isn’t running the larger risk of locking in first-order demands. ↩︎
This is where the assumption that the selection function is transitive comes in. The outcomes Alice’s PMP adds are weakly better for both agents than the empty circle, so the selected outcome is too. ↩︎
On the other hand, he might simply think Alice is likely to exclude no matter what. ↩︎
(Soft prerequisites: A high-level model of AI bargaining and Taboo “equilibrium”: Less confused frames for research on AI bargaining.)
Section II.1 of CLR’s safe Pareto improvements (SPI) agenda briefly reviews the challenges for making SPIs individually rational. Here, I’ll flesh out these challenges, taking renegotiation SPIs as a running example. The ideas here overlap a lot with DiGiovanni et al. (2024) and its distillation. But this post will hopefully be more accessible, and will consider bargaining problems with non-simultaneous commitments. I’ll also revisit some controversial arguments made in those previous writings.
My aim is to precisely yet informally answer: “Why might an agent expect opting out of SPIs to have better consequences? What are the mechanisms leading them to expect this?” The high-level mechanisms I’ll discuss apply to a variety of SPIs, not just renegotiation, though the details importantly differ between different SPIs. I’m not confident that this list of mechanisms is exhaustive. Rather, the point is to show that at least these obstacles need to be overcome in order for SPIs to be individually rational.
See Appendix I for a table of this post’s notation, and Appendix II for how the analysis maps onto the agenda’s breakdown of “Risks 1-3”.
Summary: Consider some bargaining interaction between Alice and Bob. When would Bob prefer to agree to a given SPI with Alice? Here’s a sketch of what this post will argue:
Setup
Recap: Renegotiation SPIs
We’ll focus on SPIs implemented via renegotiation programs, and for simplicity we’ll just consider two agents. Renegotiation programs roughly work like this: By default, use whichever bargaining strategy you like — the same strategy regardless of whether the other agent also uses a renegotiation program. But, if the other agent also uses a renegotiation program, and your programs agree on some outcome that (weakly) Pareto-improves on the default outcome, take that outcome instead.
More precise definition. As discussed here, a bargaining program is a commitment to behavior in a bargaining problem conditional on the other agent’s program/commitment. Now, take any “default” program
We’ll say that two programs “renegotiate” with each other if they each run line 6 of Algorithm 1 (or, an algorithm that matches Algorithm 1 except that line 5 includes additional conditions besides matching proposals; [2] this possibility will be important later). In that case, we call the outcome the programs go to the renegotiation outcome.
SPIs via renegotiation programs. Recall the agenda’s definition of SPIs (with new notation):
Then we have a straightforward way to construct SPIs, given credible conditional commitment ability: Take some renegotiation function
“Default demands” of renegotiation programs. The hard part is to argue that the agents individually prefer to use such an SPI. Going forward, we’ll look at how the agents’ preferences to renegotiate depend on the demands they expect each other to make. For convenience, we’ll let an agent’s “demands” — as a quantity — refer to the payoff they get if the counterpart concedes to their demands-in-the-qualitative-sense, e.g., “give me 60% of the pie”. Then, for a renegotiation program
Problem setup
We’ll use the framework from “A high-level model of AI bargaining”. Let
Then: Under which conditions would Bob prefer to use a
Naïvely, we might think the following argument is enough: Against any non-renegotiating program
But that’s too quick. We’ve only established that Bob prefers to renegotiate holding two things fixed: Alice’s default demands against both renegotiators and non-renegotiators, and the outcome they’d renegotiate to. What if Bob is uncertain of these default demands when choosing
To crudely summarize, Bob might choose a program that doesn’t renegotiate with Alice’s if he expects any of the following (not necessarily exhaustive): [5]
We’ll run through different cases to see when each of these possibilities might (not) apply. (See footnote for some notes on scope. [6] )
First possibility: Preempting the incentive to make higher demands
To start, suppose Bob decides whether to credibly commit to never renegotiate. We’ll call the set of programs under this constraint
Various writings about SPIs have discussed similar problems before. But it’s important to spell this out carefully, because in my experience, looser discussions of this problem have led to confusions about the conditions necessary to avoid it. [9]
More general formulation. This first possibility encompasses any case where:
When is this a problem?
Intuitively: At least as far as this first possibility goes, Bob only wants to opt out of renegotiation insofar as this makes a difference to Alice’s default demands
If
Then, the option of
However, suppose that at the time of bargaining, Alice can’t make credible that
Second possibility: Signaling aggressiveness
(This is less well-studied than the other mechanisms in this post, so take the following analysis as more uncertain.)
Above, we considered Alice’s incentive to make higher demands due to the SPI lowering the risk of such demands. Verifiable weak PI suffices for that problem. Here’s another worry: Suppose that when Bob chooses his commitment, he doesn’t yet know whether
Specifically, suppose Bob expects Alice to learn whether he’s committed to
So Bob might expect that committing to
This isn’t to say the signal is strong evidence, or that it clearly goes in a direction that favors committing to
When is this a problem?
The root of this problem is Bob’s uncertainty about whether Alice is committed to weak PI. If he decides whether to commit to
Third possibility: The SPI selection problem
So far, we’ve implicitly only considered a single SPI, namely, the mapping from
But, as originally discussed in Oesterheld & Conitzer (2022, Sec. 6), there can be a bargaining problem over multiple SPIs: the “SPI selection problem”. Specifically, Alice’s and Bob’s renegotiation functions might propose different outcomes, and their programs won’t necessarily agree on a proposal. The mechanistic explanation for this is very similar to the explanation for miscoordination in the agents’ original bargaining problem: Suppose Bob’s program conditions its proposed renegotiation outcome on incompatibility with Alice’s, i.e., backs down when their proposals don’t match. Then Alice’s program can insist on an outcome that’s better for her (and worse for Bob) risk-free. (And vice versa.) See also the table below.
When is this a problem?
It’s an open question whether, and how, the SPI selection problem can be solved or at least mitigated. Here’s a (non-exhaustive) survey of responses to the problem.
Alice verifiably commits “first”
Of course, if Alice commits to only accept a particular renegotiation outcome, and Bob verifies this before making an incompatible commitment, there’s no SPI selection problem. But the whole motivation for SPIs in the first place is that Alice and Bob might blindly commit to incompatible demands. So “just commit first” doesn’t seem to engage with the problem. [12]
Set-valued renegotiation and the “Pareto meet minimum”
Suppose, as in DiGiovanni et al. (2024), that each agent’s program can propose a set of renegotiation outcomes (see Algorithm 2 in Appendix IV). And suppose:
Then, we can argue the following: Under the programs Alice and Bob individually prefer, they’ll at least renegotiate to an outcome no worse for either agent than the Pareto meet minimum (PMM), i.e., the payoff pair where each agent’s payoff is that of their least-preferred Pareto-efficient outcome. See Appendix IV for a (mostly) informal, corrected version of the paper’s argument for this result. Very briefly, the intuition is: Even though Bob has an incentive to insist on the best renegotiation outcome for himself, the PMM isn’t better for Alice than that outcome (by definition). So if he offers the PMM as well, Alice has no incentive to refuse his preferred outcome in order to get the PMM. (And the same holds switching the roles of Alice and Bob.)
But since writing that paper, I’m now significantly more worried that its argument for assumption (1) above doesn’t work (thanks to Nathaniel Sauerberg for raising this issue). I unpack this more in Appendix V, but roughly: The paper argues that agents can get the same outcome under any selection function by jointly switching their renegotiation sets. But we need each agent to expect the same outcomes by individually switching their set, and this seems to require strong assumptions.
“Dove-ish” renegotiation
Finally, consider this approach from Oesterheld & Conitzer (2022, Sec. 6):
In the context of renegotiation, such a “dove-ish” program would be a renegotiation program that matches whichever proposal Bob’s renegotiation function insists on, conditional on Bob satisfying weak PI.
To assess this idea, it’s helpful to separate two claims: 1) “Alice individually prefers to use a dove-ish program over any other renegotiation program.” 2) “Alice individually prefers to use a dove-ish program over a non-renegotiation program. And using a dove-ish program would significantly mitigate conflict, without being equivalent to just conceding to Bob’s demands.”
Concluding thoughts on SPI selection
SPI selection currently seems like a serious conceptual challenge for SPIs. For example, it’s plausible that in order for the PMM argument to work, we’d need to basically assume away a fundamental bargaining problem (over selection functions). Another approach might be to argue that certain ways of resolving SPI selection are focal points. But I also find this unsatisfying, because the motivation for SPIs is that agents might fail to coordinate on a focal point in the first-order bargaining problem. (Though, coordinating on an SPI might still be easier in degree, even if the qualitative problem is the same as in first-order bargaining.)
However, I’m cautiously optimistic about a few directions for progress here.
First, even if DiGiovanni et al.’s argument as a whole fails, the argument that the PMM avoids perverse incentives does still seem sound. That is, the properties of the PMM itself are doing some work in the argument. So my intuition is that something like the PMM result might be provable under weaker assumptions than “the agents solve a bargaining problem that’s just as hard as the original problem”.
Second (h/t Nathaniel Sauerberg): Consider SPIs implemented via delegating to an agent with different goals, but with the twist that this delegate can itself implement another delegation-based SPI. Then the delegate themselves faces an SPI selection problem, not just the original agent. So if SPI selection breaks down, the downsides might fall on the delegate’s goals rather than the original agent’s. (There’s a lot more to say on this idea, which I leave to future work, but that’s the basic sketch.)
Lastly, a promising approach is to have “our” AIs err somewhat on the side of caution at the level of choosing an SPI, while still allowing them to be more or less assertive in first-order bargaining. Dove-ish renegotiation is one instance of this. While it’s possible that we’ll conclude this strategy is too conservative, it seems well-motivated under severe uncertainty about what kinds of SPIs others might accept.
Acknowledgments
Thanks to Caspar Oesterheld, Matt Hampton, Paul Knott, Jesse Clifton, Cadence James, Neil Crawford, Julia Karbing, Nathaniel Sauerberg, Tristan Cook, and Claude for comments.
Appendix I: Notation table
Appendix II: How the possibilities in this post map onto the agenda’s Risks 1–3
Recall the distinction between the analysis of Alice’s incentive to inflate demands and her possible responses to signaling.
Appendix III: More on the PI counterfactual
What does it mean to check the counterfactual condition in the definition of weak PI? It’s a bit of an open question how to make counterfactuals precise in general. But here’s one sketch, which I think is fairly standard:
Whenever Bob looks at a counterfactual like “if [agent] had [done
The variable
For every variable that isn’t causally downstream of
For every variable that is causally downstream of
Appendix IV: Revised definition of “PMP-extension” in DiGiovanni et al. (2024), and informal argument for the PMM bound
This appendix presents an updated version of the section “Pareto meet projections (PMP)” from this post, to address the problem discussed in this comment. I’m not highly confident this updated version avoids that problem either (even setting aside the selection function problem). Importantly, the following analysis assumes the agents choose programs simultaneously.
First, to recap from the post:
Now consider conditional set-valued renegotiation (CSR) programs, defined as follows, where “select” is a transitive selection function:
We’ll define modifications to CSR programs that add the PMP, as follows:
How do PMP-extensions help the agents achieve the PMM or better?
Suppose Alice considers whether to (i) use some base renegotiation function or (ii) use its PMP-extended version. As illustrated by Fig. 6, if we hold fixed the default renegotiation outcome (empty circle) between (i) and (ii), it’s clear that (ii) is weakly better for her (arrow). [14] The default renegotiation outcome is indeed fixed if we assume Bob’s base function responds the same to Alice’s extended function as to her base, and vice versa.
Therefore, Alice PMP-extending her renegotiation function never makes her worse off under that assumption. If we set aside signaling incentives, this assumption seems justified because the addition of the PMP is conditional:
On Bob’s side:
On Alice’s side: The argument immediately above applies just as well here, with Alice’s and Bob’s roles swapped. That is, Alice has no incentive to use a base that responds differently to Bob’s PMP-extended function than to his base.
And why only add the PMP, rather than a larger set? Suppose Alice also added some outcome
By a symmetric argument, Bob is never worse off PMP-extending his renegotiation function (adding the red dotted line in Fig. 6), under the same assumption as above. And if both Bob and Alice add their PMP, they are guaranteed the PMM (left plot of Fig. 6) or Pareto-better (right plot of Fig. 6).
Appendix V: More on the selection function problem
For simplicity, let’s consider a setting with two possible outcomes the agents might include in their renegotiation sets:
The argument from DiGiovanni et al. (2024), in this setting (see its Appendix B):
But that argument needs an extra step, like: “Suppose Alice will likely get
Why might Bob’s set-valued renegotiation function drop
This algorithm is similar to DiGiovanni et al.’s (2024) “conditional set-valued renegotiation”, but without the “set-valued” part. The “conditional” part will be sufficient to convey the points in the main text of this post. ↩︎
Specifically, line 5 would be of the following form, for some conditions
More formally: Let
That is, renegotiation programs satisfy demand preservation. ↩︎
I’ve deliberately used a different breakdown from the agenda’s “Risks 1-3”, to make it easier to see Bob’s motivations for ignoring SPIs. (More in Appendix II.) The breakdown isn’t exhaustive because, e.g., in principle Bob might expect benefits (from opting out of renegotiation) other than lowering other agents’ demands or getting a better renegotiation outcome, and this post’s three mechanisms might not exhaust “lowering other agents’ demands or getting a better renegotiation outcome”. ↩︎
First, I’m setting aside the question of whether/when Alice herself would prefer to commit to renegotiation. Second, for simplicity, I’ll assume the agents involved follow a causal, updateful decision theory. But the analysis broadly goes through for acausal and/or updateless decision theories if, instead of saying (e.g.) “Alice verifies Bob’s commitment before deciding what to commit to”, we say something like “Alice doesn’t condition her choice of commitment on any predictions that Bob might consider (decision-relevantly) dependent on his choice of commitment”. ↩︎
Where the “expectation” is with respect to Bob’s uncertainty over both Alice’s choice of program and his own choice of program (which isn’t fully determined by whether he commits to
Or at least, she might gain some information about whether he’s made this commitment. ↩︎
For example, we’ll see below that we don’t need to invoke any references to Alice making higher demands than if “SPIs weren’t a thing”. Relatedly, in some discussions, people define “SPIs” such that they avoid this problem. Consider, e.g., a perspective on which the point of an SPI is to make the agents better off when it’s “exogenously” introduced to them, and they respond to this information. We might define an “exogenous” SPI as follows: a transformation such that, for any way of choosing a program pair
As opposed to PI per se, which holds only under the equality. ↩︎
Note: The argument here doesn’t require these specific assumptions as stated in DiGiovanni et al. (2024); it only requires the more modest assumption stated in the above argument. ↩︎
That said, one possibility is that Alice can make her first-order demands verifiable to Bob before he makes any incompatible commitments, but she doesn’t want to risk doing so, because she isn’t very confident Bob won’t in fact make incompatible commitments first. In this case, she might still prefer to commit early to a particular renegotiation outcome. While this, too, is risky if it turns out Bob is committed to an incompatible renegotiation outcome, she isn’t running the larger risk of locking in first-order demands. ↩︎
Formally,
This is where the assumption that the selection function is transitive comes in. The outcomes Alice’s PMP adds are weakly better for both agents than the empty circle, so the selected outcome is too. ↩︎
On the other hand, he might simply think Alice is likely to exclude