Sometimes people say things like "If the humans and AIs have linear utility in resources, then their interactions are zero-sum". Here, "linear utility in resources" typically means something like: "Supposing AIs already control all the galaxies, then they'd accept a bet with a 60% chance of gaining one more galaxy and a 40% chance of losing one."
I think this is too hasty.
I'll list several reasons why interactions can be positive-sum even when both humans and AIs have such preferences. I've ordered these from most important to least important.
1. Epistemic public goods. Humans and AIs both benefit from learning true things about the universe — knowledge will have value to almost any agent regardless of its terminal goals. Hence, if both parties need to invest resources in acquiring knowledge, they can share the costs and both come out ahead. This includes basic mathematics and science. But I also expect this will include very expensive simulations of counterfactual worlds.
2. Security public goods. Some expenditures protect everyone, e.g., both humans and AIs might need to spend resources ensuring their galaxies don't trigger false vacuum decay in neighbouring regions. Or defending against external threats (hostile aliens, natural disasters, exogenous risks).
3. Common values. It's possible that human utility is something like hedonium+a⋅X and AI utility is something like paperclips+b⋅X, where X is some component of value that both parties share. Then, if producing hedonium, paperclips, and X is linear in resources, then both humans and AIs have linear utility in resources. However, the game isn't zero-sum, because the X-component means total welfare increases when both parties spend more resources on the common value.
4. Different marginal rates of substitution across resource types. "Linear returns to resources" elides the fact that there are many kinds of resources. Humans and AIs might differ in how much they value proximal galaxies versus distal ones (e.g. because of time discounting), even though each party's utility function u(proximal, distal) is linear. If humans value proximal galaxies relatively more and AIs value distal galaxies relatively more, then trading proximal-for-distal makes both parties better off.
5. Complementarities in production. If humans love hedonium and AIs love paperclips, maybe we can build paperclips out of hedonium (or vice versa). I expect this kind of direct complementarity to be rare in practice, because of "tails come apart" reasoning — removing the looks-like-a-paperclip constraint probably more than doubles the hedonium you can extract from the same resources. But less extreme versions of production complementarities might exist.
6. Comparative advantage. Even when both parties are linear in the same single resource, if they differ in their relative productivity across tasks, there are gains from specialisation and trade. This is why most transactions between humans are positive-sum despite each transaction being so small that both parties' utility functions are approximately linear over the relevant range.
I'm unsure whether comparative advantage persists at cosmological timescales, where both parties are optimising with mature technology. But it plausibly matters during the transition period.
7. Gains from trade under uncertainty. If humans and AIs have different beliefs, they can make bets that both parties expect to gain from. These bets are ex post zero-sum but not ex ante zero-sum. As long as the parties disagree about probabilities, both can expect to gain from the same wager.
Sometimes people say things like "If the humans and AIs have linear utility in resources, then their interactions are zero-sum". Here, "linear utility in resources" typically means something like: "Supposing AIs already control all the galaxies, then they'd accept a bet with a 60% chance of gaining one more galaxy and a 40% chance of losing one."
I think this is too hasty.
I'll list several reasons why interactions can be positive-sum even when both humans and AIs have such preferences. I've ordered these from most important to least important.
1. Epistemic public goods. Humans and AIs both benefit from learning true things about the universe — knowledge will have value to almost any agent regardless of its terminal goals. Hence, if both parties need to invest resources in acquiring knowledge, they can share the costs and both come out ahead. This includes basic mathematics and science. But I also expect this will include very expensive simulations of counterfactual worlds.
2. Security public goods. Some expenditures protect everyone, e.g., both humans and AIs might need to spend resources ensuring their galaxies don't trigger false vacuum decay in neighbouring regions. Or defending against external threats (hostile aliens, natural disasters, exogenous risks).
3. Common values. It's possible that human utility is something like hedonium+a⋅X and AI utility is something like paperclips+b⋅X, where X is some component of value that both parties share. Then, if producing hedonium, paperclips, and X is linear in resources, then both humans and AIs have linear utility in resources. However, the game isn't zero-sum, because the X-component means total welfare increases when both parties spend more resources on the common value.
4. Different marginal rates of substitution across resource types. "Linear returns to resources" elides the fact that there are many kinds of resources. Humans and AIs might differ in how much they value proximal galaxies versus distal ones (e.g. because of time discounting), even though each party's utility function u(proximal, distal) is linear. If humans value proximal galaxies relatively more and AIs value distal galaxies relatively more, then trading proximal-for-distal makes both parties better off.
5. Complementarities in production. If humans love hedonium and AIs love paperclips, maybe we can build paperclips out of hedonium (or vice versa). I expect this kind of direct complementarity to be rare in practice, because of "tails come apart" reasoning — removing the looks-like-a-paperclip constraint probably more than doubles the hedonium you can extract from the same resources. But less extreme versions of production complementarities might exist.
6. Comparative advantage. Even when both parties are linear in the same single resource, if they differ in their relative productivity across tasks, there are gains from specialisation and trade. This is why most transactions between humans are positive-sum despite each transaction being so small that both parties' utility functions are approximately linear over the relevant range.
I'm unsure whether comparative advantage persists at cosmological timescales, where both parties are optimising with mature technology. But it plausibly matters during the transition period.
7. Gains from trade under uncertainty. If humans and AIs have different beliefs, they can make bets that both parties expect to gain from. These bets are ex post zero-sum but not ex ante zero-sum. As long as the parties disagree about probabilities, both can expect to gain from the same wager.
Thanks to Alexa Pan for discussion, inspired by Lukas Finnveden's Notes on cooperating with unaligned AIs.