tldr: There seems to be a dynamic closely related to instrumental convergence that occurs when many agents with diverse goals interact.
Epistemic Status: I think there is a version of this that is trivial and obvious, and a version that is empirically false. But somewhere in between the two there is a useful concept.
When multiple agents with heterogenous and non conflicting goals interact, they are incentivised to collaborate on any shared sub-goals. As the number of agents and the diversity of the goals increases, the possible sub-goals that will be useful to all agents must become more and more general. This means that sufficiently large and diverse groups of agents should collaborate on instrumentally convergent subgoals.
Imagine two agents with different but non conflicting goals deciding whether or not to collaborate with each other. The best reason to do so would be if there is some action that they can take together that useful to both of them, and that they cannot do (or is harder to do) individually. Now consider that as we increase the number of agents and goals:
The number and scope of possible actions available to collective increases because a larger collective can achieve things a smaller one can't.
The number of actions useful to the entire collective shrinks because its a function of the intersection of the respective sets of useful subgoals of each agent.
If any subgoals remain in this intersection, they will be those that are useful for a great variety of tasks. In other words the instrumentally convergent ones.
Even if the collective cannot agree on any single shared subgoal, if the setting/agent type allows for nonlinear returns on collaboration (e.g. through emergent collective intelligence) then we should expect the general pattern to hold with large coalitions forming to pursue ambitious general sub-goals.
Relation to the instrumental convergence hypothesis
In some sense this is just a reformulation of the classic instrumental convergence hypothesis. By definition instrumentally convergent goals are ones that are useful for a wide range of end goals, which is exactly what I am claiming we should expect a sufficiently large group of agents to pursue. I think there are at least two differences between my hypothesis and the original one:
In the original IC hypothesis convergence is correlated with how long horizon/ambitious the goal is and how instrumentally rational the agent is. In this case it is correlated with the number and diversity of agents.
We might expect the convergent goals to be slightly different e.g.:
Collectives should want to improve their capacity to communicate and organise,
Resources that can be shared for free (e.g. admin credentials to the shared environment) might be more appealing to the coalition than ones that have to be split (such as money).
We might still be able to reduce these back to normal instrumental convergence, especially if we had a sufficiently good theory of hierarchical or scale-free agency. For example with respect to 1. we could think that pursuing the intersection of many diverse goals is effectively a long horizon one, and that the capacity to communicate/collaborate/negotiate is a kind of instrumental rationality. Similarly for 2. the goal to increase communication and collaboration is like the classic goal of improving cognition.
Additional notes
I am making the assumption that the goals do not conflict. If they do I would probably still expect some version of these kinds of dynamics to play out but things get a lot more complicated.
I think this holds if the setting is one where collaboration can yield nonlinear returns.
I have deliberately left things quite vague because I think this principle should generalise across quite a lot of different circumstances and models of agency, and mechanisms of collective decision making. For example in a rational economic model goods that are useful to all agents should be highly priced, in a more chaotic social model more useful subgoals should be more memetic.
Thanks to Samuel, Shashvat and Aleksi for discussions, and to RWX for the perfect setting to think and write this.
tldr: There seems to be a dynamic closely related to instrumental convergence that occurs when many agents with diverse goals interact.
Epistemic Status: I think there is a version of this that is trivial and obvious, and a version that is empirically false. But somewhere in between the two there is a useful concept.
When multiple agents with heterogenous and non conflicting goals interact, they are incentivised to collaborate on any shared sub-goals. As the number of agents and the diversity of the goals increases, the possible sub-goals that will be useful to all agents must become more and more general. This means that sufficiently large and diverse groups of agents should collaborate on instrumentally convergent subgoals.
Imagine two agents with different but non conflicting goals deciding whether or not to collaborate with each other. The best reason to do so would be if there is some action that they can take together that useful to both of them, and that they cannot do (or is harder to do) individually. Now consider that as we increase the number of agents and goals:
If any subgoals remain in this intersection, they will be those that are useful for a great variety of tasks. In other words the instrumentally convergent ones.
Even if the collective cannot agree on any single shared subgoal, if the setting/agent type allows for nonlinear returns on collaboration (e.g. through emergent collective intelligence) then we should expect the general pattern to hold with large coalitions forming to pursue ambitious general sub-goals.
Relation to the instrumental convergence hypothesis
In some sense this is just a reformulation of the classic instrumental convergence hypothesis. By definition instrumentally convergent goals are ones that are useful for a wide range of end goals, which is exactly what I am claiming we should expect a sufficiently large group of agents to pursue. I think there are at least two differences between my hypothesis and the original one:
We might still be able to reduce these back to normal instrumental convergence, especially if we had a sufficiently good theory of hierarchical or scale-free agency. For example with respect to 1. we could think that pursuing the intersection of many diverse goals is effectively a long horizon one, and that the capacity to communicate/collaborate/negotiate is a kind of instrumental rationality. Similarly for 2. the goal to increase communication and collaboration is like the classic goal of improving cognition.
Additional notes
Thanks to Samuel, Shashvat and Aleksi for discussions, and to RWX for the perfect setting to think and write this.