The Instrumental Convergence of Crowds

julius vidal

tldr: There seems to be a dynamic closely related to instrumental convergence that occurs when many agents with diverse goals interact.

Epistemic Status: I think there is a version of this that is trivial and obvious, and a version that is empirically false. But somewhere in between the two there is a useful concept.

When multiple agents with heterogenous and non conflicting goals interact, they are incentivised to collaborate on any shared sub-goals. As the number of agents and the diversity of the goals increases, the possible sub-goals that will be useful to all agents must become more and more general. This means that sufficiently large and diverse groups of agents should collaborate on instrumentally convergent subgoals.

Imagine two agents with different but non conflicting goals deciding whether or not to collaborate with each other. The best reason to do so would be if there is some action that they can take together that useful to both of them, and that they cannot do (or is harder to do) individually. Now consider that as we increase the number of agents and goals:

The number and scope of possible actions available to the collective increases because a larger collective can achieve things a smaller one can't.
The set of subgoals that will be useful to the entire collective shrinks because its a function of the intersection of the respective sets of useful subgoals of each agent.

If any subgoals remain in this intersection, they will be those that are useful for a great variety of tasks. In other words the instrumentally convergent ones.

Even if the collective cannot agree on any single shared subgoal, if the setting/agent type allows for nonlinear returns on collaboration (e.g. through emergent collective intelligence) then we should expect the general pattern to hold with large coalitions forming to pursue ambitious general sub-goals.

Relation to the instrumental convergence hypothesis

In some sense this is just a reformulation of the classic instrumental convergence hypothesis. By definition instrumentally convergent goals are ones that are useful for a wide range of end goals, which is exactly what I am claiming we should expect a sufficiently large group of agents to pursue. I think there are at least two differences between my hypothesis and the original one:

In the original IC hypothesis convergence is correlated with how long horizon/ambitious the goal is and how instrumentally rational the agent is. In this case it is correlated with the number and diversity of agents.
We might expect the convergent goals to be slightly different e.g.:
1. Collectives should want to improve their capacity to communicate and organise,
2. Resources that can be shared for free (e.g. admin credentials to the shared environment) might be more appealing to the coalition than ones that have to be split (such as money).

We might still be able to reduce these back to normal instrumental convergence, especially if we had a sufficiently good theory of hierarchical or scale-free agency. For example with respect to 1. we could think that pursuing the intersection of many diverse goals is effectively a long horizon one, and that the capacity to communicate/collaborate/negotiate is a kind of instrumental rationality. Similarly for 2. the goal to increase communication and collaboration is like the classic goal of improving cognition.

Additional notes

I am making the assumption that the goals do not conflict. If they do I would probably still expect some version of these kinds of dynamics to play out but things get a lot more complicated.
I think this holds if the setting is one where collaboration can yield nonlinear returns.
I have deliberately left things quite vague because I think this principle should generalise across quite a lot of different circumstances and models of agency, and mechanisms of collective decision making. For example in a rational economic model goods that are useful to all agents should be highly priced, in a more chaotic social model more useful subgoals should be more memetic.

Thanks to Samuel, Shashvat and Aleksi for discussions, and to RWX for the perfect setting to think and write this.

Toy idea, the anti instrumental convergence of crowds:

each agent is aware of the dynamic presented in this post
insofar as they would disprefer the collective's goals to approach a set of instrumentally convergent goals, the agents have an incentive to coordinate to stop other agents from falling into this dynamic

Iirc Audrey Tang argued for something similar in her dialogue with plex

This does seem more fragile than the original concept, and maybe a similar but opposite thing applies one recursion level up (agents affected by the described dynamic have an incentive to stop other agents from coordinating against this dynamic).

Thankfully we have some control over how we create agents and agent collectives, and I imagine there's things we can do to make it more easy and natural for agents to coordinate to stop this kind of instrumental convergence.

An example of anti-convergence is the situation of "Highlander": a war of all the immortals against all until there is only one. That One will win the Prize: all the power of all the Immortals who ever lived, giving the ability to dominate all of mankind and rule the future.

Ted Chiang's story "Understand" takes the same view: two superintelligent humans cannot coexist, but one must destroy the other and rule the future alone.

When comicbook supervillains conspire with each other, the same dynamic typically obtains: all unity falls into defection. I am not a regular reader of comics, but it seems that superheroes have had much greater success in cooperating, notably in the Justice League and the X-Men. But there's a story idea, if no-one has yet filled this much-needed gap.

These are allegories and prophesies for the AI companies of the present day. They are racing for exactly the Prize described above. (But in Singularity, Prize wins you.)

Perhaps the only reason we cooperate at all is that on our own, each of us can accomplish little beyond bare survival for a limited time. Given superpowers great enough to eliminate dependence on anyone else for anything, how would we actually relate to each other? How would you?

ETA: Related sayings:

"Three may keep a secret, if two of them are dead."
"All the world is queer save thee and me, and even thou art a little queer."

ETA2: To put this another way, “I alone shall rule!” is not an anti-convergent goal, but a convergent one, indexed on the one having the goal. But all goals are so indexed—goals are had by individuals, whether the goals are cooperative or competitive.