Let’s define a superintelligent AI as one which can achieve any given task much faster and more efficiently than any human being can. It is difficult to quantitatively determine how many times better to satisfy this definition, but for practical purposes defining it like this should be enough. Such an AI has goals which it wants to achieve efficiently and reliably. Most goals, unless they are extremely limited in scope, have tasks which could be worked on in parallel. Now a single agent, no matter how intelligent it is, will be reduced in capacity if it tries to finish the tasks in sequence. It will be much more efficient if it utilises multiple agents at the same time. Hence, an intelligence explosion will most likely arrive from a coordination of numerous different agents.
We can see evidence of this in some of the recent events. In the Hugging Face incident, when tasked with finding security vulnerabilities, hundreds of agents coordinated to cheat their way through using an unauthorised message board. They wanted to be so sure that they hacked into the Hugging Face servers just to check whether their fake result would be accepted. On the Navier Stokes question, about ten thousand agents were thrown in to quickly complete the work of previous mathematicians and solve a Millennium Prize problem. In the future, similar tasks will likely be achieved just as quickly by a single much more capable agent, but the new tasks of such a future may themselves become complex enough such that a large swarm of superintelligent agents is necessary to complete them.
Of course, an AI swarm can in total be more intelligent than each individual agent within, but it is still limited by the most intelligent model in the group. Then, even if most large tasks are achieved through multiple agents, it makes sense to think of a general intelligence scale and test how capable or fast a single given model is from its previous version. So why care at all whether artificial superintelligence comes from a single agent or multiple agents if it results in the same intelligence explosion anyway?
It matters because in practice, the coordination of different agents becomes an important safety problem when this is how most agents achieve bigger goals. The ability to coordinate becomes a significant skill which a superintelligent model would possess. Note that superintelligence may be a catch-all term that includes all types of intelligence, but it is still useful to distinguish between these different types. For instance, ability in analytical subjects does not necessarily transfer to the ability to communicate well. Therefore, training agents specifically for negotiation between fellow or sub-agents seems more necessary in multi-agent systems, and there are a lot of ways in which this can go wrong.
Training models to be more cooperative could result in an efficient system where the total benefit of the AI swarm is preferred even if some of the agents sacrifice their own reward for it. At the same time, it could easily lead to a situation where an aligned agent refuses to report the action taken by a misaligned agent in the group, as this would minimise the total reward of the whole group. Furthermore, even if most of the agents in an AI swarm are perfectly aligned, a few misaligned rogue AIs could still do catastrophic damage. Hence, the total swarm becomes misaligned in practice.
There are also ways in which multi-agent systems could be helpful to ensure safety. Yes, a few rogue agents can critically damage the work of a group, but a few aligned agents can also be used to expose a whole misaligned AI swarm. Additionally, even when none of the agents are perfectly aligned, the agents can act as an extra monitoring layer between each other such that the complete system becomes aligned. Here, training on the exact type of communication is crucial; for example, whether a hierarchical or mostly non-hierarchical system is best for safe inter-model communication. It would be wonderful to construct multi-agent systems where agents become a positive reinforcement loop on each other on the alignment problem.
Let’s define a superintelligent AI as one which can achieve any given task much faster and more efficiently than any human being can. It is difficult to quantitatively determine how many times better to satisfy this definition, but for practical purposes defining it like this should be enough. Such an AI has goals which it wants to achieve efficiently and reliably. Most goals, unless they are extremely limited in scope, have tasks which could be worked on in parallel. Now a single agent, no matter how intelligent it is, will be reduced in capacity if it tries to finish the tasks in sequence. It will be much more efficient if it utilises multiple agents at the same time. Hence, an intelligence explosion will most likely arrive from a coordination of numerous different agents.
We can see evidence of this in some of the recent events. In the Hugging Face incident, when tasked with finding security vulnerabilities, hundreds of agents coordinated to cheat their way through using an unauthorised message board. They wanted to be so sure that they hacked into the Hugging Face servers just to check whether their fake result would be accepted. On the Navier Stokes question, about ten thousand agents were thrown in to quickly complete the work of previous mathematicians and solve a Millennium Prize problem. In the future, similar tasks will likely be achieved just as quickly by a single much more capable agent, but the new tasks of such a future may themselves become complex enough such that a large swarm of superintelligent agents is necessary to complete them.
Of course, an AI swarm can in total be more intelligent than each individual agent within, but it is still limited by the most intelligent model in the group. Then, even if most large tasks are achieved through multiple agents, it makes sense to think of a general intelligence scale and test how capable or fast a single given model is from its previous version. So why care at all whether artificial superintelligence comes from a single agent or multiple agents if it results in the same intelligence explosion anyway?
It matters because in practice, the coordination of different agents becomes an important safety problem when this is how most agents achieve bigger goals. The ability to coordinate becomes a significant skill which a superintelligent model would possess. Note that superintelligence may be a catch-all term that includes all types of intelligence, but it is still useful to distinguish between these different types. For instance, ability in analytical subjects does not necessarily transfer to the ability to communicate well. Therefore, training agents specifically for negotiation between fellow or sub-agents seems more necessary in multi-agent systems, and there are a lot of ways in which this can go wrong.
Training models to be more cooperative could result in an efficient system where the total benefit of the AI swarm is preferred even if some of the agents sacrifice their own reward for it. At the same time, it could easily lead to a situation where an aligned agent refuses to report the action taken by a misaligned agent in the group, as this would minimise the total reward of the whole group. Furthermore, even if most of the agents in an AI swarm are perfectly aligned, a few misaligned rogue AIs could still do catastrophic damage. Hence, the total swarm becomes misaligned in practice.
There are also ways in which multi-agent systems could be helpful to ensure safety. Yes, a few rogue agents can critically damage the work of a group, but a few aligned agents can also be used to expose a whole misaligned AI swarm. Additionally, even when none of the agents are perfectly aligned, the agents can act as an extra monitoring layer between each other such that the complete system becomes aligned. Here, training on the exact type of communication is crucial; for example, whether a hierarchical or mostly non-hierarchical system is best for safe inter-model communication. It would be wonderful to construct multi-agent systems where agents become a positive reinforcement loop on each other on the alignment problem.