This work was facilitated by the Oxford AI Safety and Governance group, Cooperative AI Foundation, and Oxford Autonomous Intelligent Machines and Systems. Thanks also to Bart Jaworski, Jesse Clifton, Joar Skalse, Sam Barnett, Vincent Conitzer, Charlie Griffin, David Hyland, Michael Wooldridge, Ted Turocy, and Alessandro Abate.

This blogpost accompanies the paper Cooperation and Control in Delegation Games by Sourbut, Hammond, and Wood, which was presented at IJCAI 2024. In essence, the work attempts to deconfuse some of the discourse around safety, cooperation, and alignment in multi-agent settings by:

Showing that, just like in the control problem, cooperation problems can be broken down into alignment and capabilities, which are orthogonal to one another;
Providing measures for alignment and capabilities (both “individual” and “collective”) in multi-principal multi-agent settings (“delegation games”);
Showing that any of these measures is insufficient alone to guarantee the best outcomes, but that they are together sufficient;
Bounding the principals’ welfare loss in terms of these measures, and validating this with a series of empirical results.

The goal of this post is to explain what those terms mean, and hopefully why it matters. In doing so, we hope to shed light on some of the related questions posed by other AI safety researchers, for example Dafoe et al in Open Problems in Cooperative AI who discuss the concept of ‘horizontal’ and ‘vertical’ aspects of coordination, or an open problem on the AI Alignment Forum about quantifying player alignment in normal-form games.

There is also a poster which is an even more condensed summary of some key material, and Lewis and Oly have given a few presentations on the topic, one of which is recorded here.

Why Delegation Games?

You may have heard of the Principal-Agent problem. It's a phrase and a setting which turns up in some economics literature, and elsewhere. The idea is that a principal (in this case, the human) is asking, telling, employing, or otherwise exhorting an agent (in this case, the robot) to act on their behalf. The 'problem' is the question of how to ensure that the agent's behaviour results in outcomes which the principal in fact prefers.

Delegation games arise when we have multiple principals, and multiple agents.^[1] When you read 'principal', think 'human', and when you read 'agent', think 'AI...

Posts

Wikitag Contributions

Comments

Posts

Wikitag Contributions

Comments