Key links?

• Here's the paper: https://www.science.org/doi/10.1126/science.ade9097
• Here's the blog post: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/
• Here's the website: https://ai.facebook.com/research/cicero/diplomacy/

What is Diplomacy?

• Diplomacy is a negotiation-based board game. There are 7 players corresponding to the 7 major European powers during WW1. In each round, the players write down their moves, which are then executed simultaneously. The moves might be attacking/defending territory, supporting/opposing another player's action, etc.
• Unlike Risk, there are no dice or other random mechanism.
• In "Full Press Diplomacy", players can also hold private conversations between rounds. The dialogue is used to establish trust and coordinate actions with other players. Players can make agreements, but all these agreements are non-binding.
• In "No Press Diplomacy", players can't communicate.
• Because the moves are simultaneous, game-play requires recursive theory of mind. The players must reason about the other players, and reason about what others are reasoning about them, and so on.
• Diplomacy is notorious for ending friendships, probably because it erodes trust. Anecdotally, I've played Diplomacy twice in my life. Each game lasted at least 6 hours. The first game was played with slightly LessWrong-ish friends and wasn't friendship-destroying. The second was played with un-LessWrong-ish friends and it did cause outside-the-game distrust.
• Diplomacy was the favourite game of John F. Kennedy and Henry Kissinger.

How well did CICERO perform?

• On Nov 22 (today!) Meta AI presented an AI which achieved human-level performance at Full Press Diplomacy.
• CICERO achieved more than double the average score of the other players and ranked in the top 10% of players.
• The evaluation consisted of anonymous blitz Diplomacy games against humans on webDiplomacy.net
  • 40 games
  • 82 human players
  • 5,277 messages
  • 72 hours of gameplay
• Here is commentary by a Diplomacy expert playing against six CICEROs: https://www.youtube.com/watch?v=u5192bvUS7k
• "CICERO is so effective at using natural language to negotiate with people in Diplomacy that they often favoured working with CICERO over other human participants." (blog)
• They are open-sourcing the code and models, and "interested researchers can submit a proposal to the CICERO RFP to gain access to the data." (?!!)

How does CICERO behave?

• In each game, the other players were convinced this was a human player.

• "What impresses me most about CICERO is its ability to communicate with empathy and build rapport while also tying that back to its strategic objectives." — Andrew Goff (3x Diplomacy World Champion)
• "CICERO is ruthless. It's resilient. And it's patient. [...] CICERO's dialogue is direct, but it has some empathy. It's surprisingly human." — Andrew Goff
• Apparently, CICERO is more honest than most human players. Andrew Goff also suggests that CICERO learned to become more honest over time, which is the same learning curve for high-level human players.
• CICERO (in green) de-escalates with another player by reassuring them it will not attack them. 

• CICERO (in blue) suggests mutually beneficial moves a human missed.

• CICERO can both coordinate and negotiate.

• Sometimes CICERO generates inconsistent dialogue and makes mistakes. For example, CICERO (below) contradicts its first message asking Italy to move to Venice.

How does CICERO work?

• CICERO has two components: a strategy engine and a dialogue engine.

• The strategy engine constructs a plan which works for itself and the other players and then the dialogue engine persuades the other players of the plan.
• "To build a controllable dialogue model, we started with a 2.7 billion parameter BART-like language model pre-trained on text from the internet and fine-tuned on over 40,000 human games on webDiplomacy.net." (blog)
• They automatically labelled the messages in this training set with the corresponding planned action. The labels were used as control tokens for the language model. During games, these control tokens are submitted by the strategy engine.
• Erroneous messages are filtered out using various methods.

How does CICERO compare to prior models?

• No-Press Diplomacy was solved by Deepmind in 2020. Learning to Play No-Press Diplomacy with Best Response Policy Iteration
• Human-Level Performance in No-Press Diplomacy via Equilibrium Search (SearchBot played anonymously on webDiplomacy.net and ranked in the top 2 percent of players.)
• No-Press Diplomacy from Scratch
• Other games:
  • Superhuman chess was achieved in 1997.
  • Superhuman Go was achieved in 2016.
  • Superhuman poker was achieved in 2019.

Were people surprised by CICERO?

• Here is Yorum Bachrach (from the DeepMind team that solved No-Press Diplomacy) talking about Full-Press Diplomacy in February 2021:
  
  "For Press Diplomacy, as well as other settings that mix cooperation and competition, you need progress,” Bachrach says, “in terms of theory of mind, how they can communicate with others about their preferences or goals or plans. And, one step further, you can look at the institutions of multiple agents that human society has. All of this work is super exciting, but these are early days.”
• Here are related Metaculus and Manifold predictions: