More Victories, Less Cooperation: Assessing Cicero's Diplomacy Play

Wichayaporn Wongkamjan, Feng Gu, Yanze Wang, Ulf Hermjakob, Jonathan May, Brandon M. Stewart, Jonathan K. Kummerfeld, Denis Peskoff, Jordan Lee Boyd-Graber · 2024

Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 12423-12441, 2024.

Generative AI & LLMs Prediction & Algorithmic Systems

Preprint Publisher's Version

Figure 1 from More Victories, Less Cooperation showing how initial orders, communication, and final orders are used to detect deception and persuasion in Diplomacy.

Abstract

The boardgame Diplomacy is a challenging setting for communicative and cooperative artificial intelligence. The most prominent communicative Diplomacy AI, Cicero, has excellent strategic abilities, exceeding human players. However, the best Diplomacy players master communication, not just tactics, which is why the game has received attention as an AI challenge. This work seeks to understand the degree to which Cicero succeeds at communication. First, we annotate in-game communication with abstract meaning representation to separate in-game tactics from general language. Second, we run two dozen games with humans and Cicero, totaling over 200 human-player hours of competition. While AI can consistently outplay human players, AI-Human communication is still limited because of AI's difficulty with deception and persuasion. This shows that Cicero relies on strategy and has not yet reached the full promise of communicative and cooperative AI.

Related Publications

[Paper] Fine-Tuned Large Language Models Can Replicate Expert Coding Better Than Trained Coders: A Study on Informative Signals Sent by Interest Groups
[Conference Proceedings] GPT Deciphering Fedspeak: Quantifying Dissent Among Hawks and Doves
[Conference Proceedings] Credible without Credit: Domain Experts Assess Generative Language Models
[Conference Proceedings] EgoPolice: A Benchmark for Egocentric Video Understanding in High-Stakes Police Body-Worn Camera Footage
[Working Paper] Where's the Evidence that Respondents Understand Your Survey Questions?
[Paper] State Media Control Influences Large Language Models
[Conference Proceedings] STATe-of-Thoughts: Structured Action Templates for Tree-of-Thoughts
[Paper] Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses