663: Astonishing CICERO negotiates and builds trust with humans using natural language

Episode Summary

Executive Summary: This episode explores Meta’s Cicero, an AI agent that mastered the complex negotiation game Diplomacy by combining planning, reinforcement learning, and grounded language generation. Alex Holden Miller explains why Diplomacy is a uniquely hard benchmark, how Cicero was built to play convincingly against humans, and why the work points toward future grounded, goal-oriented AI agents in games, virtual worlds, and assistants.

Main Topics: Why Diplomacy is a uniquely hard AI benchmark: Alex explains Diplomacy as a mix of Risk, poker, and Survivor: hidden information, simultaneous moves, strategic alliance-building, and betrayal make it far harder than chess or Go for AI. Cicero’s no-press and full-press versions: The discussion contrasts the simpler no-press variant with the full-language version, showing how the team first solved strategic play without dialogue before tackling natural-language negotiation. Architecture: planning plus grounded language generation: Cicero uses a modular pipeline: strategic planning models, human-behavior regularization, plan inference, and a transformer-based encoder-decoder language model that generates grounded messages. Why self-play alone was insufficient: Alex argues that multiplayer settings with cooperation require training on human data and regularizing toward human-like behavior; pure self-play produces exploitable or unnatural strategies. Research management, team building, and the Meta FAIR environment: Alex describes how his role as a senior research engineering manager emphasizes helping people succeed, coordinating cross-functional launch work, and combining scientific and engineering talent. Future applications beyond Diplomacy: The episode closes on broader implications for grounded agents in games, metaverse interactions, NPCs, and other virtual assistants that need to act strategically rather than just generate plausible text. Tools and workflow in AI research: The team largely uses Python and PyTorch, with minimal abstraction, because research requires control over model behavior, training, and experimentation.

Key Arguments: Diplomacy is a superior AI benchmark because it combines incomplete information, simultaneous actions, coalition management, and adversarial negotiation. Self-play alone is not enough in multiplayer cooperative/adversarial environments; models must be regularized toward human behavior to remain effective against people. Planning and language generation should be decoupled: generate strategic plans first, then condition the language model on those plans to keep dialogue grounded and legal. Human data improves both strategy and language quality by teaching the model what actions and utterances are plausible in context. A grounded agent can outperform a generic LLM in task-specific settings because it has explicit goals, context, and constraints. Cicero’s success suggests a path toward AI systems that can negotiate, collaborate, and act in dynamic real-world-like environments. Research management in AI is about assembling the right mix of talent, compute, and data while helping researchers succeed technically and professionally.

Data Points: Project duration: About 3.5 years - Alex says the Diplomacy project started roughly three and a half years before the interview. Expected timeline: 10 years - The team initially believed Diplomacy might take a decade to solve. Board-game players: 7 players - Standard Diplomacy games involve seven powers at the table. Turn timer: 15 minutes - In-person tournaments typically give players 15 minutes per turn. No-press tournament turn timer: 5 minutes - The tournament for the no-press bot used five-minute turns. Evaluation volume: 40 games - Cicero was evaluated across 40 games against humans. Gameplay duration: About 72 hours - Those 40 games amounted to roughly 72 hours of total gameplay. Message count: 6,000-7,000 messages - The paper reports Cicero sent thousands of messages during evaluation. Performance ranking: Top decile - Cicero performed in the top 10% of the human players it faced. Board-state scale: More possible board states than Go relative to chess - Alex emphasizes Diplomacy’s enormous state space; exact count not given, but it exceeds Go’s relative complexity over chess. Action-space scale: 10-20 possible actions per unit - Each unit has a large branching factor, contributing to the game’s complexity.

Pivotal Quotes: "The biggest recent achievement in AI is an algorithm called Cicero that was developed by researchers working at the tech giant Meta." — John Crohn: John frames the episode by elevating Cicero above other recent AI breakthroughs. "The real hard part is now do all of that while having strategic, convincing conversations with people that can go basically as fast as you can send messages back and forth." — Alex Holden Miller: Alex explains why full-press Diplomacy is harder than the no-press version. "By doing this, then you actually get this very powerful language model." — Alex Holden Miller: Alex summarizes the benefit of combining planning, human regularization, and grounded language generation.

Implications: Cicero shows that AI can move beyond fluent text into grounded, strategic, goal-driven interaction. For industry, this points to better game agents, NPCs, and virtual assistants; for research, it highlights the value of combining planning, RL, and language models.