#344 – Noam Brown: AI vs Humans in Poker and Games of Strategic Negotiation

Episode Summary

Executive Summary: Noam Brown explains how superhuman AI in poker and Diplomacy was achieved not by raw neural nets alone, but by combining self-play, search, regret minimization, and human data. The discussion contrasts zero-sum games with socially complex, natural-language negotiation, and argues that human-compatible planning is key for AI in real-world interaction, trust, and strategy.

Main Topics: No-Limit Poker as a benchmark for game-solving AI (Priority: 5/5): Brown describes why no-limit Texas Hold’em is harder than chess: imperfect information, bluffing, betting balance, and huge strategic space. He explains how Libratus and Pluribus achieved superhuman performance by approximating Nash equilibrium and using search. Nash equilibrium, regret minimization, and self-play (Priority: 5/5): He breaks down Nash equilibrium and counterfactual regret minimization as the core theoretical tools behind poker AIs. The idea is to learn balanced mixed strategies through self-play that remain robust even when opponents know the policy. The role of search in AI performance (Priority: 5/5): A major thesis is that search/planning is underestimated. Brown argues that search was essential in poker, chess, and Go, and that even strong neural nets are much weaker without planning at test time. From heads-up to six-player poker (Priority: 4/5): The jump from two-player to six-player poker was enabled by depth-limited search and better scaling, showing that algorithmic improvements—not just more compute—can drastically reduce training cost and broaden applicability. Diplomacy as a harder, more human game (Priority: 5/5): Diplomacy adds seven players, alliances, deception, and natural-language negotiation. Brown argues it is closer to the real world than previous benchmarks because success requires understanding human trust and cooperation, not just optimal play. Cicero: language + strategy + human compatibility (Priority: 5/5): Cicero combines a language model with strategic planning. It uses human data to anchor behavior, then conditions generation on intents derived from reinforcement learning and search, while filtering for nonsense and harmful deception. Ethics, trust, and real-world implications (Priority: 4/5): The conversation closes on risks and benefits of AI that can negotiate, persuade, and even deceive. Brown emphasizes trust, anti-AI bias, cheating detection, and the potential for AI to improve human strategy, education, and future negotiation systems.

Key Arguments: Poker AI succeeded because the core challenge was not feature extraction but balancing probabilities, bluffing frequencies, and hidden information. Nash equilibrium gives a strategy that is unbeatable in expectation in two-player zero-sum games, but this framework weakens outside that setting. Search at test time is crucial; even highly capable neural nets are far less effective without planning ahead. In poker, overbets and other seemingly “crazy” moves are not chaos for its own sake; they can be optimal because they create difficult decisions for opponents. Six-player poker remained solvable in practice because it is still highly adversarial, but Diplomacy required human data because pure self-play would learn non-human communication and coordination patterns. Cicero’s success came from combining self-play and planning with a human-anchored policy and a language model conditioned on strategic intents. In Diplomacy, trust is more important than deception; obvious lying hurts long-run performance because players stop cooperating. Human-like AI can help with training, games, and negotiation, but it also raises concerns about cheating, manipulation, and misuse in real-world interactions.

Data Points: Hands played in Libratus competition: 120,000 hands - Libratus played top human heads-up no-limit Texas Hold’em players over 20 days. Prize money at stake: $200,000 - Competition incentive pool for the 2017 Libratus match. Bot winnings in Libratus match: close to $2 million - Libratus won against professional heads-up players over the course of the match. Decision points in heads-up no-limit Texas Hold’em: 10^161 - Brown cites the scale of the game’s search space. Players in six-player poker: 6 players - Pluribus extended superhuman poker AI from heads-up to multiplayer poker. Approximate cost of Libratus final training run: $100,000 - Brown contrasts the compute cost of Libratus with later improvements. Approximate cost of Pluribus final training run: less than $150 on AWS - Algorithmic improvements and depth-limited search made Pluribus dramatically cheaper. Poker hands in a standard starting deal: 52 choose 2 = 1,326 combinations - Brown uses this to explain hidden information and hand-range reasoning. Human-card dataset for Diplomacy: 50,000 games - Used to train human-compatible behavior for Cicero. Message count in Diplomacy dataset: over 10 million messages - Large-scale language data for negotiation and persuasion modeling. Cicero evaluation size: 40 games - Bot played with real humans online in Diplomacy. Cicero ranking: 2nd out of all players with 5+ games - Performance result in the Diplomacy study. Approximate total player pool in evaluation: about 80 players - Brown describes the online Diplomacy tournament setting. Players with 5+ games in evaluation: 19 - Subset used for ranking comparison with Cicero. Overbet example: $20,000 into a $1,000 pot - Illustrates extreme betting sizes that pressured human opponents in poker. Tournament length: 20 days - Duration of the Libratus human-vs-bot match. Self-play training data source: webdiplomacy.net - Primary site supplying Diplomacy game logs for training.

Pivotal Quotes: "The whole secret lies in confusing the enemy so that he cannot fathom our real intent." — Lex Friedman (closing quote from Sun Tzu): Used as the episode’s final framing for strategy, deception, and intent. "If you play it, you are guaranteed to not lose in expectation no matter what your opponent does." — Noam Brown: Explaining Nash equilibrium in two-player zero-sum games like poker. "What we found is that the human players were really struggling with that decision, that's when I realized, oh, actually, this is maybe a good thing to do after all." — Noam Brown: Reflecting on the overbet strategy that put opponents into difficult situations.

Implications: The episode suggests the next wave of AI will pair strategic planning with human-compatible communication. This could improve games, negotiation tools, training systems, and human-AI collaboration, while also sharpening concerns about deception, cheating, and trust.