An inside look at X’s Community Notes | Keith Coleman (VP of Product) and Jay Baxter (ML Lead)

Episode Summary

Executive Summary: The episode explores how X’s Community Notes became a scalable, trusted, open system for adding context to misleading posts. Keith Coleman and Jay Baxter explain the core algorithm, why bridging agreement across disagreeing users drives neutrality, and how small, focused, self-selecting teams plus strong principles enabled the product to survive leadership changes and influence internet misinformation at massive scale.

Main Topics: What Community Notes is and why it works: Community Notes lets users add contextual notes to posts; notes become visible when people who usually disagree both rate them as helpful, producing neutral, accurate context instead of one-sided moderation. The bridging-agreement algorithm: The team explains that the algorithm looks for surprising agreement across polarized users rather than simple majority vote, using ML techniques and conservative thresholds to prevent biased or manipulated notes. Scale, impact, and measurable effects: The product now handles hundreds of notes per day, hundreds of thousands of contributors, and tens of billions of views, with evidence that notes reduce engagement and resharing of misleading content. Origin story and internal experimentation: Community Notes began as a response to misinformation problems at Twitter, first as Birdwatch, then evolved through prototypes, pilots, MTurk-style testing, and an internal ML bake-off before becoming a live system. Team structure and operating model: The project succeeded as a small thermal team: one clear owner, one senior decision-maker, 100% focus, self-selection into the team, and lightweight coordination via a single Google Doc instead of heavy process. Principles: transparency, openness, and no company override: The system is designed to represent the voice of the people, not the company; code and data are open, notes are not arbitrarily removed by staff, and all posts—including those by elites and advertisers—are eligible. Future: AI-assisted and community-built notes: The team sees future promise in AI tools that help contributors write faster, and in external collaboration like ‘super notes,’ where public models and open-source methods help improve the system.

Key Arguments: Community Notes works because it uses disagreement-bridging, not majority rule, to surface notes that are neutral and broadly useful. The product is not fact-checking in the traditional sense; it adds context and lets users decide for themselves. Every post should be eligible for notes—no exemptions for politicians, Elon, or advertisers—because the system should treat everyone equally. Open source transparency is essential to trust; people should be able to inspect the code, data, and behavior of the system. A conservative bar is necessary: it is better to hide a borderline-good note than show a bad one that could damage trust. Small, fully focused, self-selecting teams can outperform larger org structures when the mission is clear and the feedback loop is fast. The notes’ effect is not just informational; they measurably reduce reposting and resharing, slowing the spread of misinformation. Anonymity/pseudonymity increases honesty and willingness to cross partisan lines, improving note quality and participation. The product survived multiple leadership transitions because the core principles were strong and the output was visibly useful to users across the spectrum.

Data Points: Notes shown per day: Hundreds - Current scale of Community Notes output across topics and posts. Traditional fact checks per day: About 10 - Cited comparison from UC Berkeley research to show the higher throughput of Community Notes. Notes shown in 2024: 95,000 - Number of notes seen on X in 2024. Views of notes in 2024: About 30 billion - Total times 2024 notes were seen. Prior-year notes seen: About 37,000 - 2023 comparison figure referenced in the discussion. Prior-year views: About 14 billion - 2023 comparison figure referenced in the discussion. Contributors around the world: About 950,000 - Current contributor base nearing one million. Share of proposed notes shown: About 8% - Approximate proportion of submitted notes that ultimately appear publicly. Helpful threshold: 0.4 on an arbitrary internal scale - Described as the conservative cutoff in the algorithm. Engagement drop in A/B test: 30% to 40% - Likes and reposts dropped when a note was shown versus not shown. Total repost reduction from external studies: 50% to 60% - Measured network effect on resharing after notes are applied. Post deletion likelihood after being noted: 80% more likely - Authors were more likely to delete posts after receiving a note. Time to note during Israel-Hamas conflict: Median of 5 hours - Median delay from post publication to note appearance in early October 2023. Speed improvement before conflict: About 3 hours faster - A recent launch cut three hours off the time needed for notes to go live. Team size at start: About 5 people - Initial thermal team composition: backend, frontend, ML, design, research. Pilot audience: 1,000 people - Early public pilot scale before broader rollout. RAM needed to run open-source replication: About 500 GB - Open-source algorithm is runnable on one machine with enough memory.

Pivotal Quotes: "We actually look for agreement from people who have disagreed in the past." — Jay Baxter: Explaining the core bridging-based algorithm behind Community Notes. "We want all of humanity to participate." — Keith Coleman: Describing the philosophy behind open participation and contributor diversity. "If there’s a problem with a note, it’s so bad you want to do something about it. It’s a problem with the system." — Keith Coleman: Explaining why the team refuses arbitrary overrides and holds a high quality bar.

Implications: The episode argues that misinformation can be addressed at internet scale without central gatekeeping if systems are transparent, conservative, and designed for cross-partisan consensus. It also suggests small, mission-driven teams can build outsized products with lasting social impact.