Don't hate the replicator, hate the game

Episode Summary

Executive Summary: The episode follows economist Abel Brodeur’s response to the replication crisis in social science. After discovering how incentives can push researchers toward p-hacking and fragile findings, he built the Institute for Replication and the “Replication Games,” a crowdsourced audit system where teams check published papers’ code, data, and robustness. The story shows both the seriousness of errors in published work and how transparency and monitoring can change research norms.

Main Topics: The replication crisis in social science: The episode frames a long-running credibility problem across psychology, medicine, and economics: many published findings cannot be reproduced when data and code are rechecked. Abel Brodeur’s personal encounter with p-hacking: Brodeur’s master’s thesis on smoking bans revealed how easy it was to manipulate analysis until a statistically significant result appeared, leading him to question academic incentives. Incentives in academic publishing: Career advancement depends on publishing novel, statistically significant results, which can encourage selective analysis, trimming, and subconscious manipulation of data. Building a surveillance system for science: Brodeur created the Institute for Replication and the Replication Games to make researchers think their work may be checked at any time, shifting norms through uncertainty and monitoring. How replication games work in practice: Teams of economists and social scientists spend a day reproducing code and testing robustness, sometimes finding major data issues, sometimes confirming papers, often learning better research practices. What replication failures mean for journals and authors: The episode argues that journals often miss errors and that replication findings should be treated as system failures, not necessarily accusations of fraud; authors usually get a chance to respond first.

Key Arguments: Academic incentives reward significance and novelty more than accuracy, which helps explain why fragile results get published. P-hacking can be conscious or subconscious: over years of small analytic decisions, researchers may end up with results that look stronger than the underlying evidence supports. Monitoring changes behavior more effectively than severe punishment alone; the chance of being checked can encourage cleaner analysis and more transparent research. Replication is not about catching villains as much as creating a reliable system for producing knowledge. Most replication findings uncover errors, missing data, or robustness problems rather than outright fraud, but those failures still matter because published science shapes policy and future research. Crowdsourcing replication through events like the Replication Games makes large-scale auditing feasible at relatively low cost. Even when a paper “passes,” the exercise can improve practices by teaching participants better coding, documentation, and robustness habits.

Data Points: Significance threshold: 5% - A result is typically considered statistically significant if it would occur by chance less than 5% of the time. Confidence threshold: 95% - The episode explains the common 95%/5% cutoff that strongly influences publication decisions. Teams at Montreal replication game: 16 teams - Abel Brodeur hosted 16 teams working on reproducing recently published papers. Participants in Montreal replication game: 71 replicators - The Montreal event involved 71 people checking published research. Brodeur’s replication output so far: About 300 papers - By the time of the episode, the Replication Games had replicated roughly 300 papers across 50+ events. Number of replication games: More than 50 - Brodeur has organized over 50 replication games by this stage. Oslo workshop registration surge: 70-80 people - A planned small workshop unexpectedly drew roughly 70 to 80 registrants. Paper finding in the example replication: -18.11432 vs. -18.114 - A team reproduced a result from an agriculture/egg-pricing paper and matched the published coefficient almost exactly. Data anomaly in cartel paper: 1 cartel removal changes significance - Replicators found that excluding one cartel made the paper’s results insignificant. Montreal event duration: 7 hours - Teams had seven hours to replicate and assess their assigned papers.

Pivotal Quotes: "It needs to be a big institution with a website with tons of famous people on it." — Abel Brodeur: Brodeur explains how he created the Institute for Replication to make requests for code and data feel official and harder to ignore. "It's like the IRS for the Ivory Tower." — Abel Brodeur: He describes the goal of making academic work feel continuously auditable, like taxes are for finances. "Just the idea that someone might walk through their apartment one day, that's enough of a threat to keep it clean." — Abel Brodeur: He argues that monitoring, not just punishment, is what changes behavior in research communities.

Implications: The episode suggests science becomes more trustworthy when data, code, and assumptions are routinely exposed to scrutiny. For researchers, it raises the cost of sloppy analysis; for listeners, it shows why replication and transparency matter for policy, medicine, and public trust.