What Is The Replication Crisis?

Episode Summary

Executive Summary: The episode explains the replication crisis in science and finance: many published findings fail to reproduce because of publication bias, p-hacking, low statistical power, and incentive structures that reward sensational results over reliability. It uses medical and finance examples to show how false positives can enter the literature and affect real-world decisions, while noting reforms like preregistration and replication studies are improving standards.

Main Topics: The replication crisis in science (Priority: 5/5): The episode opens with medical research failures to reproduce, using Ioannidis and Begley to show that many influential studies are not reliable when retested. Incentives and publication bias (Priority: 5/5): Academic careers depend on publishing novel, positive results in prestigious journals, discouraging replication and encouraging selective reporting of favorable outcomes. How p-values and statistical testing work (Priority: 4/5): The host explains null hypothesis testing, p-values, and the conventional 0.05 threshold as the basis for judging whether a result is likely due to chance. False positives, power, and p-hacking (Priority: 5/5): Using a simple hypothetical example, the episode shows how even well-functioning statistical systems can generate many false positives, especially when studies are underpowered or data are manipulated. Replication crisis in finance (Priority: 4/5): Campbell Harvey’s critique is used to argue that many market anomaly papers in top finance journals may be false or overstated, with implications for investors and funds. Reforms and improving research practices (Priority: 4/5): The episode notes positive changes such as preregistration, replication efforts, retraction tracking, and databases of negative results that can reduce bias and improve reliability.

Key Arguments: Many published findings are false or fail to replicate, even in prestigious journals and reputable labs. The issue is driven not just by bad actors but by structural incentives: grants, tenure, journal prestige, and media attention reward surprising positive results. A p-value below 0.05 does not prove a hypothesis; it only indicates evidence against the null under specific assumptions. Even with an 80% powered test and a 5% significance threshold, a large share of published results can still be false positives when many hypotheses are tested. Underpowered studies and p-hacking substantially increase the likelihood of false discoveries. Replication is undervalued because it is less glamorous and often less publishable than novel discovery. Finance faces a similar credibility problem: many published market-beating strategies may not be real, which can mislead investors and asset managers. The scientific system is imperfect but still more reliable than intuition or unsupported claims, and reforms are making it better.

Data Points: Failed replications in Begley sample: 47 of 53 - C. Glenn Begley’s attempt to reproduce landmark cancer studies at Amgen Researchers unable to reproduce others' experiments: More than 70% - Nature survey cited in the episode Researchers unable to reproduce their own experiments: More than 50% - Nature survey cited in the episode False positive rate in hypothetical published literature: 45 out of 145 papers (~31%) - Example using 1,000 hypotheses, 10% true, 80% power, and p < 0.05 Statistical power assumption: 80% - Used in the illustrative example of hypothesis testing Significance threshold: 0.05 - Conventional cutoff for publication-worthy results Beatles youth experiment effect: Nearly 18 months younger - Psychology paper used as an example of how absurd results can arise from misused methods Harvey's estimate of false finance strategies: At least half of 400 strategies - Campbell Harvey’s argument about top finance journal findings Ioannidis false positive estimate: Around 14% - A later replication attempt of Ioannidis’ claim in biomedical studies Year of Ioannidis essay: 2005 - Publication of 'Why Most Published Research Findings Are False'

Pivotal Quotes: "they'd done the experiment six times, got the published result once, and put it in the paper because it made the best story" — Patrick Boyle (quoting the scientist's response): Explains selective publication and why a result may appear in a paper despite being non-reproducible "It's a huge issue" — Campbell Harvey: Harvey’s description of the replication crisis in finance "we can never prove that something is definitely true, we can only prove that something is false" — Patrick Boyle: Summary of Popperian falsification and the limits of statistical hypothesis testing

Implications: Listeners should treat sensational studies and market claims with skepticism, especially when replication is absent. The episode suggests better transparency, preregistration, and replication are essential to protect both scientific credibility and investor outcomes.