Brian Nosek on the Reproducibility Project

Episode Summary

Executive Summary: Russ Roberts and Brian Nosek discuss the Reproducibility Project in psychology: what reproducibility means, why published findings can diverge from truth, and what a large-scale replication effort found. The project showed substantial decline from original to replication effects, exposing publication bias, low power, and flexibility in analysis, while also highlighting field differences and the need for open, transparent science.

Main Topics: What reproducibility means in science (Priority: 5/5): Nosek distinguishes between rerunning analyses on the same data and true replication using new data under comparable conditions. Why published results may not be reliable (Priority: 5/5): The conversation emphasizes incentives for novelty, prestige, and statistically significant findings, which can distort the published literature away from truth. The Reproducibility Project: design and scope (Priority: 5/5): The project sampled 2008 studies from three top psychology journals, used original materials when possible, and coordinated large-scale replications with transparent procedures. What the replication results showed (Priority: 5/5): Replication effects were smaller than originals, with far fewer significant results; multiple criteria were used to judge success because replication is not a single binary concept. Interpreting decline effects and false positives (Priority: 4/5): They discuss publication bias, underpowered studies, exploratory data mining, and how these can produce inflated original effects that later shrink on replication. Field differences and future open science efforts (Priority: 4/5): Cognitive psychology appeared to replicate better than social psychology, and Nosek describes ongoing replication projects in cancer biology and other fields.

Key Arguments: Reproducibility is a core norm of science because credibility should come from independent confirmation, not authority or prestige. The published literature is likely biased upward because researchers are rewarded for positive, novel, statistically significant results, not for accuracy. Exploratory analysis is valuable, but conclusions from exploratory work must be tested on new data rather than treated as confirmatory. The project intentionally used transparency, original-author input, and open materials to reduce hostility and improve the fairness of replications. A replication outcome is not always a simple yes/no; effect size, statistical significance, and subjective judgment can each tell a different story. Smaller replication effects may reflect publication bias, but could also reflect context sensitivity, especially in social psychology. The project should be viewed as a large empirical probe into reproducibility, not a final verdict on an entire field. Open science infrastructure and larger cross-disciplinary replication efforts are needed to better estimate where reproducibility problems are strongest.

Data Points: Year sampled: 2008 - The project selected studies from three psychology journals published in 2008. Journals sampled: 3 - Psychological Science, JPSP, and JEP: Learning, Memory, and Cognition. Completed replications: 100 - The final Science report included 100 completed replication studies. Original studies with significant results: 97% - Nearly all original studies in the sample reported statistically significant findings. Replication studies with significant results: 36% - Only about a third of replications achieved statistical significance. Subjective replication success: 39% - Replication teams judged 39% of effects as reproducing the original result. Effect size of replications: About half of originals - Average replication effects were described as half the magnitude of original effects. Original effect sizes inside replication CI: 47% - Less than half of original effects fell within the replication effect's 95% confidence interval. Combined result under no-bias assumption: 68% significant - When original and replication results were combined, 68% were significant if no bias in original results is assumed. Scope of eligible studies: About 165 eligible out of 488 possible - Nosek described a constrained sampling frame from the 2008 journal issues. Started replications: 113 - More studies were started than finished. Cognitive vs. social psychology: About 2x higher replication rate in cognitive psychology - Cognitive findings replicated at roughly twice the rate of social psychology findings.

Pivotal Quotes: "published and true are not synonyms" — Russ Roberts: Roberts emphasizes that journal publication does not guarantee truth. "Reproducibility is central to science" — Brian Nosek: Nosek explains why independent confirmation is foundational to scientific credibility. "the only way you can get statistical significance is to take advantage of chance" — Brian Nosek: He describes how underpowered studies and publication bias can inflate original findings.

Implications: The episode argues for preregistration, transparency, and more replications across disciplines. For researchers and readers, it is a warning that published findings can overstate reality and that stronger evidence practices are needed.