WS MoreOrLess: How Reliable is Psychology Science?

Episode Summary

Executive Summary: The episode examines the psychology replication crisis, explaining why many published findings fail to reproduce and what that means for scientific credibility. It then shifts to a listener correction about the word “decimate,” concluding that modern usage as “devastate” is historically established and defensible despite pedants.

Main Topics: Reproducibility Project in psychology (Priority: 5/5): The program opens with the large-scale replication effort led by Brian Nosek, which re-tested nearly 100 studies from top psychology journals and found many could not be confirmed. Why results fail to replicate (Priority: 5/5): Tim Harford and guests discuss how flexible data analysis, many possible subgroup tests, and publication bias can produce apparently significant but unreliable findings. How psychologists should change publishing (Priority: 4/5): Andrew Gelman argues for weaker, more realistic claims and for publishing more data, including null and disappointing results, to improve scientific transparency. Is psychology uniquely problematic? (Priority: 4/5): Daryl O'Connor says the findings are disappointing but also an opportunity to improve methods and scientific culture, while Gelman notes similar issues affect medicine too. The meaning of ‘decimate’ (Priority: 3/5): A listener complaint prompts a discussion of whether Harford’s use of ‘decimated’ was wrong; the segment concludes that modern usage is valid and longstanding. Language, etymology, and usage (Priority: 3/5): Oliver Kam explains that the historical Roman meaning of decimate is now a curiosity, and that current English usage has used the broader meaning for centuries.

Key Arguments: Only 39 of the replication attempts in the project confirmed the original findings, suggesting a serious reproducibility problem in psychology. A result can appear statistically significant because researchers have many analytic choices and subgroup comparisons, increasing the chance of chance findings. Publication bias means null or boring results are less likely to appear in journals, while surprising results are favored even when false. Psychologists should claim less dramatic findings and make all data publicly available, including unsuccessful experiments. Non-replication does not automatically prove the original study was false; some failures may reflect methodology, power, or context differences. The replication project is both a warning and an opportunity to improve reporting standards and scientific culture. The word “decimate” is correctly used today to mean devastate or destroy a large part of something; the Roman “one in ten” sense is historical rather than prescriptive.

Data Points: Number of original studies examined: almost 100 - Studies from three leading psychology journals assessed by the Reproducibility Project Researchers involved: 270 - Collaborators coordinated in the replication effort led by Brian Nosek Successful replications: 39 - Replication attempts that confirmed the original findings, in the team’s opinion Statistically significant replications: 36% - Share of replication attempts that found statistically significant results Relative effect size: half as big - Effect sizes in replications were about 50% of those in the original studies Historical decimation rate: 1 in 10 - Roman punishment referenced in the discussion of the word “decimate” Time span of modern usage: over 300 years - Oliver Kam says “decimate” has been used in the broad modern sense for centuries

Pivotal Quotes: "I think they should publish less and they should publish more at the same time." — Andrew Gelman: On how psychology should change its publication practices "The results were not a surprise to me." — Andrew Gelman: His reaction to the low reproducibility found by the project "The meaning of decimate in the sense of destroy a large part of is overwhelmingly the dominant sense in which the word is used." — Oliver Kam: On defending Tim Harford’s use of the word ‘decimated’

Implications: The episode suggests psychology must improve power, transparency, and publication norms to reduce false positives. For listeners, it also reinforces that language rules evolve: common modern usage can be valid even when etymology suggests otherwise.