Science Fictions with Stuart Ritchie

Episode Summary

Executive Summary: Garrett Peterson interviews psychologist Stuart Ritchie about his book Science Fictions and the replication crisis in science. They trace how fraud, bias, negligence, and hype distort research, why social and institutional incentives reward flashy but unreliable findings, and how reforms like registered reports, open science, and better transparency can make science more trustworthy.

Main Topics: Origins of Ritchie’s concern with replication (Priority: 5/5): Ritchie explains that his interest began with a failed replication of Darrell Bem’s psychic precognition paper during his PhD, plus surrounding fraud stories and other failed replications around 2011-2012 that helped spark the replication crisis. Science as a social process (Priority: 5/5): The discussion emphasizes that science depends on persuading other scientists through peer review, journals, and organized skepticism; this is a strength, but also a weakness because the system is run by fallible humans with incentives to cut corners or hype results. Replication crisis across disciplines (Priority: 5/5): They discuss how failed replications began in social psychology but spread to other areas like cognitive psychology, economics, and medicine, raising questions about how often published findings are overstated or false. Fraud, bias, negligence, and hype (Priority: 5/5): Ritchie breaks down the book’s four core problems: deliberate data fabrication, unconscious and institutional bias toward positive findings, careless errors and underpowered studies, and exaggeration of claims in press releases and publications. Perverse incentives in academia (Priority: 5/5): The conversation links bad scientific behavior to publish-or-perish pressures, journal prestige, grant competition, career advancement, media attention, and metrics such as citation counts and impact factors. Reforms to improve scientific reliability (Priority: 4/5): They discuss solutions such as registered reports, preregistration, open data/code, transparent exploratory analysis, and broader changes to hiring, tenure, and publishing to reward truth-seeking rather than flashy outputs.

Key Arguments: A failed replication is often more informative than the original flashy result because it reveals whether an effect is robust rather than a statistical fluke. Science’s social nature is essential, but the same social dynamics create vulnerabilities to hype, fraud, and conformity. The replication crisis is not simply a problem of failed replications; it reflects weaknesses in original studies and the broader incentive structure producing unreliable results. Social psychology provided early high-profile examples of implausible findings that later failed to replicate, demonstrating that standard statistical methods alone do not guarantee truth. Fraud is rarer than bias or negligence, but it is serious, often underdetected, and can persist for years when universities and journals hesitate to act. Publication bias and p-hacking systematically inflate the number of positive findings by selecting or modifying analyses until p-values fall below 0.05. Many studies are underpowered, meaning they are too small to detect real effects reliably; when they do find significance, effects are often exaggerated and unlikely to replicate. Hype is often generated by scientists themselves through press releases and popular writing, not only by journalists overstating results. Registered reports and preregistration can reduce publication bias and make it harder to hide analytic flexibility, improving trust in the literature. The root issue is incentive design: if academia rewards volume, prestige, and headlines, researchers will be pushed toward behavior that undermines truth-seeking.

Data Points: Timing of failed replication attempt: 2011 - Ritchie describes his replication attempt of Darrell Bem’s psychic precognition paper during his PhD. Original Bem paper experiments: 9 experiments - The 2011 paper claimed evidence for precognition across multiple experiments. Journal policy on replications: blanket refusal to publish replication studies - The journal that published the original psychic paper would not consider Ritchie’s replication regardless of outcome. Peer review as standardization: widely standardized by the 1970s - Ritchie notes peer review existed earlier but became a rigid norm much later. Original social priming paper date: 1996 - He references an age-priming study from the mid-1990s as an infamous example of social priming. Human self-reported misconduct: about 2% - Ritchie cites a survey in which roughly 2% of scientists admitted to having committed misconduct. Perceived colleague misconduct: about 14% - In the same survey, about 14% said they believed colleagues had committed misconduct. Psychology papers with positive results: about 94% - He cites meta-research suggesting an extremely high share of psychology papers report positive findings. Conventional significance threshold: p < 0.05 - Ritchie discusses how researchers often try to push results below the accepted cutoff for publication. Large-scale statistical checking: over half - He says StatCheck found more than half of thousands of psychology papers contained some kind of inconsistent number. Effect-size inflation in replications: about half the original size - He notes that replication studies often find effects substantially smaller than the original papers reported. Example of cell contamination problem: since the 1950s - He describes imposter cell line contamination as a long-running issue in biology and cancer research.

Pivotal Quotes: "Science is not just, you know, a thing which can happen with one person." — Stuart Ritchie: Explaining science as a social process requiring persuasion, peer review, and scrutiny by other researchers. "We’re giving prestige to people who publish lots of good papers with flashy looking results." — Garrett Peterson: Summarizing the incentive structure that can reward attention-grabbing but unreliable research. "It’s a kind of make-work program for very smart people." — Stuart Ritchie: Describing the worst-case scenario where academic activity resembles science without reliably uncovering truth.

Implications: Listeners should be skeptical of flashy findings, especially when based on small studies or weak transparency. For science, the path forward is stronger preregistration, open methods, and incentives that reward accuracy over novelty.