Episode Summary
Executive Summary: Michael Shermer interviews Stuart Ritchie about his book Science Fictions, focusing on how fraud, bias, negligence, and hype distort scientific knowledge. They examine replication failures, weak causal inference, publication incentives, and high-profile cases in psychology, medicine, and social science, then outline open-science reforms such as preregistration, data sharing, and reward structures that favor reliability over novelty.
Main Topics: Replication crisis and the Darrell Bem psi studies: The conversation opens with Bem’s controversial precognition experiments and Shermer/Ritchie’s failed replication attempt, using it to show how weak design, flexible statistics, and journals’ preference for novel findings can elevate unreliable claims. Fraud in science and why it persists: They discuss notorious cases like Diederik Stapel, Jan Hendrik Schön, and Andrew Wakefield, emphasizing how career pressure, weak deterrence, and systems that reward results over rigor can enable intentional misconduct. Bias, theory lock-in, and bad incentives: They explore how scientists can become attached to dominant theories, from amyloid hypotheses in Alzheimer’s to priming and social psychology, and how publication norms can turn exploratory work into falsely confirmatory narratives. Hype and oversimplification in popular science: Books and media coverage by figures like Matthew Walker, Amy Cuddy, and educational researchers are used to illustrate how compelling narratives, selective reporting, and press-friendly claims can outpace evidence. Causality problems in nutrition, aging, and epidemiology: The discussion highlights how correlational studies, self-report bias, and multiple comparisons make diet, sleep, and cognitive-aging research hard to interpret causally, often producing contradictory findings. Open science reforms and institutional solutions: Ritchie proposes preregistration, registered reports, replication-friendly journals, transparency, data/code sharing, and changing university and funder incentives to value reliable, reproducible work. Triangulation and the limits of one-study explanations: Using smoking and lung cancer, they argue that robust science often comes from converging evidence across methods rather than a single decisive experiment, especially for complex human behavior and disease.
Key Arguments: Novel, statistically significant findings are not enough; robust claims require replication, converging evidence, and plausible mechanisms. Fraud is rare relative to sloppy science, but career pressure and weak oversight make misconduct and selective reporting more likely than they should be. Many famous psychology findings were not outright fraud but were built on flexible analyses, small samples, and storytelling that overstated what the data showed. Causal claims from correlational data are inherently fragile, especially in nutrition, aging, education, and social psychology. Hype distorts public understanding because researchers, books, and media favor simple narratives over messy uncertainty. The scientific system should reward openness, preregistration, replication, and accurate reporting rather than only novelty and prestige. Even if an effect replicates, it may still be theoretically misinterpreted; experiments must test the right mechanism, not just produce a significant result. Triangulation across different methods and datasets is often the best route to strong causal inference in complex domains.
Data Points: Bem study experiments: 9 experiments - Stuart Ritchie describes Daryl Bem’s precognition paper as containing nine separate experiments. Bem pornography effect: 53.1% - Participants reportedly chose the curtain with pornography slightly above chance in one Bem experiment. Amy Cuddy TED Talk views: 60+ million - Shermer notes the power posing talk has been watched by tens of millions of people. Power posing study sample size: 42 people - Ritchie says the original power posing evidence rested on one small experiment with 42 participants. Stapel retractions: 58 papers - Diederik Stapel’s fabricated social psychology findings led to dozens of retractions. Mendelian randomization: New epidemiological method - Ritchie cites it as an emerging way to infer causality from genetic variation in observational data. Impact factor example: Nature around 44 - Ritchie uses Nature’s high impact factor to illustrate prestige-driven citation incentives. H-index example: Shermer 28 - Shermer mentions his own Google Scholar h-index during the discussion. Predi-Med trial: Large randomized trial with errors - They discuss a Mediterranean diet trial in Spain that had randomization problems and required retraction/replacement.
Pivotal Quotes: "Science is super important and we need it. It's really precious and we need to get it right." — Stuart Ritchie: Ritchie explains why he wrote the book and why reform matters. "We need to do this ourselves." — Michael Shermer: Shermer argues that scientists, not critics like creationists, should uncover and correct errors in science. "You can replicate all you want, but it won't actually get us any further in our understanding of the world." — Stuart Ritchie: Ritchie warns that replication alone does not solve bad theory or poor experimental design.
Implications: Listeners should treat sensational findings cautiously and look for preregistration, replication, transparent data, and triangulated evidence. For science, the path forward is structural reform: reward rigor, not hype; openness, not secrecy; and reliability, not novelty.