Ed Yong on Science, Replication, and Journalism

Episode Summary

Executive Summary: Russ Roberts and Ed Yong discuss the fragility of scientific claims, especially in psychology, epidemiology, and cancer research, and how replication failures, publication bias, and researcher discretion can distort the literature. They also explore how science journalism can either amplify or correct these problems, emphasizing skepticism, context, and the internet’s role in exposing error while also speeding misinformation.

Main Topics: Replication crisis in psychology (Priority: 5/5): Yong explains the controversy around John Bargh’s famous age-priming study and a failed Belgian replication, using it to show how difficult it can be to verify influential findings. Conceptual replication vs. direct replication (Priority: 5/5): The conversation distinguishes between testing the same broad idea in different ways and reproducing the exact original experiment, with concern that conceptual replication can conceal weak evidence. Publication bias and researcher degrees of freedom (Priority: 5/5): They discuss how flexible decisions about sample size, outliers, significance checks, and wording can manufacture significant results and crowd out negative findings. Fraud, confirmation bias, and scientific culture (Priority: 4/5): The speakers note that the bigger danger is often not obvious fraud but subtle self-deception and field norms that encourage researchers to explain away failed replications. Science journalism and verification (Priority: 5/5): Yong reflects on the challenges of covering complicated research honestly, the need to consult critics and specialists, and the temptation to treat studies as authoritative facts too quickly. Epidemiology, alcohol, and cancer coverage (Priority: 4/5): Roberts and Yong debate how observational studies are reported in the media, using alcohol-and-cancer findings to illustrate both legitimate scientific accumulation and the risk of oversimplified headlines. The internet as amplifier and corrector (Priority: 3/5): The internet helps surface criticism, methods debates, and corrections, but it does not reliably ensure that accurate critiques reach as wide an audience as the original claims.

Key Arguments: Direct replication is essential because a study can be widely cited and intuitively plausible yet still fail when exactly repeated. Conceptual replications can be useful, but only if individual studies are internally robust and not products of flexible analysis or publication bias. Researcher degrees of freedom allow scientists to unknowingly or deliberately tune analyses until they obtain statistically significant results. Publication bias creates a literature that overrepresents positive findings; Sterling’s work suggested this problem has long been severe in psychology. Scientists and journalists are both human and subject to incentives, so claims of pure objectivity often mask real biases. Science journalism should provide context, cite critics, and examine methodology rather than merely relaying the newest result. Epidemiological evidence can be strong even without randomized trials when many studies point in the same direction and mechanism is plausible. The internet improves access to criticism and methodological scrutiny, but correction still lags behind sensational claims.

Data Points: Original study year: 1996 - John Bargh’s age-priming study on walking speed was first published in 1996. Replication paper year: early 2012 - Yong discusses a replication attempt that appeared at the start of the year and failed to find the original effect. Significant results in psychology journals: 97% - Sterling’s 1959 analysis found that 97% of studies in four major psychology journals reported statistically significant positive results. Survey admission rate: 40–50% - A survey described as finding many psychologists admitted to common questionable practices such as data tweaks and flexible stopping rules. Sample size of one cancer study: well over 100,000 - Roberts cites a UK women’s health study on alcohol and cancer with a very large sample size. Cancer study count: 53 landmark studies - Roberts references a review claiming 53 landmark cancer studies were examined. Failed replications in cancer review: 47 of 53 - Roberts says a cited review allegedly found 47 of 53 cancer studies could not be replicated. Repeated experiment attempts: 50 times - Begley reportedly said his team redid one experiment 50 times and never got the original result. Positive result obtained in original paper: 1 out of 6 times - A researcher reportedly said a result was obtained once in six tries and included because it made the best story. NFL player life expectancy claim: 58 years - Roberts cites a widely repeated but disputed statistic about NFL players’ average life expectancy.

Pivotal Quotes: "The first principle is that you must not fool yourself, and you are the easiest person to fool." — Russ Roberts: Roberts invokes Feynman to frame the danger of self-deception in scientific research. "And I think that the act of actually verifying whether something is true, you know, calling around, getting different people to comment on something." — Ed Yong: Yong explains how science journalism should move beyond translation to active verification. "we went through the paper line by line, figure by figure. I explained that we redid their experiment 50 times and never got their result." — Russ Roberts quoting Begley: Used to illustrate how many published findings may not survive serious replication attempts.

Implications: Listeners should treat dramatic findings cautiously, especially when they rest on a single study. Better incentives for replication, transparency, and contextual journalism are needed to keep science trustworthy and useful.