#143 - John Ioannidis, M.D., D.Sc.: Why most biomedical research is flawed, and how to improve it

Episode Summary

Executive Summary: Peter Attia interviews John Ioannidis about the credibility of medical research, the mathematics behind false positives, and why many published findings—especially in nutrition epidemiology—are unreliable. They discuss p-values, power, bias, genetics as a model for better science, the PREDIMED trial, and Ioannidis’s early COVID seroprevalence work and the backlash it triggered.

Main Topics: Ioannidis’s scientific identity and training (Priority: 5/5): Ioannidis describes himself as a scientist shaped by mathematics, medicine, epidemiology, and a desire to combine rigorous quantitative methods with clinical impact. He recounts his upbringing in Greece, physician-scientist parents, and formative mentors in epidemiology, infectious disease, and evidence-based medicine. Why most published biomedical research is unreliable (Priority: 5/5): The discussion centers on Ioannidis’s 2005 model arguing that under low prior probability, low power, and high bias, nominally significant findings are often false positives. He explains how multiple scientists, selective reporting, and publication bias distort the literature. Statistics, p-values, and power (Priority: 5/5): Attia and Ioannidis unpack p-values, statistical significance, clinical significance, and the dangers of underpowered studies. Ioannidis argues that low power increases both false negatives and false positives, while overpowered big-data studies can produce statistically significant but clinically meaningless results. Genetics as a model for better research practice (Priority: 5/5): Genetics and GWAS are presented as a field that improved credibility by using better measurement, larger sample sizes, genome-wide testing, shared protocols, and consortia. Ioannidis contrasts this with nutrition, where measurement error, strong priors, and fragmented studies persist. Critique of nutritional epidemiology (Priority: 5/5): The conversation critiques nutrition research for weak measurement tools, strong pre-existing beliefs, selective reporting, and overinterpretation of small associations. Ioannidis argues that many claims should be tested with randomized trials, controlled metabolic studies, Mendelian randomization, or exposure-wide analyses. Bradford Hill criteria and causal inference (Priority: 4/5): They revisit Austin Bradford Hill’s criteria using tobacco as the classic example of a strong, consistent causal association. Ioannidis argues that most nutritional associations lack comparable strength and consistency, making causal claims much less secure. COVID-19 seroprevalence studies and scientific backlash (Priority: 5/5): Ioannidis discusses early seroprevalence studies in Santa Clara and Los Angeles that suggested SARS-CoV-2 was far more widespread than confirmed cases implied. He says the work was later validated but drew intense political and personal attacks, highlighting the need to protect scientists from partisan pressure.

Key Arguments: Scientific progress requires constant self-correction; a scientist must be willing to revise prior beliefs when evidence changes. A statistically significant p-value does not equal a true or clinically meaningful effect; significance must be interpreted in context. In low-power environments, the literature is enriched for false positives and exaggerated effect sizes, not just false negatives. Bias can create apparently strong signals even when the underlying effect is null; p-values alone do not account for this. Genetics improved credibility by using accurate measurement, large-scale collaboration, and pre-specified genome-wide testing. Nutrition research suffers from poor exposure measurement, strong ideological priors, and too many small, selectively reported studies. Randomized trials remain the best way to assess many nutritional questions, especially when adherence reflects real-world effectiveness. Large meta-analyses are not inherently better if they are built from cherry-picked studies that already match a desired conclusion. Bradford Hill’s criteria are most persuasive when associations are strong, consistent, and replicated, as with tobacco. Science should remain distinct from politics; researchers need protection from ideological and social-media attacks. Public and philanthropic funding should support high-risk, high-uncertainty research rather than demand guaranteed positive outcomes.

Data Points: Citation count of 2005 paper: close to 10,000 citations - Ioannidis notes his 2005 paper on why most published research is false is highly cited, though not his most cited work. Alternative p-value threshold proposed: 0.005 - He references proposals to make statistical significance more stringent in fields that have not adapted their thresholds. Traditional p-value threshold: 0.05 - Discussed as the conventional cutoff that often leads to overclaiming significance. Estimated number of people who have co-authored at least one scientific paper: about 35 million - Used to illustrate the scale and heterogeneity of the scientific workforce. Estimated number of principal investigators globally: less than 1 million - Ioannidis estimates the core group actually leading research is far smaller than total authorship counts. Number of large randomized trials in nutrition: over 200 - He cites this as evidence that randomized evidence in nutrition exists, though many trials are small or inconclusive. Odds ratios for tobacco-related harms: 10, 20, 30; average about 14x - Used as the classic example of a strong, consistent causal association in Bradford Hill reasoning. Genetic effect sizes: odds ratios around 1.01 - Illustrates how very small effects can still be real in genetics, though often not clinically meaningful. Estimated annual scientific output: easily 5 million papers added every year - He uses this to emphasize the scale of the literature and the difficulty of quality control. Seroprevalence under-ascertainment in early COVID: about 50 times more common than confirmed cases - Ioannidis describes early Santa Clara findings as showing far more infections than PCR-confirmed cases suggested. Infection fatality rate in nursing homes: 25% - He highlights the steep risk gradient for vulnerable populations during COVID-19. Estimated global infections by early December: close to 1 billion people - Ioannidis gives a rough estimate of cumulative global SARS-CoV-2 infections. Hazelnut claim from nutrition paper: 12 hazelnuts/day = 12 years longer life - Used as a tongue-in-cheek example of absurdly inflated nutritional epidemiology claims. Coffee claim from nutrition paper: 3 cups/day = 5 years longer life - Another example of exaggerated translation of relative risks into life expectancy. Egg claim from nutrition paper: 1 egg = 6 years shorter life - Illustrates how implausible effect sizes can emerge from weak observational data. Bacon claim from nutrition paper: 2 strips = 10 years shorter life - Used to show how nutritional epidemiology can produce sensationalized estimates.

Pivotal Quotes: "I think that it's very difficult to know yourself. And I've been struggling on that front for a long time. So I'm trying to be a scientist." — John Ioannidis: On how he defines himself and the ongoing process of scientific self-correction. "If you get a nominally statistically significant signal with a traditional p-value of slightly less than 0.05, then the chances that you have a red herring... are higher than 50%." — John Ioannidis: Explaining the core conclusion of his 2005 model on false positives in biomedical research. "I care about managing and helping real people." — John Ioannidis: On why real-world effectiveness matters more than idealized efficacy in nutrition and medicine.

Implications: Listeners should treat many published health claims cautiously, especially in nutrition. Better measurement, preregistration, larger collaborations, and humility are essential. Science advances by correction, not certainty, and researchers need protection from politicized backlash.