#269 - Good vs. bad science: how to read and understand scientific studies

Episode Summary

Executive Summary: This episode is a practical primer on how to read scientific studies critically. Peter Atiyah and Bob break down the research pipeline, study types, common biases, trial phases, statistical significance, power, effect sizes, publication bias, and peer review. The core message: don’t trust headlines—evaluate design, randomization, blinding, confounding, absolute risk, and whether a study was adequately powered.

Main Topics: How studies are conceived and designed (Priority: 5/5): The hosts explain the scientific process from hypothesis formation through protocol design, ethics review, pre-registration, and funding, emphasizing that rigorous studies begin with a clear null and alternative hypothesis. Types of studies and evidence hierarchy (Priority: 5/5): They distinguish observational studies, experimental studies, and reviews/meta-analyses, while cautioning that a meta-analysis is only as good as the studies it includes. Clinical trial phases and regulatory pathway (Priority: 5/5): The discussion outlines phase 1 through phase 4 drug trials, including safety, efficacy, approval, and post-marketing surveillance, using oncology and semaglutide examples. Bias, confounding, and limitations of observational research (Priority: 5/5): They detail healthy user bias, recall bias, performance bias, selection bias, and confounding, especially in nutrition and lifestyle epidemiology. Statistics: p-values, confidence intervals, power, and effect size (Priority: 5/5): The episode explains statistical significance versus clinical significance, false positives/negatives, power analysis, confidence intervals, hazard ratios, absolute risk reduction, and number needed to treat. Why studies stop early and how publication works (Priority: 4/5): They cover stopping trials for safety, benefit, or futility, then explain peer review, journal selection, impact factor, and publication bias. How to read a paper efficiently (Priority: 4/5): Peter describes his workflow: abstract first, then methods, figures/tables, results, and discussion, with emphasis on understanding study design before conclusions.

Key Arguments: Good science should be hypothesis-driven, with a clearly stated null hypothesis and a design that can falsify it. Randomization is the main tool for reducing bias in experimental studies; without it, causal inference is much weaker. Observational studies can generate hypotheses, but healthy user bias and confounding often make causal claims unreliable. Meta-analyses do not automatically outrank randomized trials; poor studies aggregated together still produce poor evidence. Phase 1 trials prioritize safety, phase 2 adds early efficacy, phase 3 is the main approval-grade randomized test, and phase 4 monitors real-world safety and new indications. Blinding matters because both participants and investigators can unconsciously alter outcomes when they know treatment assignment. Primary outcomes should be pre-specified and weighted more heavily than secondary outcomes; multiple comparisons inflate false positives. Statistical significance is not the same as clinical significance; a tiny effect can be statistically real but medically trivial. Power analysis is essential because underpowered studies miss real effects and overpowered studies can detect meaningless ones. Absolute risk reduction and number needed to treat are more clinically useful than relative risk alone. Publication bias distorts the literature because negative studies are less likely to be published. A disciplined reading strategy—abstract, methods, figures, results, discussion—helps readers judge rigor before accepting conclusions.

Data Points: P-value threshold: 0.05 - Used as the conventional cutoff for statistical significance and false-positive tolerance. False negative rate (beta): 10-20% - Typical acceptable range when designing studies, corresponding to 80-90% power. Power: 80-90% - Common target for clinical trials to reduce the chance of missing a true effect. PREDIMED sample size: ~7,500 participants - Large randomized diet trial discussed as an example of randomization issues and early stopping. PREDIMED clinic issue: 11 clinics - Some clinics were randomized as clusters rather than individuals, complicating the original analysis. CTEP inhibitor trial deaths: 82 vs 51 - Torcetrapib trial stopped early for safety after more deaths in the treatment arm than control. Look AHEAD trial size: ~5,000 participants - Lifestyle intervention trial in overweight/obese patients with type 2 diabetes, stopped for futility. Look AHEAD hazard ratio: 0.95 - Suggested only a 5% reduction in cardiovascular events, not enough to justify continuing. Look AHEAD confidence interval: 0.83 to 1.09 - Crossed 1, indicating no statistically significant benefit. Women’s Health Initiative breast cancer risk: 25% relative increase - Example showing how relative risk can sound alarming while absolute risk change is small. Women’s Health Initiative absolute risk: 5 per 1,000 vs 4 per 1,000 - Illustrates the small absolute difference behind the relative increase. Hazard ratio example: 0.667 - Derived from 20% vs 30% event rates to show how to interpret relative hazard. NNT example: 1,000 - A 5 per 1,000 to 4 per 1,000 reduction in events requires treating 1,000 people to prevent one event. Impact factor distribution: 98% of journals < 10; 95% < 5; ~50% < 2 - Used to explain how journal prestige is often reflected in citation-based impact factor. New England Journal of Medicine citations: ~347,000 - Example of a top-tier journal with very high citation volume and impact factor. Lancet citations: ~250,000 - Another high-impact clinical journal cited as an example. Cancer Journal for Clinicians impact factor: 292 - Highlighted as an outlier due to its highly cited statistics/reporting format.

Pivotal Quotes: "A thousand sows ears makes not a pearl necklace." — Peter Atiyah: Used to argue that a meta-analysis of poor studies does not become high-quality evidence simply by aggregation. "The default position is that the null hypothesis is correct." — Peter Atiyah: Explaining why studies must be designed to rigorously falsify a claim rather than confirm it casually. "It’s the single most important table you should ever familiarize yourself with if you want to be in the business of designing clinical trials." — Peter Atiyah: Referring to the power table and emphasizing the importance of sample size planning.

Implications: Listeners should become far more skeptical of headlines and more attentive to design, bias, absolute risk, and power. For researchers and journalists, the episode argues for better pre-registration, clearer reporting, and less overreliance on relative risk or weak observational claims.