#188 - AMA #30: How to Read and Understand Scientific Studies

Episode Summary

Executive Summary: This AMA episode explains how to evaluate scientific studies, emphasizing the hierarchy and limitations of evidence. Peter Atiyah and Bob Kaplan walk through hypothesis formation, study design, observational vs experimental research, randomized controlled trials, meta-analyses, and clinical trial phases, stressing that rigor, randomization, and critical appraisal matter more than headline claims.

Main Topics: How studies begin: hypothesis and design (Priority: 5/5): The discussion starts with the scientific process: formulating a null and alternative hypothesis, then designing an experiment with clear outcomes, sample size planning, ethics review, statistics, and preregistration before data collection. Types of studies and evidence hierarchy (Priority: 5/5): They distinguish observational studies, experimental studies, and reviews/meta-analyses, explaining case reports, case series, cohort studies, non-randomized trials, and randomized controlled trials, while cautioning against simplistic pyramids of evidence. Observational studies: strengths and pitfalls (Priority: 4/5): Case reports and cohort studies can generate hypotheses and reveal patterns, but they cannot establish causality and are vulnerable to bias, confounding, and selection effects. Randomization and control in experiments (Priority: 5/5): Randomized controlled trials are presented as the gold standard because random assignment reduces bias; non-randomized trials are shown to be more vulnerable to self-selection and other confounders. Meta-analyses and systematic reviews (Priority: 4/5): Meta-analyses can strengthen inference by combining studies, but only if the underlying studies are high quality; otherwise they simply aggregate poor evidence and may mislead. Clinical trial phases for drugs (Priority: 5/5): The episode outlines phase 1 through phase 4 trials: safety and dose escalation, early efficacy, large randomized efficacy trials, and post-marketing studies for new indications and rare side effects. How to read studies critically (Priority: 4/5): The speakers emphasize asking whether a study is rigorous, what biases may exist, whether outcomes were pre-specified, and whether the findings are truly statistically and clinically meaningful.

Key Arguments: Good science should begin with a clearly stated hypothesis, including a null hypothesis and an alternative hypothesis. Study design matters as much as the question; without careful design, results can be misleading even if they look impressive. Observational studies can identify patterns and generate hypotheses, but they generally cannot prove causation. Randomization is crucial because it reduces selection bias and makes treatment groups more comparable. Non-randomized studies are especially vulnerable because the reasons people end up in one group or another can distort results. Meta-analyses are only as good as the studies they include; combining weak studies does not create strong evidence. Clinical trial phases serve different purposes: phase 1 focuses on safety and dose, phase 2 on early efficacy, phase 3 on definitive randomized testing, and phase 4 on post-approval surveillance and new indications. A statistically significant result is not automatically a meaningful or trustworthy result; context, design, and bias still matter. Listeners should evaluate studies by asking about sample size, randomization, blinding, endpoints, preregistration, and publication bias.

Data Points: AMA episode number: 30 - The conversation is introduced as AMA number 30. Phase 1 study size: Usually less than 100 people - Phase 1 drug trials are described as small dose-escalation studies. Example cohort size per dose level: 12 people - An illustrative phase 1 escalation scheme uses cohorts of 12 participants at each dose. Phase 2 study size: 20-50 people, sometimes a few hundred - Phase 2 trials are described as small to moderate studies focused on safety and early efficacy. Phase 3 study size: Potentially thousands of patients - Phase 3 trials are described as large, rigorous randomized studies. Phase 4 timing: After drug approval - Phase 4 studies occur post-marketing to gather additional safety and indication data. Case report example: 1 patient - The melanoma hypercalcemia example is presented as an individual case report. Case series example: 27 patients - A case series is described as looking back at 27 patients with an unusual finding over 40 years. Timeframe example: 10 years - A retrospective cohort example compares sauna users over the last 10 years with non-users. Timeframe example: 5 years - A prospective cohort example follows people forward over the next 5 years.

Pivotal Quotes: "a thousand sows ears makes not a pearl necklace" — Bob Kaplan (quoting James Yang): Used to explain that a meta-analysis of poor-quality studies does not become high-quality evidence. "the signal and the noise" — Peter Atiyah: Describes the challenge of distinguishing reliable findings from misleading or contradictory study headlines. "Good science is generally hypothesis driven." — Bob Kaplan: Introduced while explaining the first step in designing a study.

Implications: Listeners should treat study headlines cautiously and judge evidence by design quality, not just publication volume or statistical significance. The episode equips audiences to spot bias, understand trial phases, and weigh meta-analyses appropriately.