Simpson’s Paradox: How to make vaccinated death figures misleading

Episode Summary

Executive Summary: The episode разобирает viral claim that vaccinated people aged 10-59 were dying at higher rates than unvaccinated people in England. Using ONS data, Tim Harford and Stuart MacDonald show this is a Simpson's paradox caused by age confounding: vaccinated people in that band are much older on average than unvaccinated people, and death risk rises sharply with age. The segment stresses careful interpretation of aggregate statistics.

Main Topics: Viral misreading of ONS vaccination mortality data (Priority: 5/5): A social media chart claimed more vaccinated than unvaccinated people died, using ONS figures for ages 10-59, but the presenters argue the chart was misleading because it ignored age structure. Simpson's paradox and confounding by age (Priority: 5/5): The episode explains how an apparent overall pattern reverses when data are split into smaller age groups, with age acting as a confounder that distorts the aggregate comparison. How vaccination rates differ by age (Priority: 5/5): Vaccination uptake is much higher in older people within the 10-59 band, while the unvaccinated subgroup skews much younger, creating a biased comparison of death rates. Difference between all-cause mortality and COVID-specific mortality (Priority: 4/5): The controversial chart referred to deaths from any cause, whereas the ONS report's headline finding concerned deaths involving COVID-19, which showed a much higher age-adjusted risk among the unvaccinated. Context, methodology, and age-adjustment in statistics (Priority: 4/5): The discussion emphasizes that data tables with broad age bands can mislead and that age-adjusted or more granular breakdowns are needed to make valid comparisons. Media literacy and confirmation bias (Priority: 3/5): The episode closes by warning that people tend to accept statistics that support preexisting beliefs without checking context, making them vulnerable to misleading charts.

Key Arguments: The viral chart was misleading because it compared vaccinated and unvaccinated people across a broad 10-59 age band without controlling for age. Death risk rises sharply with age, so a vaccinated subgroup that is older on average will naturally show higher all-cause mortality even if vaccines are protective. Within narrower age groups, the expected pattern appears: unvaccinated people generally have higher death rates than vaccinated people. The ONS report's main conclusion was the opposite of the viral claim: age-adjusted risk of death involving COVID was far higher among the unvaccinated. The issue is a textbook case of Simpson's paradox, where aggregate data reverses when broken into subgroups. Broad age grouping created a compositional bias because vaccinated 10-59 year olds skewed older, while unvaccinated 10-59 year olds skewed younger. Statistics need context; without it, people can mistakenly infer causation from a correlation driven by demographics.

Data Points: Age-adjusted risk of death involving COVID-19: 32 times greater in unvaccinated people - Main conclusion of the ONS release covering deaths from January to September Age band analyzed in viral chart: 10 to 59 years - The chart compared vaccinated vs unvaccinated all-cause death rates in this broad age group Age mortality difference within 10-59 group: 59-year-olds have death rates around 80 times higher than 10-year-olds - Used to show how strongly age affects mortality Vaccination uptake in 40s and 50s: More than three quarters had received both doses - Shows vaccinated people in the broad band skew older Second dose approval for 16-17 year olds: Only just approved - Illustrates low vaccination uptake among younger people in the 10-59 bracket Vaccination approval for children younger than 16: Not currently approved - Explains why the unvaccinated subgroup contains many younger teens/children Unvaccinated 10-59 demographic composition: Half are under 25 - Stuart's breakdown of the data file showed the unvaccinated subgroup is mostly teenagers and young adults Vaccinated 10-59 demographic composition: More than half are over 40 - Shows vaccinated subgroup is older on average All-cause mortality comparison at ages 60+: Death rates were around twice as high for the unvaccinated - In narrow older age groups, the expected vaccine-protective pattern was visible

Pivotal Quotes: "I don't think it means what you think it means." — Tim Harford: Opening reaction to the viral claim about vaccinated versus unvaccinated deaths "I'd almost certainly found an example of Simpson's paradox." — Stuart MacDonald: After seeing the pattern reverse when data were examined by narrower age groups "The chart is really showing us is that within that 10 to 59 age group, the average unvaccinated person is much younger than the average vaccinated person and therefore has a lower death rate." — Stuart MacDonald: Explanation of why the aggregate chart misleads

Implications: Listeners should be wary of aggregated statistics that ignore age structure or other confounders. For public health reporting, finer age bands and age-adjusted figures are essential to avoid misleading conclusions and viral misinformation.