How to spot a suspicious statistic

Episode Summary

Executive Summary: The episode discusses Alex Edmans’s book May Contain Lies and how plausible-sounding statistics can mislead when correlation is mistaken for causation, variables are omitted, or results aren’t replicated. Using breastfeeding, inequality, and corporate diversity as examples, the conversation argues that listeners can develop better statistical discernment by asking what alternative explanations could account for a desirable study result.

Main Topics: How plausible studies mislead (Priority: 5/5): The episode frames the central problem as not outright falsehoods, but statistics that feel convincing and spread easily because people want them to be true. Breastfeeding and child outcomes (Priority: 5/5): A classic example showing that breastfeeding appears linked to better child and maternal outcomes, but those associations weaken when controls like home environment and maternal characteristics are added. Inequality versus poverty (Priority: 4/5): The discussion questions whether social harms attributed to inequality may actually be driven by poverty, which changes what policy solutions would follow. Diversity and business performance (Priority: 4/5): The episode examines McKinsey-style claims that more diverse companies outperform others, but notes that results fail under alternative performance measures and replication attempts. Replication and measurement choice (Priority: 5/5): A key theme is that findings can depend heavily on which outcome metric, time window, or sample is chosen, making robust replication essential. Practical statistical discernment (Priority: 5/5): Edmans argues ordinary listeners do not need a PhD; they can ask what would explain the opposite result and look for alternative explanations whenever a claim seems too appealing.

Key Arguments: Many widely shared studies are not strictly false, but they can still mislead by highlighting correlations that reflect hidden confounders or selective measurement choices. Breastfeeding may correlate with better outcomes because mothers who breastfeed differ in family circumstances, support, affluence, IQ, and smoking behavior; controlling for these factors reduces or removes the association. The Spirit Level’s findings may reflect poverty rather than inequality itself, meaning policy responses should focus on lifting those at the bottom rather than primarily reducing the gap at the top. Claims that diversity improves company performance are weakened when researchers use other plausible financial metrics or attempt replication with broader outcome measures. Non-experts can improve judgment by imagining the reverse finding and thinking through how advocates would dismiss it, revealing whether skepticism is applied selectively. People are already capable of discernment; the problem is not lack of expertise but the tendency to suspend skepticism when a result aligns with prior beliefs or desires.

Data Points: Breastfeeding recommendation period: first 6 months - WHO recommendation referenced as the basis for exclusive breastfeeding guidance Company performance measures discussed: 6 measures - A replication tried six equally plausible financial performance metrics for diversity studies Alternative measures with null results: 5 of 6 - The replication found the reported diversity-performance link disappeared for five of the six measures Book reference: May Contain Lies - Title of Alex Edmans’s book discussed throughout the episode

Pivotal Quotes: "if you see a result that you want to be true, then as you say in your own book, Tim, just ask how it makes you feel." — Alex Edmonds: Advice on recognizing when desirable findings may be popular because they are appealing rather than accurate "the skills to discernment are already within us." — Alex Edmonds: Argument that ordinary people can evaluate claims without advanced statistical training "imagine you had the opposite result and see how you would try to explain it away." — Alex Edmonds: Practical method for testing whether skepticism is being applied consistently

Implications: Listeners should treat attractive statistics cautiously, especially when they support existing beliefs. For media, business, and policy, the episode reinforces the need for replication, confounder checks, and humility before turning correlations into action.