Policymaking Is Not a Science — Yet (Update)

Episode Summary

Executive Summary: The episode argues that a major reason good research fails to improve lives is not bad science, but the difficulty of scaling effective interventions in the messy real world. Through examples from cochlear implants, blood-pressure adherence, early-childhood language programs, and foster care reform, it shows why implementation science—replication, fidelity, and understanding context—is essential to turn evidence into policy.

Main Topics: The scaling gap between research and real-world impact (Priority: 5/5): The episode’s central thesis is that interventions often work in trials but fail when expanded because human systems, institutions, and incentives change at scale. Implementation science as the missing discipline (Priority: 5/5): Implementation science is presented as a newer field focused on how programs are delivered, whether fidelity is preserved, and why quality changes when interventions move from trial to practice. Three causes of scaling failure (Priority: 5/5): John List and Dana Susskind organize failures into three buckets: insufficient evidence, studying the wrong people, and using the wrong situation/context when scaling. Early childhood language and the TMW Center (Priority: 4/5): Dana Susskind explains how research on language exposure in the first three years of life led to a population-health approach and multi-setting programs aimed at parents where they already are. Treatment Foster Care Oregon and fidelity (Priority: 4/5): Patty Chamberlain’s foster care model demonstrates both the promise of evidence-based interventions and the challenge of scaling across systems with conflicting rules and procedures. Replication, incentives, and policy credibility (Priority: 4/5): The discussion argues that policymakers should wait for multiple strong replications before scaling programs and that academia should reward replication more directly.

Key Arguments: Great interventions can fail not because the idea is wrong, but because the implementation context changes when the program is expanded. Policy should be guided by evidence that has been independently replicated several times, not by a single promising trial. Researchers often study highly selected participants or ideal settings that do not represent the broader population, causing effects to shrink or disappear at scale. Scaling depends on fidelity: the delivered program must preserve the core ingredients that produced the original effect. Systems like education, health care, and child welfare are built for efficiency and compliance, not for adopting innovations, so implementation barriers are structural. The right question is not only whether a program works, but whether it can be delivered reliably, affordably, and at sufficient volume in real-world conditions. Reward structures in academia discourage replication even though replication is necessary for trustworthy policy decisions.

Data Points: Americans with high blood pressure: about one-third - Used to illustrate the public-health importance of medication adherence and policy implementation. Blood-pressure awareness rate: about 80% - John List describes the hypertension cascade, noting many people are aware but still not controlled. Blood-pressure control rate: about 50% - Only half of people with high blood pressure have it controlled, showing the gap between available treatment and real-world outcomes. Device use in cochlear implant study: only half of participants wore their device full time - Demonstrates that even obvious-benefit medical technologies can face adherence problems. 30 Million Words: estimate of 30 million fewer words - Original estimate behind Dana Susskind’s early-childhood language initiative, later rebranded as the TMW Center. Early childhood window: first 3 years of life - Susskind emphasizes this as a critical period for brain development and language exposure. Parent Academy effects: 3 to 6 months - List reports strong short-term gains in children’s cognitive and executive-function test scores in Chicago Heights. Program rollout failure in London: 0 parents signed up - The Parent Academy failed at scale because no parents enrolled, despite promising pilot results. Share of prevention programs backed by evidence: 8% - A Department of Education survey found only a small fraction of youth prevention programs had research evidence. Treatment Foster Care Oregon outcomes: fewer arrests, fewer days in institutions, fewer drugs, lower cost - Chamberlain’s program showed both better youth outcomes and cost savings. Implementation science age: about 15 years old - Lauren Suplee describes the field as relatively new and emerging from evidence-based policy movements.

Pivotal Quotes: "We need to know what is the magic sauce." — John List: Used to frame the central challenge of identifying which elements of an intervention are essential for successful scale-up. "I want policy science not to be an oxymoron." — John List: A concise statement of the episode’s ambition: making policy more scientific and trustworthy. "The science is slower than you'd like." — Dana Suskind: Suskind reflects on the tension between urgent social needs and the slower pace required to build reliable evidence.

Implications: Listeners should be skeptical of one-off “successful” programs and ask whether results replicate, preserve fidelity, and fit real-world systems. For policy and funding, the episode argues for slower, more rigorous scaling and stronger incentives for replication.