405. Policymaking Is Not a Science (Yet)

Episode Summary

Executive Summary: The episode argues that many interventions that work in controlled trials fail in the real world because scaling is not automatic. Through examples from cochlear implants, hypertension treatment, early-childhood education, and foster care programs, it shows that implementation science and replication are essential to turn promising research into durable policy.

Main Topics: Why good interventions fail at scale (Priority: 5/5): The core thesis is that effective programs often underperform when expanded because real-world settings differ from research conditions and because human behavior, systems, and logistics disrupt uptake. Implementation science as a missing discipline (Priority: 5/5): Experts define implementation science as studying how programs are delivered, fidelity to the original model, and how implementation quality changes outcomes. Scaling failures in education and social policy (Priority: 5/5): John List and Dana Suskind describe how programs that succeeded in pilot studies failed when rolled out broadly due to low participation, wrong populations, or incompatible institutions. Replication before expansion (Priority: 4/5): The speakers argue that programs should not be scaled until findings are independently replicated multiple times, to reduce false positives and avoid costly policy mistakes. Fidelity, dosage, and context matter (Priority: 4/5): Successful scaling requires measuring whether the original components are preserved, whether the right intensity is delivered, and whether the context changes the effect. Examples from child welfare and early childhood development (Priority: 4/5): Treatment Foster Care Oregon and the 30 Million Words/TMW Center illustrate how programs can adapt, track fidelity, and adjust delivery channels to achieve broader impact. People are the bottleneck (Priority: 4/5): A recurring theme is that scaling problems are mostly about people—uptake, adherence, staffing, training, motivation, and institutional coordination—not about whether the idea sounds good on paper.

Key Arguments: A program proven in a randomized controlled trial may only show what works in a specific population and context; that does not guarantee generalizability. The biggest scaling failures come from three buckets: insufficient evidence, studying the wrong people, and using the wrong situation/context. In real-world rollout, uptake can be the main failure point, as shown by the Parent Academy program that succeeded locally but failed in London because parents did not enroll. Human-service interventions are especially vulnerable to scaling problems because they depend on staff quality, delivery consistency, and institutional coordination. Policy should reward replications and credible evidence, not just original studies, because premature scaling can waste money and damage trust in research. Fidelity measurement is essential: successful scaling requires checking whether the scaled program still matches the components that produced the original effect. Researchers and policymakers often overestimate transferability; to scale effectively, they must design for implementation from the start, not after the fact.

Data Points: Americans with high blood pressure: about one-third - Example of a health problem with major adherence and control gaps Awareness rate for hypertension: about 80% - Among Americans with high blood pressure Hypertension control rate among aware patients: about 50% - Even with effective drugs, many patients remain uncontrolled Study adherence for cochlear implants: only half wore their device full time - One study cited as an example of adherence problems despite obvious benefits Time for Parent Academy results: within 3–6 months - Pilot program improved children's cognitive and executive function scores quickly TMW estimate of early language gap: 30 million fewer words - Original name of the early-childhood initiative, later deemphasized Alternative replication estimate of language gap: 4 million - Mentioned as a later study that challenged the headline number Age window emphasized for language exposure: first 3 years of life - Period when parent talk and interaction are described as catalytic for brain development Share of prevention programs backed by research: 8% - Department of Education survey on youth substance/crime prevention programs Treatment Foster Care Oregon sites: more than 100 sites - Program spread in the U.S. and abroad after scaling and fidelity systems Original implementation sites for foster care rollout: 15 sites - Federal request to replicate the program across multiple locations Replications proposed before scaling: 3 or 4 well-powered independent replications - List and Suskind's proposed standard before broad rollout Psychology of evidence threshold: 95% certain - Suggested confidence level before scaling a program

Pivotal Quotes: "The science of using science" — Dana Suskind / John List: Describes the broader research agenda focused on scaling and implementation "We do not believe that we should scale a program until you're 95% certain the result is true." — John List: Explains the proposed standard of multiple independent replications before rollout "I want policy science not to be an oxymoron." — John List: Summarizes the goal of making evidence-based policymaking actually workable in practice

Implications: Policymakers should treat rollout as a separate scientific problem from discovery. Programs need replication, fidelity tracking, and implementation planning before expansion, or else promising ideas will keep failing at scale.