John List on How to Make Good Ideas Great & Great Ideas Scale

Episode Summary

Executive Summary: Bob Hahn interviews economist John List about The Voltage Effect, a book on why many promising ideas fail when scaled. List explains how field experiments, implementation science, and “policy-based evidence” can identify false positives, situational limits, and supply-side constraints before rollout. He uses examples from education, vaccines, restaurants, culture, Uber/Lyft, and climate tech to show how to preserve “voltage” at scale.

Main Topics: Why List wrote The Voltage Effect (Priority: 5/5): List says policymakers pushed him to explain why great academic ideas often disappoint when expanded from pilot to full scale. The book is his effort to add science to the problem of scaling and implementation. Field experiments and using the world as a lab (Priority: 5/5): List describes field experiments as randomized tests conducted in real settings, extending A/B testing to uncover not just whether something works, but why it works and under what conditions. Five vital signs of scalable ideas (Priority: 5/5): List outlines five checks for scale: avoid false positives, identify who the idea works for, test situational fit, account for spillovers, and evaluate the supply-side economics of scaling. Failures and successes in scaling public programs (Priority: 5/5): He contrasts DARE/Just Say No as a false positive that wasted money with the polio vaccine as a scalable success because it passed all five vital signs and fit existing healthcare systems. COVID vaccine rollout and behavioral barriers (Priority: 4/5): List argues the vaccine was a scientific success but an uptake failure due to politicization, small transaction costs, and hesitancy. He suggests reducing transportation barriers and using behavioral nudges. Organizations, culture, and human limits to scale (Priority: 4/5): Using Jamie Oliver’s restaurant chain and Brazilian fishing villages, List shows that businesses and organizations scale best when the underlying process—not a unique person—can be replicated and culture supports cooperation. Policy-based evidence and the future of public decision-making (Priority: 5/5): List argues researchers should design studies for real-world constraints at scale, not merely publish efficacy results. Governments should measure outcomes continuously and be willing to adapt or exit failed programs.

Key Arguments: Many good ideas fail not because they are bad in small pilots, but because they lose “voltage” when scaled into different settings, larger populations, or more complex institutions. A field experiment should do more than estimate an average treatment effect; it should test the mechanisms, subgroup effects, and situational conditions that determine whether scaling is possible. The DARE/Just Say No campaign illustrates a false positive: one promising study did not replicate across other cities, yet millions were spent on rollout. The polio vaccine is a model scalable intervention because it worked in trials, generalized across groups, fit the healthcare delivery system, had minimal spillovers, and benefited from economies of scale. COVID vaccine development was a major scientific success, but uptake lagged because the problem became politicized and even small barriers (time, transport, uncertainty) reduced adoption. Restaurants often fail at scale when the “secret sauce” is a specific person; they scale when the process is replicable, as with standardized chains like Domino’s. Large organizations and governments suffer from silos and weak feedback loops; firms are often more able to measure, adapt, and reverse course than public agencies. Culture can be operationalized and studied experimentally; hiring language, compensation norms, and CSR signals affect applicant pools, diversity, cooperation, and productivity. Researchers should move from “evidence-based policy” to “policy-based evidence,” meaning they should test ideas under the constraints policymakers will actually face before claiming policy relevance. For climate and EV adoption, scaling requires both price incentives and behavioral design; technologies must be affordable, usable, and robust to ordinary human behavior, not just engineering assumptions.

Data Points: Year of podcast: January 31, 2021 - Opening introduction to the episode Chicago Heights Early Childhood Center opening: 2010 - List’s example of a program built from scratch with Fryer and Levitt Personal control group: 8 children, including a set of twins - List jokes about trying ideas on his kids first DARE Honolulu study sample size: 1,777 kids - List cites the original study behind the anti-drug program False positive rate (alpha): 5% - List explains that even well-run experiments can still lie by chance Uber tipping rollout: 1 million people per treatment arm - Example of A/B testing in a field setting with different tip-display formats Restaurant scaling example: Jamie’s closed after rapid expansion - List says the core talent could not be replicated across locations Polio vaccination spillover: Near zero negative spillovers / strong positive public-health effects - List argues vaccinated children do not create harmful spillovers in the way some programs do Chief economist role at Lyft: 4 years - List mentions recent private-sector work and transport-barrier experiments Job ad experiment: Women negotiated more when wages were described as negotiable - Illustrates how wording changes applicant behavior and wage outcomes Evidence comparison: Efficacy tests vs phase 1/2/3 style rollout - List contrasts academic publication norms with medicine’s staged testing Children’s health appointment cadence: 6-, 12-, and 18-month visits - Used to explain why the polio vaccine fit existing healthcare routines Smart thermostat finding: Zero energy savings - List says real users did not behave like engineers assumed Target adoption for EVs: Large fraction of the population in 10 years - Hahn frames the climate-policy example around adoption scale

Pivotal Quotes: "Professor List, this is a great program, but we don't think it will scale." — Bob Hahn (quoting policymakers): The moment that prompted List to focus his career on scaling and implementation "The world is replete with those kinds of examples." — John List: List introducing failed scale-ups, especially the DARE campaign "Policy-based evidence." — John List: His phrase for designing research around the constraints and realities of real-world implementation

Implications: For policymakers and businesses, the lesson is to test not just whether an idea works, but whether it survives real-world constraints, incentives, and human behavior. Scaling requires mechanism, measurement, and adaptability—not just a good pilot.