Episode Summary
Executive Summary: Carl Shulman argues that reducing existential and global catastrophic risks is justified by ordinary cost-benefit analysis, not just longtermist philosophy. The conversation surveys pandemic, biosecurity, AI, nuclear, climate, and natural-disaster risks; explains why governments underinvest; and highlights practical interventions like better sequencing, safer labs, nuclear-winter research, and clean-energy innovation.
Main Topics: Why existential-risk work is common-sense cost-benefit analysis: Shulman says many x-risk interventions are justified even without strong longtermism because the expected harm from catastrophic events is enormous relative to plausible prevention costs. COVID-19 as a case study in underpreparedness and political incentives: He uses the pandemic to show how salience, coordination failures, and bureaucratic risk-aversion caused underinvestment in direct prevention and overreliance on reactive spending. Biosecurity, gain-of-function, and lab safety: A major focus is the risk that dangerous pathogen research or bioweapons programs could leak accidentally or be misused, with proposals for stronger containment, liability, and flexible surveillance. Nuclear risk, nuclear winter, and food-system resilience: Shulman argues that the most important nuclear danger is nuclear winter-driven famine, and that better modeling plus alternative food research could materially reduce harm. AI as a transformative and potentially destabilizing technology: He sees AI as one of the biggest unresolved risks, but emphasizes that current safety efforts are still preliminary and need major empirical and engineering advances. Historic false alarms, ‘hinge of history,’ and how to interpret the future: The conversation reviews past claims of world-ending risks, arguing that some were overblown, but many real modern threats are still plausibly concentrated in this century due to rapid technological change. Defensive epistemology and why institutions hesitate: He argues that governments and large organizations systematically underreact to low-probability/high-impact risks because they punish failure more than they reward avoided disasters.
Key Arguments: Catastrophic-risk reduction does not require unusual morality; standard government-style cost-benefit analysis can already justify large investments. COVID showed that political salience, short-term incentives, and coordination failures are major reasons societies underprepare for recurring disasters. The most dangerous bio risks are not ordinary natural pathogens but engineered or lab-generated ones, especially if widely deployed or poorly contained. Biosecurity policy should internalize externalities more like pollution or insurance, because private decision-makers do not bear the full social cost of leaks. Nuclear winter is likely a bigger concern than direct blast/fallout effects, so food-production and alternative-food planning are high-leverage interventions. Clean-energy R&D may be a particularly strong climate intervention because technology can scale globally better than country-by-country emissions cuts. Nuclear power could help climate goals, but regulation and permitting often block deployment more than engineering limits do. AI risk deserves attention because it could radically change future trajectories, but current safety work is still far from solving the core alignment problem. Many past ‘extinction’ claims were rhetorical exaggerations; the real challenge is distinguishing serious, empirically grounded risks from publicity-driven alarms. The best explanation for underinvestment is institutional and psychological: people avoid being blamed for visible failures, even when expected value is positive.
Data Points: US value of a statistical life: about $4 million - Used to illustrate how government cost-benefit analysis values preventing deaths Value of saving all U.S. lives at a point in time: about $1,300 trillion - Derived from $4 million per life Toby Ord-style extinction risk estimate: about 1 in 6 over the next 50–100 years - Used as a benchmark for potential benefits of risk reduction Potential value of reducing extinction risk by 1% of 1/6: about $2.2 trillion - Illustrative U.S. government willingness-to-pay calculation COVID-19 economic cost: on the order of $10 trillion - Used to show the scale of pandemic harm COVID-19 excess deaths: approaching 10 million - Context for pandemic severity NASA asteroid search budget: roughly $100 million - Historical example of successful existential-risk mitigation Estimated rate of dinosaur-killer asteroid impacts: about 1 per million centuries - Used to justify why asteroid risk was judged low but nonzero Japanese WWII war effort: up to 70% of GDP - Example showing societies can mobilize huge resources in crises Soviet bioweapons program size: up to 50,000 workers - Illustrates scale of past biological weapons efforts Lab infection rate in a cited BSL-3 study: 4 infections over 2,000 laboratory-years - Used to argue lab work with dangerous pathogens still has nontrivial leak risk Approximate accident frequency in that study: about 1 per 500 laboratory-years - Cited as evidence of ongoing biosafety limitations Potential expected damage from a pandemic-pathogen lab year: billions to tens of billions of dollars - Derived from low leak probabilities multiplied by massive pandemic harm Climate-related foreign aid / spend levels: well under 1% of budgets - Used to argue global public-goods spending is too low Time since last nuclear detonation: about 76 years - Used to discuss nuclear stability and survivorship under risk
Pivotal Quotes: "Preventing existential risk is one of the great moral issues of our time." — Carl Shulman: His broad view on why x-risk work matters even without strong longtermism "The risks are really large, horrifyingly, unreasonably big." — Carl Shulman: On why concern about biosecurity is driven by real-world danger, not speculative tiny probabilities "That is not the reason that I'm doing this at all. It's nothing to do with this argument that while the odds are infinitesimally small, the gain would be infinitesimally large." — Carl Shulman: Rejecting the stereotype that x-risk work is justified only by Pascalian reasoning
Implications: Listeners should expect major future gains from better pandemic defenses, lab safety, food resilience, clean-energy innovation, and AI governance. The broader lesson is that institutions need better ways to price rare, catastrophic downside risks before crises force action.