Episode Summary
Executive Summary: Carl Schulman argues that many existential-risk interventions are justified by ordinary cost-benefit analysis, not just longtermist philosophy. The conversation covers why pandemics, AI, nuclear winter, nuclear weapons, and bioweapons merit attention; how political incentives and defensive epistemology distort responses; why some risks are more neglected than others; and what practical policies—like better sequencing, challenge trials, nuclear-winter food planning, clean-energy R&D, and lab safety—look most promising.
Main Topics: Existential risk can be justified by standard cost-benefit analysis: Schulman argues that even without strong longtermism, preventing catastrophic risks can be rational because the expected value of reducing large-scale harms is enormous relative to cost. Why governments under-prepare for pandemics and other rare disasters: He explains underinvestment via salience, coordination failures, electoral incentives, and defensive epistemology, using COVID-19 as the clearest case study. Nuclear weapons, nuclear winter, and food resilience: The discussion distinguishes direct nuclear destruction from later-recognized nuclear winter and emphasizes alternative food systems and better risk characterization as valuable interventions. Bioweapons and gain-of-function research: Schulman is especially concerned about accidental leaks, weak liability incentives, and the mismatch between lab safety standards and the potential global damage of engineered pathogens. AI risk and current safety efforts: He supports major AI labs’ safety work but says current methods are still far from robust solutions and that the field needs deeper engineering and forecasting research. The intellectual history of 'big deals' and false alarms: He reviews historical claims about existential threats, noting that some were overblown or rhetorical, while others—like nuclear winter and AI—were real but required better empirical grounding. Why this century may be unusually pivotal: Schulman defends the view that the 21st century is especially important because major technological transitions, lock-in risks, and preemption by early catastrophic events make this period unusually consequential.
Key Arguments: Preventing existential risk does not require strong longtermism; standard welfare economics and public policy cost-benefit analysis can already justify major spending. COVID-19 revealed that governments will spend trillions after crises, but much of the spending went to damage control rather than direct prevention; better pre-crisis investments would have been far more effective. Political salience is a major bottleneck: voters, bureaucrats, and politicians respond to visible crises and short election cycles rather than low-probability, high-impact threats. Collective action problems make pandemics, climate change, and biosecurity especially hard because the costs are dispersed while benefits are shared globally or across future administrations. Nuclear winter is more important than asteroid risk or supervolcanoes today because human-made threats are more proximate and technologically intensified. Gain-of-function research is hard to justify because even low leak probabilities imply enormous expected damages, while private and institutional incentives ignore harms imposed on outsiders. A big part of AI and biosecurity policy is improving empirical estimation: the right probabilities on leak risk, escalation risk, and technical capabilities can materially change policy priorities. The 'hinge of history' can be explained without anthropic arguments: if a century contains the first lock-in or extinction-triggering technologies, later centuries may never arrive as pivotal periods.
Data Points: Dinosaur-killer asteroid impact frequency: about 1 in a million centuries - Used to illustrate how astronomical evidence can support cost-benefit interventions like asteroid detection. Government willingness to pay for saving an American life: about $4 million - Rob cites typical U.S. regulatory valuations as a benchmark for public-sector cost-benefit analysis. Value of saving all U.S. lives at a point in time: about $1,300 trillion - Derived from multiplying the value per life by the number of Americans alive. Risk of human extinction over 50-100 years: about 1 in 6 - Attributed to Toby Ord’s estimate in The Precipice and used to justify large investments in risk reduction. Risk reduction example: 1% of a 1/6 extinction risk - Rob’s example of reducing existential risk by 1% of the total risk, generating a very large expected benefit. Willingness to pay for that 1% reduction: about $2.2 trillion - Approximate U.S. spending justified by reducing extinction risk by 1% of the 1/6 estimate. COVID-19 excess mortality: approaching 10 million - Used as evidence that pandemic damages can be enormous and real. COVID-19 economic cost: on the order of $10 trillion - Cited to show the scale of modern pandemic harm. Soviet bioweapons staffing: as many as 50,000 workers - Illustrates the scale of historical state bioweapons programs. Lab infection rate in one study: 4 infections over 2,000 laboratory years - Used as an empirical basis for concern about biosafety and accidental release risk. Lab infection rate per worker-year: about 1 per 500 laboratory years - Converted from the cited study of U.S. biosafety level-3 labs. World War II mobilization in Japan: up to 70% of GDP - Example of the scale of social mobilization that can occur in existentially serious wartime settings. Share of foreign aid budgets: well under 1% - Used to argue that governments currently allocate too little to humanitarian benefit relative to geopolitical aims. Current targeted spending on existential risk reduction: hundreds of millions of dollars annually - Estimated scale of current global spending focused specifically on existential risks. Hypothetical scaled-up spending: $10 billion per year - Used to argue that at higher spending levels, marginal returns would likely diminish sharply. Possible food-supply buffer after atmospheric disruption: years of foodstuffs and alternative foods - Used in discussing nuclear winter, supervolcanoes, and resilience planning.
Pivotal Quotes: "preventing existential risk is one of the great moral issues of our time" — Carl Schulman: His broad normative framing for why he works on these issues even while rejecting strong longtermism. "the risks are really large, horrifyingly, unreasonably big" — Carl Schulman: On why biosecurity and other catastrophic-risk work matters even without Pascalian reasoning. "the thing itself is so big and such an update that you should be taking some angle on that problem" — Carl Schulman: On why major technological risks like AI will demand attention regardless of one’s moral theory.
Implications: Listeners should take catastrophic-risk reduction seriously as a practical policy problem, not just a philosophical one. The biggest opportunities likely lie in neglected, leverage-rich interventions: better lab safety, faster diagnostics and vaccines, alternative food systems, clean-energy R&D, and AI safety research.