Episode Summary
Executive Summary: Joe Carlsmith discusses AI existential risk, longtermist philosophy, utopia, infinite ethics, anthropics, and the challenge of keeping speculative thinking grounded in reality. He argues we should take seriously the possibility of radically better futures, while also being cautious about ideology, uncertainty, and the dangers of acting too rigidly on incomplete theories.
Main Topics: AI existential risk and timelines (Priority: 5/5): Carlsmith explains his work at Open Philanthropy on AI risk, including estimating timelines and takeoff speeds to understand how quickly AI could become transformative and how much catastrophe risk exists. Utopia and radically better futures (Priority: 5/5): He defines utopia as a profoundly better future, argues it is possible, and says current thinking often underestimates how large the value difference could be between today and a much better civilization. Moral perspective on the future (Priority: 4/5): The conversation explores how future humans might judge the 21st century, and Carlsmith argues that even if future futures are initially alien, what matters is whether they are endorsable from a fully informed perspective. Infinite ethics and infinite worlds (Priority: 5/5): Carlsmith argues ethics must grapple with infinite worlds, because common theories break down there and because there is non-zero credence that the universe is infinite or that acausal influence may matter. Anthropic reasoning: SIA vs SSA (Priority: 4/5): He contrasts self-indication and self-sampling assumptions, favoring SIA while acknowledging deep problems for both; the discussion covers the doomsday argument and how anthropics affects long-termist reasoning. Estimating brain compute and AI capability (Priority: 4/5): Carlsmith describes methods for estimating the FLOPs needed to match human brain function, combining neuroscience, existing AI systems, energy limits, and communication capacity to inform AI forecasting. Grounding futurism and writing practice (Priority: 3/5): He reflects on why futurist discourse can feel unreal, how concreteness is hard to preserve when speculating about the future, and why his blog favors exploration over perfectionistic editing.
Key Arguments: The risk from advanced AI is a major priority because timelines and takeoff speed affect how urgently safety work must be done, and higher catastrophe probabilities can substantially change priorities. A future that seems alien at first could still be genuinely good if one fully understands it; the right standard is endorsement from a richer, more informed perspective rather than immediate familiarity. Utopia should mean a profoundly better future, not merely a small improvement over the status quo; current society likely underestimates the size of possible gains. The path to utopia is not just technological but also philosophical and cognitive: humans may need to become wiser and more capable before they can responsibly build a radically better civilization. Utopian thinking is not inherently tied to dangerous ideology, but it becomes risky when paired with rigidity, conviction, or willingness to break cooperative norms. Infinite ethics matters because ethical theories often fail on infinite worlds, and even low-probability infinity-related hypotheses can affect expected-value reasoning. He prefers SIA over SSA because SSA yields implausible predictions in anthropic cases, though both approaches face major difficulties once infinities are introduced. Brain-compute estimates are necessarily rough triangulations; neuroscience is still too incomplete for precise answers, so any estimate should be treated as uncertain and probabilistic. Futurism can become detached from reality because imagination is lossy; good futurist analysis should preserve a sense of concreteness without pretending the specific imagined future is literal. Practical moral reasoning should allow uncertainty and trade-offs: e.g., insects may matter somewhat, but that does not automatically imply extreme lifestyle changes without a stronger empirical case.
Data Points: AI risk organization: Open Philanthropy - Carlsmith says he works there on existential risk from artificial intelligence. Brain computation estimate: ~10^15 FLOPs - Mentioned as an estimate for matching the human brain’s task-relevant computational capacity. GPT-3 parameter count: 175 billion parameters - Cited as a reference point in discussing AI scale and training requirements. GPT-3 training cost: ~$20 million - Used as an example of current model training cost. Training compute estimate spread: ~10^23 to 10^41 FLOPs - Discussed as a broad range in the training-cost extrapolation work referenced from Open Philanthropy. Evolution anchor: ~10^41 FLOPs - Referenced as an upper-bound-style anchor in the training compute distribution. Centered estimate range: Low 30s (log10 FLOPs) - Carlsmith says the distribution is centered somewhere in the low 30s. Catastrophe probability comparison: 1% vs 10% vs 90% - He says the difference between these ranges is substantively important for prioritization.
Pivotal Quotes: "“I think the difference between, say, 1 and 10% ... is quite substantive. And the difference between 10 and 90 is quite substantive.”" — Joe Carlsmith: On how AI catastrophe probability changes prioritization and urgency. "“The best futures are going to be such that if you really understood them ... then you would think it’s really good.”" — Joe Carlsmith: On how he evaluates alien future worlds and utopia. "“We are at square one in kind of really understanding how these issues play out and how to respond.”" — Joe Carlsmith: On infinite ethics and why future civilization may be better equipped to resolve it.
Implications: Listeners should take seriously both the scale of AI risk and the possibility of dramatically better futures, while staying humble about uncertainty. The conversation suggests future-facing work should combine rigorous forecasting, philosophical caution, and grounded empirical reasoning.