Episode Summary
Executive Summary: The episode follows Ajeya Cotra’s work at Open Philanthropy on worldview diversification, longtermism, and forecasting transformative AI. It explores how Open Phil balances competing moral perspectives, why longtermist arguments about astronomical waste and existential risk matter, and how empirical work on AI timelines and future value informs grantmaking and spending decisions.
Main Topics: Worldview diversification at Open Philanthropy: Ajeya explains how Open Phil divides giving across longtermist, near-term human-centric, and animal-inclusive worldviews, and why it treats these not only as moral philosophies but also as practical strategies for allocating philanthropic capital. Fairness, credence, and fanatical expected value: The conversation contrasts three approaches to allocation across worldviews: funding in proportion to credence, giving all weight to the worldview with most at stake, or using fairness/veil-of-ignorance-style trades between worldviews. Diminishing returns and spending over time: They discuss how Open Phil thinks about the 'last dollar'—whether to spend now or later—and how declining returns, learning, and future opportunity availability affect the timing of grants. Longtermism, astronomical waste, and anthropics: The episode digs into the case for longtermism: a potentially enormous future in space, along with doomsday and simulation arguments that complicate estimates of future value and make the worldview intellectually and emotionally unstable. Forecasting transformative AI with biological anchors: Ajeya describes her report estimating the probability and timing of transformative AI using four components: compute needed today, algorithmic progress, hardware cost decline, and growth in spending on AI training. Open Phil culture and research process: The episode reflects on the difficulties of doing open-ended, decision-relevant research in a new field, including uncertainty about 'publishable units,' defensiveness in writing, stress, and the challenge of working in a siloed organization. Careers and fitting into Open Phil: Ajeya offers advice for listeners interested in similar work, emphasizing generalist research, explainers, open applications, and the importance of fit, persistence, and good epistemic style.
Key Arguments: Open Phil needs worldview diversification because different moral perspectives imply very different beneficiaries and time horizons; no single framework obviously dominates. A worldview should not necessarily receive funding strictly in proportion to credence if its favored world contains much more potential moral value than others. Fairness agreements/veil-of-ignorance reasoning can justify reallocating resources across worldviews, but they depend on hard-to-set priors about what counts as the relevant world. Longtermism is not just philosophy; it is also a strategic bet on where Open Phil has comparative advantage, versus near-termist work being more empirical and quantitative. The longtermist case depends heavily on a very large, robust future, especially one enabled by space colonization and reduced existential risk. Biological human space colonization looks much harder to defend than digital/AI-based colonization, so the strongest longtermist case likely involves uploaded minds or computers. AI timelines can be estimated by combining compute requirements, algorithmic progress, hardware cost trends, and how much society will spend on frontier systems. Even if timelines are uncertain, the probability of transformative AI within the century is high enough to strongly affect current prioritization and safety work. A lot of important research in this area cannot be fully formalized; judgment, intuition, and defensible estimates matter, though they must be handled carefully to avoid 'physics envy.'
Data Points: Open Philanthropy worldview buckets: 3 major buckets - Longtermism, near-term human-centric, and animal-inclusive worldviews Probability of transformative AI by 2036: 12% to 15% - Ajeya’s updated estimate after the biological anchors report Open Phil/AI earlier estimate: at least 10% by 2036 - Holden Karnofsky’s 2016 estimate referenced in the interview Median transformative AI year: 2050 to 2060 - Ajeya’s rough median range from the report Probability transformative AI this century: 70% to 80% - Ajeya’s bottom-line summary Human brain compute estimate: 10^13 to 10^17 FLOP/s, central estimate 10^15 - Referenced from Joe Carlsmith’s work on brain computation Compute cost of GPT-3 / AlphaStar: 1 to 10 million FLOPs (approx. training compute) - Publicly calculable range mentioned for the largest models discussed Relative model size: AlphaStar was about 3,000x smaller than GPT-3 - Used to illustrate effective horizon length / feedback density differences Algorithmic progress rate: About 12 to 24 months per 10x improvement (roughly) - Based on earlier papers and benchmark improvements Hardware progress assumption: About 6 orders of magnitude improvement over the next 80 years - Ajeya’s rough projection from historical chip progress AI compute scaling assumption: Doubling every 6 months in the near term, then slowing to about every 2 years - Her model of spend growth and frontier training runs Cost-effectiveness estimate for one biosecurity intervention: About $200 trillion per world saved - A rough longtermist last-dollar estimate for speeding pathogen response
Pivotal Quotes: "what kind of result is big enough and what kind of argument is compelling enough and complete enough that you can like package it into a paper and publish it" — Ajeya Cotra: On the difficulty of defining a publishable unit in open-ended research "I sometimes think of it as like a train going to crazy town" — Ajeya Cotra: Her metaphor for where some philosophical arguments push longtermism if taken too far "I think I was kind of previously thinking of AI as... this kind of onrushing tide" — Ajeya Cotra: How the timeline project changed her emotional and strategic view of AI risk
Implications: The episode suggests that high-stakes philanthropy increasingly depends on mixing philosophy, forecasting, and empirical judgment. For listeners, it implies that AI safety, longtermist uncertainty, and spending timing are not abstract—they directly shape career choices, grantmaking, and research priorities.