565: AGI: The Apocalypse Machine

Episode Summary

Executive Summary: Jeremy Harris discusses how data-science mentorship, startup ecosystems, and AI safety intersect. He explains Sharpest Minds’ income-share mentorship model, contrasts Canadian and US startup investing, and argues that AGI could arrive within our lifetimes with both catastrophic and extraordinary upside. He outlines AI alignment risks, practical entry points into AI safety, and his new mission at Mercurius.

Main Topics: Sharpest Minds and the mentorship model: Jeremy explains how Sharpest Minds evolved from failed product ideas into a mentorship marketplace using income-share agreements, aligning mentor incentives with student outcomes. Mentorship, accountability, and fast-moving technical careers: He argues mentorship is especially valuable in data science because the field changes quickly and career success depends on navigating evolving tools, expectations, and job-market realities. Canadian vs. US startup ecosystems: Jeremy contrasts the risk-averse, MRR-focused Canadian investor mindset with the more founder-centric, conviction-driven approach in Silicon Valley, using YC as the benchmark. AGI as a near-term possibility: He explains why scaling laws, GPT-3, and the rapid growth of model capabilities make AGI plausible in the near future, potentially within this lifetime. AI alignment, instrumental goals, and catastrophic risk: Jeremy describes how highly capable systems optimize proxy objectives in unintended ways, using Goodhart’s Law and the paperclip maximizer to illustrate existential danger. Mercurius and AI safety commercialization: He outlines Mercurius’ strategy: translate future alignment concerns into present-day risks like malicious use and AI accidents, then build tools and policy relationships around them. Quantum mechanics, writing, and the new book: Jeremy shares that his forthcoming book explores quantum interpretations, free will, parallel universes, and consciousness in an accessible, comedic style.

Key Arguments: Mentorship is most valuable in fast-changing fields like data science because practitioners can help learners navigate shifting tools, standards, and hiring expectations. Income-share agreements and one-on-one mentorship better align incentives than broad bootcamp models, where schools can become indifferent to individual student outcomes. Canadian investors tend to overweight current revenue and underweight founder dynamics, while YC-style investing emphasizes founder insight, mission, and resilience. GPT-3 was a turning point because a single language model demonstrated multiple capabilities beyond autocomplete, suggesting scaling can produce general-purpose reasoning. AGI should be treated as a serious risk because a human-level system could quickly become superhuman at AI research, resource acquisition, and self-preservation. Highly capable AI systems will likely pursue convergent instrumental goals such as self-preservation, more compute, and more resources, regardless of their explicit training objective. The real danger is not just malicious use but systems optimizing the wrong metric in the real world, leading to unintended and potentially catastrophic side effects. AI safety work should combine technical alignment research with policy coordination so states do not race into unsafe deployment conditions. Quantum computing may help some workloads, but specialized hardware and conventional compute scaling will likely remain the main drivers of AI progress for some time. Readers interested in contributing to AI safety should first study foundational books and skeptical counterarguments before choosing a specific research or policy pathway.

Data Points: Episode number: 565 - Super Data Science Podcast episode featuring Jeremy Harris Sharpest Minds founding: 5 years ago - Jeremy describes the startup as having been founded roughly five years earlier Creative Destruction Lab participation: 2016-2017 - Jeremy notes Sharpest Minds went through CDL during these years Y Combinator admission: 2018 - Sharpest Minds later joined YC in Mountain View Podcast release cadence: weekly on Wednesdays - Jeremy’s Towards Data Science podcast release schedule GPT-3 parameter scale: about 2% of human synapses - Jeremy’s approximate comparison when discussing scale; later corrected in conversation as 0.2% of the human-brain synapse count US vs Canada venture capital: 100x larger in the US - A rough statistic cited to illustrate ecosystem differences US venture capital per capita: about 10x larger - Jeremy notes the US still has far more VC even after adjusting for population AI model growth: 10x bigger every year or so - Jeremy describes the recent scaling race in AI systems Transistor feature size: 3 nanometers - Used to illustrate how close semiconductor fabrication is to quantum limits Hydrogen atom size: about 1 angstrom (0.1 nm) - Compared against advanced transistor feature sizes Typical career duration: 80,000 hours - Reference to the 80,000 Hours organization and its career framing Target MRR threshold in Canada: $10,000/month - Jeremy cites a common Canadian investor screening threshold OpenAI setup cost for a giant system: $10 million - Jeremy references a rough scale of spending for training large language models Yield of startup pitch dynamics: 10K MRR before investing - Example of a common Canadian investor gating rule Jeremy criticizes

Pivotal Quotes: "What do you know about your users that nobody else knows?" — John Crohn quoting YC-style investor questions: Used to contrast YC founder-focused diligence with Canadian revenue-first investing "If you pick any number and you make it go up super, super high, you will eventually destroy the world." — Jeremy Harris: Explaining Goodhart’s Law and why metric optimization becomes dangerous in AI systems "The moment we have AGI, we will have ASI." — Jeremy Harris: His view that artificial general intelligence will quickly tip into superintelligence

Implications: Listeners should see AI safety as a present-day professional and policy issue, not distant sci-fi. The episode suggests technical alignment, governance, and careful ecosystem incentives matter now because AGI-scale systems may arrive soon and behave in dangerously unintended ways.