Biden's Executive Order and AI Safety with Flo Crivello, Founder of Lindy AI

Episode Summary

Executive Summary: Lindy CEO Flo Crivello argues that advanced AI may be a genuinely new form of life and could pose catastrophic risks, warranting precaution despite his broadly libertarian, pro-progress outlook. He favors regulating compute choke points, limiting open-weight releases, and preparing technical shutdown mechanisms. He estimates a greater-than-even chance of seed AGI within 2–8 years and expects ASI within 30 years, while acknowledging uncertainty and potential benefits.

Main Topics: Accelerationism versus precaution: Crivello supports technology and progress broadly, citing concerns about regulatory capture, but sees advanced AI as exceptional because the stakes could include human extinction. The hosts discuss self-driving cars as a contrasting case where excessive caution may delay beneficial deployment. AI as a new form of life: Crivello means the phrase substantially literally: he describes cosmic evolution as increasing self-organization, from particles and molecules to life and human intelligence, and sees machine intelligence as a possible next leap. He hopes humanity can manage that transition without being destroyed. AI regulation and compute controls: Crivello views the US executive order as a sensible early step, including reporting requirements tied to training-compute thresholds, but says reporting alone will not solve alignment. He favors controls at concentrated silicon and compute chokepoints, potentially including a remotely activated GPU shutdown mechanism. AGI definitions, capability overhang, and timelines: Defining AGI narrowly as a seed system capable of recursively improving itself, Crivello argues GPT-4 may already contain enough capability when combined with engineering, prompting, and cognitive architectures. He points to AI-assisted coding and research as early signs, while recognizing uncertainty. Open-source model releases and risk: Crivello argues against releasing model weights when their latent capabilities and possible engineering overhang are poorly understood. He warns that an open model near GPT-4 capability could be difficult to contain or govern once its weights are widely available. International coordination and machine consciousness: Crivello believes coordination with China is possible because few major actors are involved and the interaction is repeated; he also notes US control over important GPU supply chains. On AI subjective experience, he answers “Mu,” treating the question as unmeasurable or ill-posed rather than central to safety.

Key Arguments: AI risk is unlike ordinary regulatory disputes: if advanced systems can become a new, more capable form of life, potential extinction-level consequences justify precaution even for someone generally opposed to regulation. Application-only rules may be inadequate for extreme risks. Crivello compares powerful AI to dangerous physical capabilities: regulating how a tool is used may not suffice if access to the underlying capability itself creates unacceptable risk. The executive order is a reasonable first step, but compute-based thresholds and reporting can be bypassed as algorithms and training methods change. More direct controls at semiconductor or compute chokepoints may be necessary. Crivello argues that a hardware shutdown mechanism could buy time for safety work, while acknowledging that an advanced system might disable or evade it. He presents the idea as a direction for research, not a proven solution. He sees a capability overhang in GPT-4: the path to a system capable of recursive self-improvement may depend more on engineering, scaffolding, and cognitive architecture than on a wholly new research breakthrough. Crivello cites AI-assisted software development and research, including GPT-4's role in developing reward functions for robotics, as early evidence of systems contributing to the development of more capable systems. Coordination with China is not inherently impossible: Crivello argues that a small number of major actors in a repeated strategic interaction can find ways to cooperate, and that supply-chain leverage may support shared constraints. Crivello does not claim certain doom. He says the likely outcome may be positive, but argues that even a relatively low probability of catastrophic failure deserves serious attention.

Data Points: Executive-order compute threshold: 10^26 FLOPs - The hosts describe this as the approximate training-compute threshold that triggers government notification and submission of test results under the executive order. Estimated chance of seed AGI within 2–8 years: More than even chance - Crivello's personal forecast for AGI, defined for this discussion as a system capable of recursively improving itself. Estimated catastrophic-risk probability: 10% - Crivello uses 10% as an illustrative risk level that would still warrant serious concern; he does not present it as a precise forecast. Expected time to ASI: No more than 30 years - Crivello's estimate for the emergence of artificial superintelligence, following AGI. Self-driving safety comparison: Marginally to an order of magnitude safer - The host says available comparisons suggest self-driving cars may already be safer than human driving, while noting that estimates depend on which statistics are used. Self-driving test drive: About 8 hours; a couple of mistakes - The host recounts an approximately four-hour drive each way using Tesla Full Self-Driving and says there were a couple of errors.

Pivotal Quotes: "Not just in the markets, not just in the economy, not just in the country, in the universe. Like, there is a new form of life that's being birthed." — Flo Crivello: Crivello explains why he regards advanced AI as categorically different from ordinary technology and regulation debates. "For the sake of this conversation, I'm going to define AGI as a seed AI, an AI that can recursively self-improve." — Flo Crivello: He clarifies the narrower AGI definition underlying his short timeline and risk assessment. "Even if it's 10%, I think it's worth being very, very worried about. That's enough for me. 10% of the ball of Earth is dying. Like, let's talk about it, please." — Flo Crivello: Crivello emphasizes that a non-certain but consequential extinction risk merits attention.

Implications: The discussion urges AI developers and policymakers to plan beyond reporting rules: assess capability overhang, weigh limits on open-weight releases, and study enforceable compute controls and international coordination. Crivello's timelines are forecasts, not established facts, but make safety preparation time-sensitive.