Feeling the AGI with Flo Crivello

Episode Summary

Executive Summary: Nathan Leven and Flo Crevello discuss Leopold Aschenbrenner’s "Situational Awareness" memo and conclude that AGI is likely near, the next few years are critical, and AI could rapidly escalate into a dangerous U.S.-China arms race. They argue current models are already powerful, multimodal, and cheaper/faster, while safety, regulation, and non-open-source deployment are becoming urgent.

Main Topics: AGI timelines and the case for alarm (Priority: 5/5): Both speakers endorse the memo’s core thesis: AGI may arrive within a few years, and once it does, superintelligence could follow quickly via automated AI research. Evidence that scaling is still working (Priority: 5/5): They rebut the idea that AI has hit a wall by citing cheaper/faster GPT-4-class models, longer context windows, and emerging multimodal systems. Risk from misuse, cyber, and bio capabilities (Priority: 5/5): The conversation emphasizes practical misuse risks more than sci-fi misalignment, especially superhuman cyberoffense and bio misuse becoming accessible to bad actors. U.S.-China AI arms race and geopolitical instability (Priority: 5/5): They argue that if either side believes decisive advantage is possible, competition becomes an attractor, making AI development more dangerous and harder to slow. Policy response: regulation, SB 1047, and open-source concerns (Priority: 4/5): Despite libertarian instincts, both lean toward stronger regulation, limiting open source, and potentially even a pause or stronger state intervention if companies cannot self-regulate. Building for GPT-5 and agent scaffolding (Priority: 4/5): Flo explains Lindy’s strategy: build durable layers around models—memory, planning, tool use, feedback loops, and multimodal inputs/outputs—that remain useful across model generations. Industry structure and future winners (Priority: 3/5): They expect a small set of incumbents—OpenAI, Anthropic, Google, Meta, and perhaps Mistral—to dominate, while second-tier foundation model labs may struggle to matter.

Key Arguments: AI progress has not plateaued: models are already ~GPT-4 level but far cheaper, 10-100x faster, and supported by much larger context windows. The scaling hypothesis remains persuasive; more compute, better architecture, and post-training/unhobbling can keep producing major capability gains. Once AGI exists, AI research itself becomes automatable, making superintelligence a likely near-term continuation rather than a separate distant step. Open-source frontier models are too risky because weights cannot be meaningfully audited and can be relatively easily unshackled or repurposed. Misuse risk is concrete and immediate: powerful models will lower barriers to cyberattacks and bio misuse for states and non-state actors. A U.S.-China AI arms race is a natural attractor if either side believes a decisive strategic advantage is possible. Regulation is normally undesirable, but frontier AI may require it because frontier labs cannot be trusted to self-regulate under competitive pressure. Building agent infrastructure around models—memory, planning, tool use, continuous learning—creates value regardless of the exact model generation. Even if some scaffolding becomes obsolete, current systems can yield large short-term and long-term gains while remaining relevant through the next five years.

Data Points: GPT-4-class cost reduction: ~100x cheaper - Flo says recent models achieve GPT-4-level performance for about a hundredth of the cost. Speed improvement: 10x to 100x faster - Flo cites Gemini 1.5 Flash and similar models as far faster than GPT-4. Context window growth: 4,000 tokens to 1,000,000 tokens - Used to argue that scaling and usefulness have continued advancing dramatically. Email corpus analyzed: ~250,000 tokens - Nathan describes using Gemini Flash to analyze his last 250 emails for a character sketch. Cost of that analysis: Under $0.20 - Nathan says the long-context synthesis cost less than twenty cents. Latency of that analysis: ~45 seconds - Nathan notes the long-context job completed in about 45 seconds. Training-run upper bound: $1 trillion - Flo suggests there is a practical ceiling to raw scaling size. AGI timeline: 2-5 years - Referenced as the dominant view among frontier lab leaders and the podcast’s framing. Takeoff speed: 1-4 years - Flo says he is increasingly convinced takeoff will be relatively fast once it begins. Safety budget proposal: 1 dollar and 1 flop on alignment per 1 dollar and 1 flop on training - Flo proposes a strong alignment spending mandate. Risk window: 2030 / 2032 - Flo argues if AGI does not arrive by then, it likely won’t arrive until roughly 2040. Continuous improvement horizon: 5 years - Flo says many layers they build should remain useful over at least a five-year horizon.

Pivotal Quotes: "AGI is coming and it is time to freak out." — Flo Crevello: Closing summary of Flo’s position on timelines and urgency. "If you understand what's going on, you should freak out." — Flo Crevello: Early in the discussion, Flo argues the memo’s implications warrant alarm. "We need to stop open sourcing the models, like, period." — Flo Crevello: Flo’s policy position on frontier model release and safety risk.

Implications: Listeners should expect faster capability gains, stronger pressure for regulation, and intensified geopolitical competition. The conversation suggests frontier AI is moving from a research topic to a governance and security crisis that will shape the next few years.