Why the US Must Engage China on AI Safety Before It’s ‘Game Over’

Episode Summary

Executive Summary: The episode centers on AI safety, regulation, and US-China competition, arguing that while AI’s economic impact will likely roll out more slowly than hype suggests, the safety risks are accelerating. Sebastian Mallaby contends governments—not private labs—must govern frontier AI, and that China must be included in any serious international effort to slow dangerous open-source proliferation.

Main Topics: AI’s economic impact will be slower than hype suggests (Priority: 5/5): Mallaby argues that although AI models are improving quickly, real-world deployment faces chip shortages, power constraints, and institutional friction, making major productivity and labor effects more likely over a decade-plus than in 3-5 years. Why private labs cannot solve AI governance alone (Priority: 5/5): He says companies like OpenAI, DeepMind, and Anthropic have incentives to move fast and satisfy customers, so they cannot reliably impose safety constraints without government-backed rules and international coordination. UK case study: DeepMind, NHS, and missed opportunity (Priority: 4/5): Mallaby uses DeepMind’s relationship with Google and its aborted NHS collaboration to show both the upside of foreign tech investment and the costs of public suspicion and weak policy design. US-China competition and the need for arms-control-style cooperation (Priority: 5/5): The discussion argues that AI safety should not be sacrificed to geopolitical rivalry; Mallaby believes China is only months behind the US frontier and that both sides need a channel for safety dialogue. Open source as a major safety risk (Priority: 5/5): The conversation emphasizes that open-source frontier models can spread dangerous capabilities widely, enabling cybercrime, financial theft, and possibly biothreats if released without controls. Existential risk and alignment remain unresolved (Priority: 4/5): Mallaby distinguishes between misuse by bad actors and the possibility of systems developing survival-like behavior, stressing that alignment may be possible but requires time that rapid competition erodes. Trump-era acceleration versus prior regulatory momentum (Priority: 4/5): He describes a shift away from regulation after Trump’s return, warning that the current US policy posture is more focused on competition and may allow dangerous models to diffuse before safeguards are in place.

Key Arguments: AI’s most dramatic effects are likely to be delayed by infrastructure and organizational constraints, not immediate model capability alone. Recursive self-improvement is plausible in computer-science terms, but it does not automatically translate into rapid economic transformation. Labs and firms will generally build the tools customers want, and customers often prefer labor replacement over complementarity when competitors are doing the same. Britain lost a major opportunity by allowing political suspicion to derail DeepMind’s NHS work, despite real potential benefits. Governments should shape adoption and safety standards, because private labs lack legitimacy and the incentive structure to regulate themselves properly. US-China AI safety talks are necessary even amid strategic rivalry; refusing to engage risks uncontrolled global proliferation. China is not monolithically indifferent to safety; Mallaby reports real internal discussion and concern among Chinese AI leaders. Open-source frontier models are especially dangerous because they can be copied and spread quickly across borders and into malicious hands. The biggest risks are cyberattacks, financial system compromise, infrastructure sabotage, and bioweapons, with existential machine behavior a separate but real concern. If regulation keeps lagging and models keep advancing, alignment will be harder to achieve before the next generation of systems arrives.

Data Points: Chinese AI frontier lag: about 6 months - Mallaby says China’s cutting-edge AI labs are only roughly six months behind the US frontier. DeepMind acquisition year: 2014 - He notes DeepMind was sold to Google in 2014, which critics in Britain often cite. AlphaFold timeline: 6 years later - He says no drug has yet been discovered thanks directly to AlphaFold roughly six years after its 2020 breakthrough. AlphaFold release: end of 2020 - The transcript references AlphaFold as a DeepMind system published at the end of 2020. NHS collaboration period: first 5 years pro bono - Mallaby says DeepMind was willing to help the NHS without charge for the first five years. Open source follower labs: Meta, Mistral, Chinese labs - He cites these as examples of non-frontier players competing through openness rather than best-in-class performance. AI safety institutes meeting: fall of 2024 - He mentions a meeting of national AI safety institutes in San Francisco in autumn 2024. Current scenario year referenced: 2026 - He frames the warning scenario as one that has arrived by 2026, with dangerous models already available.

Pivotal Quotes: "The door is open just a little bit. Crack." — Sebastian Mallaby: Describing the possibility of AI safety dialogue with China and urging US negotiators to engage. "We are going to arrogate unto ourselves the power to say these are the 40 entities that will get access to this technology." — Sebastian Mallaby: Critiquing Anthropic’s attempt at private governance over access to powerful models. "Inventors dream of shaping the technology that they create. Often the technology shapes them." — Stephanie Flanders quoting Mallaby: A central framing of the episode’s argument about how institutions and geopolitical forces overpower individual intent.

Implications: The episode argues that AI safety cannot be left to market competition or private lab discretion. If governments, especially the US and China, don’t coordinate quickly, dangerous models may spread globally before meaningful alignment or controls are in place.