Bioinfohazards: Jassi Pannu on Controlling Dangerous Data from which AI Models Learn

Episode Summary

Executive Summary: The episode argues that AI is rapidly lowering barriers to designing, analyzing, and potentially synthesizing dangerous pathogens, making biosecurity a near-term governance problem. Jossy Panu explains how outbreak response, vaccine design, and bio-data generation work today, then proposes a tiered biological data control framework to restrict only the small fraction of functional/pathogen-linked data that can enable harmful capabilities, while preserving open science for the rest.

Main Topics: Current outbreak detection and response pipeline (Priority: 5/5): Panu walks through how new viruses are detected today: symptom-based clinical testing, follow-on metagenomic sequencing, and then diagnostic/vaccine development. He emphasizes that detection is fragmented and slow compared with the pace of viral spread. Why pandemic-capable pathogens are a unique biosecurity risk (Priority: 5/5): The discussion distinguishes between toxins, non-transmissible agents, and truly pandemic-capable viruses. The key concern is that novel transmissible viruses are offense-dominant and far harder to defend against than localized biological threats. AI capabilities that could amplify bio-risk (Priority: 5/5): The conversation maps risk across LLMs, specialized biodesign tools, and biology foundation models. Panu argues that as AI agents, robotics, and integrated workflows improve, the ability to search, design, and test dangerous biological ideas will become more accessible. Proposed biological data-level controls (BDL framework) (Priority: 5/5): Panu co-authored a proposal for tiered controls on biological data, analogous to biosafety levels for labs. The framework keeps most data open but restricts a narrow slice of functional, pathogen-linked data that is directly relevant to transmissibility, virulence, host range, or immune evasion. Wet-lab gain-of-function research and current policy gaps (Priority: 4/5): The episode reviews historical gain-of-function debates, especially the 2012 avian flu ferret studies and horsepox/smallpox-related information leakage. Panu says much of this work is defunded in the public sector but still exists privately and is not comprehensively governed. Defense-in-depth biosecurity strategy (Priority: 5/5): The closing segment broadens the answer beyond data controls: delay dangerous capabilities, deter misuse, detect outbreaks earlier, and defend through vaccines, wastewater surveillance, PPE, HEPA/FAR UV, and other built-environment protections.

Key Arguments: Outbreak response today is still largely symptom-driven and fragmented; new pathogens are not detected through a unified passive alert system. Vaccine design can be very fast once a pathogen is sequenced, but clinical trials, manufacturing, and global deployment remain major bottlenecks. Nation-states are not the most likely actors for pandemic bioweapons because they cannot easily protect their own populations; lone actors and small extremist groups are more concerning. Open biological data is broadly valuable, but a very small subset of functional data can directly support harmful capabilities and should be controlled. Data controls should target future high-risk datasets rather than trying to scrub the internet of already-published information. The proposed BDL framework would protect roughly the vast majority of data while limiting access to only a narrow, high-risk subset. Empirical evidence from EVO2 and ESM3 suggests that excluding dangerous data can sharply reduce harmful model capabilities while preserving much of the useful performance. Private labs remain a blind spot for governance, since public funding restrictions do not fully reveal what private actors are doing. Biosecurity should be treated as defense-in-depth rather than relying on a single policy lever or a nuclear-style deterrence model. Trusted research environments offer a practical way to allow code-to-data access without broadly distributing sensitive biological data.

Data Points: Case fatality rate of wild-type bird flu referenced in 2012 example: ~60% - Panu cites avian influenza as a highly lethal but not-human-transmissible pathogen used in gain-of-function studies. Mutations required for transmissibility in the 2012 avian flu experiment: 5 mutations - One of the cited studies found human-to-human transmissibility could emerge with only five mutations. COVID-19 sequence disclosure timeline: Sequenced in January 2020 - Panu notes the virus likely emerged in late Nov/Dec 2019 but was sequenced and shared in January. UK OpenSafely coverage: 95% of the UK population - Used as an example of a national trusted research environment for clinical data. GenBank size: 40 petabytes plus - Panu describes the NIH-supported repository as containing massive amounts of unannotated raw DNA sequencing data. Protein databank size: Less than 1 terabyte - He contrasts the small size of curated protein structure data with massive sequence repositories. Biological data control framework tiers: 0 to 4 - The proposed biological data-level (BDL) system mirrors biosafety-style tiering. Estimated share of data needing restriction: About 1% - The proposal is meant to keep the vast majority of biological data open while restricting a tiny high-risk subset. Model holdout effect in EVO2: Function effectively random on held-out viral tasks - Panu says excluding dangerous data caused strong degradation on targeted harmful capabilities. Gene synthesis screening adoption: 80% of companies - Most providers already screen orders, though the system is currently voluntary. Public-data access to UK research platform: 95% - OpenSafely gives researchers access to clinical data through a secure code-to-data model.

Pivotal Quotes: "the whole globe is your patient" — Jossy Panu: Used to explain why pathogen-related data should be treated as a societal-risk issue rather than just an individual privacy issue. "data wants to and ought to be free" — Host: The host contrasts his prior techno-optimism with his support for narrowly targeted biosecurity controls. "Delay, deter, detect, and defend" — Jossy Panu: Panu’s summary of a layered, defense-in-depth biosecurity strategy.

Implications: Biosecurity policy may need to shift from broad openness to narrowly targeted controls on high-risk biological data, paired with stronger monitoring and physical defenses. For AI builders, the episode signals increasing scrutiny on models and datasets that could enable pathogen design.