Episode Summary
Executive Summary: The episode centers on California SB 1047, a proposed AI safety law discussed with Nathan Calvin. The bill would require major model developers to implement safety plans, testing, auditing, and liability for catastrophic harms. Calvin argues it is modest, aligns with existing tort principles, and mainly codifies practices many AI firms already claim to follow, while critics overstate risks to startups, open source, innovation, and national security.
Main Topics: What SB 1047 is trying to do (Priority: 5/5): The bill aims to reduce catastrophic AI risks tied to biosecurity, cyberattacks, and loss of control by requiring safety plans, third-party audits, testing, and liability for severe harms. How the bill was developed (Priority: 4/5): Nathan Calvin describes the bill as emerging from California policy work shaped by Senator Wiener, technical advocates, and AI safety organizations, building on the Biden executive order. Supporters vs. opponents (Priority: 5/5): Support comes from prominent AI researchers, unions, nonprofits, and some startups; opposition comes from major venture firms, big tech interests, and critics like Jan LeCun and some political figures. Criticisms about innovation and startups (Priority: 5/5): Calvin argues the bill’s thresholds are high enough that it does not target startups and that compliance costs are small relative to the cost of frontier model training. Open source and liability debate (Priority: 5/5): A major dispute is whether the bill would chill open source model release. Calvin says the bill is nuanced, does not ban open source, and only affects the largest models under specific circumstances. Federal vs. state regulation (Priority: 4/5): The conversation argues California can act where Congress has stalled, with state-level regulation potentially influencing national and global practice through market power.
Key Arguments: SB 1047 translates voluntary AI safety commitments into enforceable law, creating consequences if developers fail to take reasonable precautions. The bill is targeted only at very large models, so it is not aimed at current models or most startups. Many claims that the bill would send founders to prison or ban open source are inaccurate or exaggerated. Compliance costs are described as modest compared with frontier training budgets, plausibly in the single-digit percent range. Liability is framed as a sensible extension of existing negligence law, not a novel punishment for software companies. The bill is designed to be robust under uncertainty: it does not assume catastrophe is certain, only that risks are serious enough to warrant precaution. If California waits until harms are undeniable, regulation will likely be more drastic and less nuanced than SB 1047. State regulation can matter because large markets like California can shape product design and corporate behavior even beyond their borders.
Data Points: Training compute threshold: 10^26 FLOPs - Bill coverage threshold discussed for future frontier models. Training cost threshold: $100 million - Bill applies to models requiring at least this much compute/training cost. Fine-tuning threshold: $10 million - Fine-tuned models only become covered if additional fine-tuning spend exceeds this amount. Critical harm threshold: $500 million - Definition of critical harm in the bill includes incidents causing this level of damages. Current covered models: 0 - At the time of the discussion, no currently existing models met the bill’s thresholds. California legislative deadline: August 31 - Both legislative houses must pass bills by this date. Governor decision deadline: September 31 - As stated in the transcript, the governor has until this date to sign or veto. Public support: About three-quarters of Californians - Calvin cites polling showing unusually high public support for the bill. Compliance cost estimate: Single-digit percent of training cost - Calvin estimates safety testing and compliance would be a small fraction of training expense. Existing frontier model scale: Around 10^25 FLOPs - He describes current frontier systems as roughly one order of magnitude below the bill threshold.
Pivotal Quotes: "Everyone has a duty to take reasonable care to prevent harms." — Nathan Calvin: Opening argument connecting SB 1047 to existing negligence principles. "What I really view this bill as is taking those voluntary commitments and actually instantiating them into law." — Nathan Calvin: Explanation of the bill’s purpose as codifying current voluntary AI safety norms. "That is, I think, often what this really is." — Nathan Calvin: On companies saying they want regulation but often meaning only regulation that mirrors their current practices.
Implications: The discussion suggests AI governance may move first at the state level, with California setting a precedent for liability-based frontier AI regulation. For companies, the key issue is whether they can show reasonable safety precautions before harms occur.