#379 — Regulating Artificial Intelligence

Episode Summary

Executive Summary: Sam Harris speaks with California senator Scott Wiener and AI pioneer Yoshua Bengio about SB 1047, a California bill meant to require safety testing and mitigation for frontier AI models before release. The conversation centers on near-term misuse risks, long-term alignment concerns, liability, open-source exemptions, and whether state-level regulation is necessary before federal action arrives.

Main Topics: Why AI safety became a political priority (Priority: 5/5): Scott Wiener explains that concern emerged from conversations within San Francisco's AI community, while Yoshua Bengio says ChatGPT changed his view of how quickly frontier AI is advancing. Near-term misuse vs. long-term alignment risk (Priority: 5/5): The discussion distinguishes between immediate harms from powerful tools in bad hands and the possibility of misaligned superhuman systems that could escape human control. SB 1047’s core requirements (Priority: 5/5): The bill would require safety evaluations and mitigation steps for models above specified training thresholds, aiming to push large labs to formalize existing safety practices. Liability, enforcement, and industry objections (Priority: 4/5): Critics fear lawsuits, business flight, and overregulation, while Wiener argues the bill is narrow, limited to large labs, and creates less liability than existing tort law. Open source and frontier-model scope (Priority: 3/5): The speakers clarify that the bill targets the most powerful future systems rather than most academic/startup open-source work, and they discuss amendments to avoid impossible shutdown obligations. Why state law instead of waiting for Congress (Priority: 4/5): Wiener argues federal action would be preferable but unlikely, pointing to California’s history of stepping in on tech regulation when Washington fails to act.

Key Arguments: AI risk is not settled science; because timelines and capabilities are uncertain, precautionary regulation is rational. Even if systems are aligned, they can still be weaponized by bad actors for cyberattacks, misinformation, or bioweapons. Current large AI labs already claim to do safety testing, so SB 1047 mainly turns voluntary commitments into enforceable obligations. The bill is aimed at frontier models above a training threshold, not startups or most existing open-source work. Existing tort law already allows lawsuits for AI-caused harms; the bill’s liability provisions are narrower than critics claim. State-level action is justified because Congress has been slow or unwilling to pass major tech regulation. Psychological and financial incentives may bias some researchers and investors toward dismissing genuine risk. Catastrophic outcomes could happen within years or decades, so policy should prepare now rather than after an incident. The attacker/defender asymmetry in cyber and biosecurity means more “good AI” does not automatically neutralize misuse risk.

Data Points: Training compute threshold: 10^26 FLOPs - SB 1047 applies to models trained above this level. Training cost threshold: $100 million - The bill also applies to models with at least this much spent on training, indexed for inflation. Safety spend estimate: 2% to 3% - Wiener estimates large labs are already devoting only a small share of spending to safety testing. Bill status: Passed California Senate; heading toward Assembly floor vote - Harris describes the bill as nearing a crucial legislative stage. Timeline shift: A few years to a few decades - Bengio says ChatGPT led him and Jeff Hinton to revise their AGI timeline downward. Existing AI scale comparison: Above all existing models - Bengio says the bill’s threshold is beyond current frontier models like GPT-4-era systems. Household/political risk example: 48 hours - Harris posits a scenario in which a model helps knock out power in half of America for two days. Statewide economic context: 5th largest economy in the world - Wiener uses California’s market power to argue the bill would still matter even if companies relocate some operations.

Pivotal Quotes: "The rational thing to do is to consider all of these scenarios and then act accordingly, according to the precautionary principle." — Yoshua Bengio: Bengio explains why uncertainty about AI timelines still justifies regulation. "This is not about eliminating risk. Life is about risk. It's about trying to get ahead of the risk instead of saying, well, let's wait and see." — Scott Wiener: Wiener summarizes SB 1047’s philosophy of preemptive safety regulation. "If you really believe that these risks are fabricated, made up, just pure science fiction, then why are you concerned about the bill?" — Scott Wiener: Wiener responds to critics who say the dangers are unreal but still lobby against regulation.

Implications: The episode argues that frontier AI may warrant enforceable safety rules now, especially for large labs. For listeners and industry, the takeaway is that voluntary promises may be insufficient and that regulation could shape how powerful models are built, tested, and deployed.