Episode Summary
Executive Summary: Miles Brundage argues that AI policy is still in an early, high-level phase, but it is already urgent to study near-term security risks, bad actors, and long-term coordination problems. He emphasizes building incentive-compatible safety mechanisms, avoiding arms-race dynamics, and moving from broad principles to concrete governance proposals across academia, industry, and government.
Main Topics: Miles Brundage’s background and research focus: Brundage explains his move from energy policy to AI policy because it is neglected and potentially high impact. His work spans methods for rigorous AI forecasting, uncertainty about future AI trajectories, openness, and misuse by bad actors. AI security and misuse by bad actors: The discussion explores how AI could be used for cybersecurity attacks, fake news, autonomous drones, surveillance, and oppression. Brundage stresses that risks differ by domain and actor type, including states, corporations, terrorists, and rogue individuals. Coordination and incentives in AI safety: A major theme is the need for mechanisms that make it in actors’ interests to follow best practices. Brundage worries that competitive pressure between firms or countries could cause them to cut safety corners to gain performance or strategic advantage. Government, regulation, and international coordination: Brundage discusses the role of government experts, national security framing, and cautious international coordination. He is supportive of government involvement but warns that poorly designed action could intensify arms-race dynamics. State of AI policy progress: He frames current AI policy as being at the principle-setting stage, similar to early AI safety before concrete research agendas emerged. Existing work includes Asilomar principles and policy criteria, but actionable models and case studies are still missing. Career paths into AI policy: The conversation gives practical advice on careers in academia, industry labs like DeepMind/OpenAI, think tanks, government, and fellowship programs. He recommends building complementary technical and policy skills, networking, and working in adjacent fields when needed.
Key Arguments: AI policy is neglected relative to its potential impact, making it a promising area for research and career investment. Understanding plausible near-term misuse matters even when precise forecasting is impossible; the key is mapping what AI systems could enable. Different bad actors matter in different contexts: states, corporations, terrorists, authoritarian regimes, and individuals may each pose distinct risks. AI capabilities will likely diffuse, raising the floor of what individuals can do while also empowering large institutions with more resources. The central governance challenge is not only inventing safety methods but incentivizing actors to adopt them under competitive pressure. Safety-performance trade-offs may create arms-race dynamics, so governance should align safety with self-interest. A plausible policy approach is conditional access: developers gain access to cutting-edge models or compute only if they submit to monitoring and safety practices. Government should build internal AI expertise and be prepared for crises, but it should avoid unintentionally accelerating AI competition. International coordination may be possible among a small number of major actors because AI capability is concentrated in a few organizations. AI policy has made meaningful progress in shared principles, but it still lacks concrete, actionable proposals and mature models. Career entry does not require a single fixed path; useful contributions can come from synthesis, networking, technical work, policy work, or adjacent roles. People should strengthen weak areas rather than relying only on their existing background, because the field is interdisciplinary.
Data Points: AI policy growth period: past few years - Brundage says AI policy has become a more neglected but high-impact area over the past few years. AI safety workshop participants: some participants from major AI groups - He says a recent workshop on bad actors and AI included participants from relevant organizations. DARPA cyber grand challenge: recently - Used as an example of automated hacking and defense research already underway. Asilomar AI principles conference: 2 years later this year - He describes the principles as emerging two years after an earlier beneficial AI conference and open letter. Open letter signatories: thousands of people - Thousands, including many AI scientists, signed the open letter following the Puerto Rico beneficial AI conference. AI conference examples: NIPS, ICML, IJCCHI/ICJCHI, AAAI, ICLR - He lists major conferences as networking venues for AI and policy-minded attendees. AI lab hiring growth: a fair number of advertisements for postdocs - He notes growth in AI safety jobs at top labs compared with a few years earlier. Organizations expanding in the field: many did not exist five years ago; some not one year ago - He emphasizes how quickly the AI policy ecosystem is changing.
Pivotal Quotes: "it seems like it was more of a neglected area and could have a very large impact" — Miles Brundage: On why he moved from energy policy into AI policy "there's been less attention paid to how do you actually incentivize people to act on the best practices and the good theoretical frameworks that are developed" — Miles Brundage: Explaining the core governance gap in AI safety and policy "we don't yet have anything super actionable" — Miles Brundage: On the current state of AI policy progress beyond high-level principles
Implications: For listeners and practitioners, the message is to treat AI policy as an emerging field where interdisciplinary talent is valuable now. The biggest opportunities lie in turning safety principles into enforceable, incentive-compatible governance before competitive pressures harden into dangerous norms.