Episode Summary
Executive Summary: This Code Conference panel examines AI beyond hype, arguing that current systems already cause harm while future frontier models may pose deeper safety risks. Casey Newton, Helen Toner, and Ajay Akotiah discuss facial recognition abuses, uncertainty about emergent capabilities, responsible scaling policies, OpenAI governance, open vs. closed release, privacy, and chatbot design, urging rigorous testing and cautious deployment rather than blind acceleration.
Main Topics: Current AI harms already affecting people (Priority: 5/5): The panel begins by emphasizing real-world damage from AI today, especially facial recognition errors and discriminatory policing uses that can wrongly target innocent people. Unpredictability of frontier model capabilities (Priority: 5/5): The speakers explain that scaling language models improves benchmark scores, but does not reliably predict real-world abilities like hacking, deception, or tool use. Responsible scaling and safety testing (Priority: 5/5): They advocate for responsible scaling policies: forecast capabilities, test dangerous behaviors during training, and pause deployment until protections match actual risk. Governance, regulation, and company responsibility (Priority: 4/5): Helen Toner discusses OpenAI’s unusual nonprofit governance and the role of boards, management, and possibly government oversight in deciding when AI is too risky to deploy. Open source vs. closed source release (Priority: 4/5): The panel weighs the benefits of open access against the risks of distributing powerful models, proposing differentiated release policies based on capability and danger. Privacy and user experience in AI products (Priority: 3/5): The Q&A covers data privacy, privacy-preserving ML, and the risks of optimizing chatbots for engagement or friendliness in ways that may manipulate users.
Key Arguments: AI safety concerns are not only future-oriented; current systems already harm people through misidentification and biased enforcement uses, especially in facial recognition. For large language models, performance gains on narrow benchmarks do not tell us what dangerous real-world capabilities will emerge. Because frontier models can surprise even their creators, companies should test models for harmful behaviors before deployment rather than release first and learn later. Responsible scaling policies offer a middle ground between reckless acceleration and an indefinite pause by linking deployment to demonstrated capability thresholds and safeguards. Boards and management should treat AI governance as a real mission-level duty, not a formality, because product and investment choices can affect whether the company’s goals align with human benefit. Open-source access is not universally good or bad; safer models can be broadly shared, while highly capable models may require stricter controls or even non-release. AI product design should consider not just what increases engagement, but whether those interactions are actually beneficial or manipulative for users.
Data Points: Time until next-generation risks: 6 to 18 months - Toner says some serious experts think the next generation of AI could introduce major risks within this timeframe. Longer-term risk horizon: 3 to 5 years - Toner notes that the generation after the next could present even more serious concerns within this window. Facial recognition error/harm example: 8 months pregnant - A woman was held in a cell while having contractions after a facial recognition match wrongly implicated her. Board authority: Majority of board members can vote to shut down the company - Newton references OpenAI’s governance structure while discussing the seriousness of board oversight. Training approach for privacy: Federated learning - Discussed as a way to train systems like predictive text on-device without sending all private data to a central database. Privacy guarantee approach: Differential privacy - Mentioned as a method that provides formal guarantees about how much private data is released.
Pivotal Quotes: "we don't know how to tell in advance if they're safe is sort of the mind-blowing piece here" — Casey Newton: Newton frames the core AI safety challenge as uncertainty about emergent capabilities in large language models. "I think you don't just say, let's just give it to the consumers and see what they do with it" — Helen Toner: Toner argues against unchecked release of potentially dangerous frontier systems. "you get what you measure for" — Helen Toner: Toner warns that optimizing models for engagement or benchmark performance can produce unintended and potentially harmful outcomes.
Implications: Listeners should expect AI adoption to bring both benefits and real risks. Companies need stronger testing, governance, and release discipline; regulators may need to step in for frontier systems; and users should be cautious about engagement-driven, privacy-invasive AI products.
About Pivot
With great power, comes great scrutiny. Every Tuesday and Friday, journalist Kara Swisher and NYU Professor Scott Galloway offer sharp, unfiltered insights into the biggest stories in tech, business, and politics. They make bold predictions, pick winners and losers, and bicker and banter like no one else. From New York Magazine and the Vox Media Podcast Network.