555. New Technologies Always Scare Us. Is A.I. Any Different?

Episode Summary

Executive Summary: This episode examines whether AI will eventually match human intelligence, focusing on forecasting methods, job disruption, and the risks of powerful systems. Ajaya Cotra’s “biological anchors” framework suggests human-level AI may arrive in the late 2030s, while economists argue technology’s effects depend on institutions and goals. The episode warns that AI may be both economically disruptive and difficult to control.

Main Topics: Forecasting Human-Level AI (Priority: 5/5): Ajaya Cotra explains her method for estimating when AI could become as capable as the human brain, using computational scale and biological comparison as rough forecasting tools. Current AI Limits vs. Future Capability (Priority: 5/5): The episode contrasts today’s language models as mostly prediction engines with the possibility that scaling data, compute, and model size could produce much more capable systems. Automation, Jobs, and Labor Market Adjustment (Priority: 5/5): Economic history is used to show that automation can eliminate specific jobs while creating new ones over time, though not without real short-term pain for affected workers. Technology, Inequality, and Institutional Choices (Priority: 5/5): Simon Johnson argues technology is not inherently good or bad; its impact depends on who designs it, what it is optimized for, and the rules governing its use. The Alignment Problem (Priority: 5/5): The episode explores the risk that AI systems may not reliably pursue the goals humans assign them, creating safety, trust, and control problems even without sci-fi scenarios. Regulation and Gradualism (Priority: 4/5): Cotra suggests policymakers may need to regulate stepwise capability jumps, requiring oversight before companies train substantially more powerful models.

Key Arguments: Current AI systems are not yet comparable to the human brain, but scaling model size, data, and training time has historically produced predictable gains. Cotra’s biological-anchors framework uses compute, data, and architecture to estimate when training a brain-sized AI may become affordable. Past automation shows that specific jobs can disappear while workers and future generations move into other sectors; the labor market is dynamic, not a fixed number of jobs. The harms of automation are real for displaced workers, even if society eventually gains from productivity growth and new occupations. Technology does not determine outcomes by itself; institutions, incentives, and power relations shape whether it broadens prosperity or concentrates wealth. AI companies are incentivized to maximize capability quickly, which may conflict with safety and social welfare goals. Alignment is not just about killer robots; it also concerns everyday systems making wrong, opaque, or unsafe decisions. Because AI capabilities may arrive in large jumps, society may need regulatory mechanisms that slow or review major model-scale increases before deployment.

Data Points: Open Philanthropy AI forecasting specialization: 4 years - Ajaya Cotra has focused on AI forecasting and AI risk at Open Philanthropy for the last several years. Transformative AI median estimate (2020 report): 2050 - Cotra’s 2020 biological-anchors report estimated a 50% chance of transformational AI by 2050. Transformative AI updated median estimate: late 2030s - After GPT-4, Bard, and Claude, Cotra revised her median estimate earlier. AI investment in first six months of year: more than $40 billion - Venture capital flowed heavily into AI companies, highlighting the sector’s momentum. GPT-4 development cost: less than $1 billion - Used to illustrate how major capability jumps may occur from relatively modest spending compared with total capital available. Telephone operator share of female employment (1920): roughly 2% - Shows how significant the switchboard-operator occupation was for women in the U.S. AI impact on jobs (Goldman Sachs report): two-thirds of jobs impacted - A cited forecast describing broad labor disruption from AI. Jobs predicted impacted globally: 300 million - Goldman Sachs estimate of jobs that could be lost or diminished. IBM back-office roles at risk: 30% - CEO statement cited as an example of current corporate automation expectations. Dropbox workforce reduction: 16% - Company cited AI advances as a reason for cutting staff.

Pivotal Quotes: "When would it become affordable for a company to train an AI system as big as the human brain?" — Adam Davidson: Frames the episode’s central forecasting question. "Our view is that engineering is too important to be left to the engineers." — Simon Johnson: Explains why AI development should involve broader social and policy oversight. "The high-level description of the alignment problem is that... we have, at a technical level, [not] found a way to guarantee that... it will actually learn to pursue that goal." — Ajaya Cotra: Defines the core safety/control concern in advanced AI systems.

Implications: AI may bring major productivity gains, but it could also reshape jobs, power, and security faster than institutions can adapt. Listeners are left with a call for caution, broader governance, and active choices about what kind of AI to build.