The System Behind Self-Driving: Waymo’s Dmitri Dolgov

Episode Summary

Executive Summary: Waymo co-CEO Dmitry Dolgov explains how Waymo evolved from research into scaled deployment, now serving hundreds of thousands of autonomous rides weekly. He argues that full autonomy requires a modular AI stack: a foundation model, specialized teacher models, simulation, critics, and intermediate representations—not just end-to-end pixels-to-trajectories. The discussion also covers sensors, hardware generations, city expansion, operations, and the future of autonomous mobility.

Main Topics: Waymo’s architecture: sensors, encoders, decoders, and onboard inference (Priority: 5/5): Dolgov describes the vehicle as a 360-degree sensing system using cameras, LiDAR, and radar feeding local AI models that build a world model, plan actions, and actuate the car in real time. Cloud compute is used for non-real-time tasks. Why full autonomy is not just end-to-end learning (Priority: 5/5): He rejects a simplistic pixels-in/trajectory-out framing for fully autonomous driving at scale, arguing that Waymo needs intermediate representations, structured priors, and multiple specialized models to achieve safety and robustness. Foundation model, simulator, and critic as a teacher-student system (Priority: 5/5): Waymo uses a large offboard foundation model and then specializes it into three teachers: the driver, simulator, and critic. These are distilled into smaller deployable models for the car and training/evaluation workflows. Safety, edge cases, and human behavior in the driving product (Priority: 4/5): The conversation explores how Waymo optimizes not just for destination arrival, but for safety, smoothness, predictability, and social behavior, including nuanced pickup/drop-off interactions and long-tail freeway incidents. Scaling from research to deployment across cities and climates (Priority: 5/5): Dolgov says Waymo has moved beyond core R&D into global scaling. Expansion depends on operating-domain specialization rather than just city-by-city mapping, with hard cases like cold weather requiring hardware and software changes. Hardware evolution and the move to purpose-built vehicles (Priority: 4/5): The discussion contrasts retrofitted cars with Waymo’s sixth-generation custom vehicle and sensor stack, which is simpler, cheaper, more capable, and designed around the passenger experience rather than the driver. Operations, logistics, and the economics of ride-hailing (Priority: 4/5): Waymo’s depot operations, cleaning, charging, and fleet management are becoming more automated. Dolgov also notes future possibilities like personal ownership of Waymo-equipped vehicles and reduced parking demand.

Key Arguments: Waymo’s real-time driving inference runs locally in the car; the cloud is used only for non-real-time tasks like cleaning detection or lost-item recovery. A purely end-to-end model can work in nominal cases, but it is not sufficient for the long tail of edge cases or for achieving superhuman safety at scale. A shared foundation model can support the driver, simulator, and critic because all require understanding object relations and future behavior in the physical world. Intermediate structured representations are not a limitation; they are necessary to make simulation, safety validation, and reward design tractable. Waymo’s progress came from a series of enabling breakthroughs in AI, compute, data, evaluation, and training recipes—not a single magic architecture. Driver-assist systems may be able to rely on simpler end-to-end approaches, but full autonomy demands a much more elaborate system. The biggest technical leaps came when Waymo made AI the backbone of Gen 5 and then refined the system into Gen 6 with lower-cost, more capable hardware. Generalization across cities, vehicle platforms, and some weather conditions is improving, but cold weather remains especially hard because it affects sensors, hardware, and control. Autonomous driving improves traffic flow by reducing human unpredictability, but the larger societal effects may be even bigger through reduced parking needs and better land use. The core technology is now good enough to support global scaling, but deployment still requires extensive validation, specialization, and operational maturity.

Data Points: Fully autonomous rides per week: nearly 500,000 / over 500,000 - Waymo’s current scale across multiple cities Ride volume per year: 25 million rides a year - Waymo’s Jaguar I-Pace fleet discussed in the context of scaling Initial commercial deployment: 2020 - Waymo began offering fully autonomous commercial service in Chandler, Arizona Waymo driver generations mentioned: Gen 4, Gen 5, Gen 6 - Framework for describing major platform shifts Google self-driving project start year for Dolgov: 2009 - He joined as one of the first engineers Dolgov’s return to Russia: 1994 - He returned for undergraduate and master’s study International expansion plans: London and Tokyo this year - Planned new operating geographies Hardware cost reduction: fraction of the cost - Gen 6 sensor/hardware stack compared with previous systems and comparable driver-assist hardware Sensor modalities: 3 primary modalities - Cameras, LiDAR/lasers, and radar are the core sensing stack Coverage: 360-degree coverage - Waymo driver sees around the vehicle continuously Model structure: 3 offboard teachers - Driver, simulator, and critic distilled from the foundation model

Pivotal Quotes: "I would say that we've clearly moved past the stage of scientific research and deep core technology development to this new phase of accelerated global scaling and deployment." — Dmitry Dolgov: Describing Waymo’s transition from R&D to large-scale commercial deployment "I can imagine building a big model that understands how the physical world works and understands the important properties of what it means to drive, the social aspects of driving, and what it means to be a good driver as opposed to a bad one." — Dmitry Dolgov: Explaining the role of a foundational world model in autonomy "If you think about the hard parts of driving, it's not unlike having a conversation." — Dmitry Dolgov: Describing the social, interactive nature of driving behavior

Implications: Waymo’s approach suggests full autonomy is becoming a systems-engineering and scaling problem, not a single-model breakthrough. Expect broader deployment, better passenger-focused vehicles, and pressure on cities, parking, and ride-hailing economics.