Mac Schwager: How engineers are putting the ‘auto’ in autonomous

Episode Summary

Executive Summary: Stanford professor Mack Schwager explains that autonomous vehicles and drones must be designed as multi-robot systems that coordinate with humans and each other, not as isolated machines. He argues safety, human-robot interaction, uncertainty about others’ intentions, and robust AI are the core challenges. Deep learning is essential but brittle, so his lab combines optimization, game theory, and out-of-distribution detection to make autonomy safer and more useful.

Main Topics: Autonomous vehicles must coexist with humans (Priority: 5/5): Schwager argues it is unrealistic to build separate road networks for autonomous and human drivers, so robots must learn to operate safely in mixed traffic and adapt to human behavior. Robots as co-robots and multi-robot coordination (Priority: 5/5): He frames robots as inherently social systems that must cooperate with other robots and humans, making interaction, communication, and coordination central research problems. Optimization and game theory as the core framework (Priority: 4/5): His research uses mathematical optimization when robots share goals and game theory when agents do not communicate, allowing each robot to model others’ actions and intentions. Inverse reinforcement learning and theory of mind (Priority: 4/5): When robot goals are unknown, his lab infers objectives from observed behavior, aiming to give robots a human-like ability to model others’ intentions and react accordingly. Deep learning in autonomy: powerful but unsafe at the edges (Priority: 5/5): Deep learning enables perception and decision-making in autonomous systems, but it can fail unpredictably in rare, unusual situations, creating major safety concerns. Drones for large-scale monitoring and coordination (Priority: 4/5): Schwager describes drones as especially promising for wildfire detection, ecological monitoring, penguin counting, agriculture, and animal shepherding, with coordination as the key technical challenge.

Key Arguments: Autonomous vehicles will not likely have dedicated infrastructure; they must safely co-drive with humans on the same roads. Communication between autonomous vehicles can simplify coordination, but many real-world settings require reasoning without explicit communication. Human drivers rely on motion, timing, and social cues; autonomous systems must infer intention from behavior and eventually from facial and attentional cues. Mathematical optimization is useful when robots share a common objective, while game theory is more appropriate when agents have separate goals and incomplete information. Inverse reinforcement learning helps robots infer hidden objectives by observing actions that appear optimal. Robots need a form of theory of mind so they can anticipate others’ goals and reactions, similar to human empathy. Deep learning is central to modern autonomy, but its practical power has advanced faster than the theory for guaranteeing safety. A major safety challenge is out-of-distribution detection: systems must recognize when they are encountering unfamiliar scenarios. Multi-robot systems can create emergent behaviors such as coordinated lifting, distributed wildfire monitoring, and drone-based animal herding. Drones offer major societal benefits in surveillance, disaster response, ecology, and agriculture, but require robust coordination and control at scale.

Data Points: Autonomous-car accident rate per mile: Twice as high as human-driven cars - Schwager cites this as the current state over several years of autonomous-car deployment, while noting improvement over time. Legal fault rate: Lower than for human drivers - He notes that despite higher accident rates, autonomous cars are found legally at fault less often than human drivers. Number of neural networks in an ensemble: 5 - He describes using multiple neural networks that vote to detect out-of-distribution situations. Scale of imagined wildfire-monitoring fleet: 10,000 drones - He uses this as an example of a future large-scale coordinated drone system for forest-fire monitoring.

Pivotal Quotes: "All robots are co-robots." — Mack Schwager: Opening framing of the interview, emphasizing that robots should be designed for interaction with other robots and humans. "How should that robot interact with other humans and robots around it." — Mack Schwager: He explains why designing robots in isolation misses the central challenge of real-world autonomy. "The practical power has accelerated way beyond the theoretical framework we have for understanding when deep networks go wrong." — Mack Schwager: He summarizes the safety gap between deep learning performance and reliable guarantees in autonomous systems.

Implications: Autonomy’s next phase depends on making robots socially aware, safe in mixed human environments, and robust to rare edge cases. The biggest opportunities are coordinated mobility and large-scale drone applications, but trust will hinge on safety and human-compatible behavior.