#81 – Anca Dragan: Human-Robot Interaction and Reward Engineering

Episode Summary

Executive Summary: Anca Dragan argues that the future of robotics lies less in isolated autonomy and more in modeling humans as active partners: predicting intentions, learning preferences, probing for information, and adapting behavior in shared environments. She emphasizes that humans are not simply noisy or irrational, but often rational under different beliefs, constraints, or intuitive physics, and that robotics should account for this in planning, reward design, and human-robot coordination.

Main Topics: Origins in math, AI, and robotics (Priority: 5/5): Dragan describes a gradual path from childhood programming and math olympiad participation to computer science, AI, and the Carnegie Mellon Robotics Institute, where robotics became a practical outlet for optimization and algorithms. Expressive and emotionally resonant robots (Priority: 4/5): She discusses how robots like Google’s self-driving car and WALL-E inspired her by creating a sense of connection, and how motion, timing, and style can make robots feel expressive and legible to humans. Human-robot interaction as a modeling problem (Priority: 5/5): Her core research view is that robots must operate in environments with humans who have their own actions, preferences, and perceptions, making HRI fundamentally different from isolated robot tasks. Inference, prediction, and human preferences (Priority: 5/5): Dragan explains inverse reinforcement learning, Boltzmann rationality, and why these models work for some tasks but fail when humans are highly noisy, under-informed, or operating with different mental models. Information-gathering actions and game theory (Priority: 5/5): A robot should not merely observe humans passively; it can act to reveal information, probe human responses, and update beliefs, especially in coordination tasks like driving or shared manipulation. Reward design, leakage, and unintended consequences (Priority: 5/5): She argues that reward functions are hard to specify correctly, that humans communicate preferences indirectly through corrections, interventions, and the environment itself, and that robots should treat these signals as evidence rather than literal commands. Meaning, mortality, and human values (Priority: 3/5): The conversation closes with reflections on death, finitude, vulnerability, community, and the limits of human understanding, linking robotics to broader questions about purpose and what AI should help us learn.

Key Arguments: Robots should be designed for interaction, not just isolated task execution, because humans actively shape the environment and influence robot outcomes. Human behavior is often best understood as goal-directed under different assumptions, beliefs, or constraints, not simply as irrational noise. Inverse reinforcement learning and Boltzmann rationality are useful but incomplete because they can fail when humans use different intuitive models of the world. Robots can and should take actions that gather information about human preferences or driving styles rather than passively infer everything from observation. In driving, the human element is not a minor detail; other agents change behavior in response to the robot, making the problem strategic and game-theoretic. Reward functions are not “the answer”; they are uncertain, incomplete specifications that must be learned, refined, and treated as evidence of intent. Physical interventions, e-stops, and everyday environmental structure all leak preference information that robots can use to infer what people value. Semi-autonomous systems can be risky if they assume supervision is equivalent to active driving; the human’s state and engagement change when they become an observer. Simulation and end-to-end learning are useful, but they still require structured priors, world models, and priors about human behavior to generalize out of distribution.

Data Points: Degrees of freedom in a Wally robot: 7 - Dragan says her husband built an actuated WALL-E-like robot for his proposal, including the lens motion. School grade when she started programming: 4th grade - She says her first program was in QBasic in fourth grade, drawing a circle or square. School grade when she got into the Math Olympiad: 5th grade - She describes getting into math competition early, which shaped her path into STEM. Year of self-driving-car ride at RSS: 2014 - She recalls riding in Google’s self-driving car during RSS 2014 in Berkeley. Behavioral economics origin referenced: 1940s to late 1950s to 1970s - She cites von Neumann/Morgenstern, Luce/Shepard, and later behavioral economics as intellectual antecedents for modeling human choice. Driverless-cars deployment status referenced: A few companies in small areas - She notes that fully driverless systems have begun operating in limited geographies. Anecdotal reward-design horizon: 10 years - She says she has spent about a decade tuning cost functions and still finds reward specification difficult. Human-robot interaction framing: Underactuated - She uses underactuation as a metaphor: humans are not fully controllable degrees of freedom, but they are influenced by robot actions.

Pivotal Quotes: "“Maybe we can think of them as actually being relatively rational, but just under different assumptions about the world.”" — Anka Dragan: She reframes apparent human irrationality as mismatched beliefs or world models rather than pure chaos. "“The robot can act too. And so there’s these information gathering actions that the robot can take to sort of solicit responses that are actually informative.”" — Anka Dragan: She explains why robots should probe and interact to infer human intent, not only observe. "“Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.”" — Isaac Asimov: Closing quote read by the host, reinforcing the episode’s theme of revising assumptions.

Implications: HRI will increasingly shape autonomy, safety, and usability across cars, homes, and assistive robots. The field’s future depends on better models of humans, richer signals than rewards alone, and systems that can learn, adapt, and stay legible in real time.