30 | Derek Leben on Ethics for Robots and Artificial Intelligences

Episode Summary

Executive Summary: Sean Carroll and philosopher Derek Lieben explore how to program ethics into autonomous systems, especially self-driving cars. They compare utilitarianism, Kantian and contractarian/Rawlsian approaches, arguing that machines will inevitably face hard tradeoffs and should use a principled framework—Lieben favors maximizing the welfare of the worst-off among shared human primary goods rather than simply adding up happiness.

Main Topics: Why robots need ethics (Priority: 5/5): Autonomous systems in transport, medicine, and warfare will make decisions affecting human well-being without close supervision, so designers must specify which actions are permissible and impermissible. Limits of Asimov-style rules (Priority: 4/5): Asimov’s laws are treated as too vague and too blunt for real-world dilemmas because they cannot handle property, dignity, indirect harms, or cases where every option causes harm. Trolley problems and moral intuition (Priority: 5/5): The conversation uses trolley-style scenarios to show how people distinguish between doing and allowing harm, and how intuitions change when the victims are personalized or related to the decision-maker. Moral theory as formal decision-making (Priority: 5/5): Lieben argues that ethics for machines should be explicit and quantitative, drawing on utilitarianism, deontology, game theory, and Rawlsian contractarianism rather than vague virtue language. Contractarianism and Rawls’s Maximin principle (Priority: 5/5): Lieben advocates a Rawls-inspired framework using primary goods and a maximin rule to prioritize the worst-off, aiming to identify actions that promote cooperation among self-interested agents. Game theory, cooperation, and Pareto improvements (Priority: 4/5): Nash equilibrium, prisoner’s dilemma, and Pareto optimality are presented as useful tools for defining cooperation problems and comparing moral theories by their ability to sustain mutual benefit. Real-world deployment and wartime robotics (Priority: 4/5): The discussion closes with practical concerns about how such ethical frameworks could be implemented in cars, security robots, and military systems, and whether autonomy in war should be limited or rejected.

Key Arguments: Autonomous technologies inevitably make choices that affect human health, safety, and opportunity; programmers must encode ethical rules rather than avoid ethics altogether. Asimov’s laws are too vague to guide real machines because they do not resolve indirect harms, property violations, or moral dilemmas where harm cannot be avoided. Moral intuitions are often inconsistent; trolley problems expose tensions between abstract judgments and concrete emotional responses. A machine ethics system should be formal, specific, and quantitative so it can compare outcomes rather than rely on slogans like 'be excellent to each other.' Moral theories should be evaluated by their role in promoting cooperation among self-interested organisms, not by whether they perfectly match all intuitions. Lieben favors a Rawlsian contractarian approach because it uses shared human primary goods—life, health, opportunity, and essential resources—rather than happiness alone. The Maximin principle is preferable to simple utilitarian summation because it protects the worst-off and better captures fair social and machine decision-making. In autonomous systems, irrelevant identity markers such as race, religion, doctor status, or gender should not affect collision decisions; only factors tied to expected harm, such as size, age, or helmet use, may matter. Deep-learning systems may not need to explain their choices in philosophical language, but they should still reliably approximate the right moral rule. Military autonomy is especially problematic because systems would need far more sophistication to make proportional and lawful decisions about force.

Data Points: PhilPapers survey: normative ethics—deontology: 25% accept or lean - Sean cites the philosophy survey to show lack of consensus among professional philosophers. PhilPapers survey: normative ethics—consequentialism: 23% and some change accept or lean - Used to illustrate that consequentialism is only one among several major camps. PhilPapers survey: normative ethics—virtue ethics: 18% accept or lean - Referenced as part of the diversity of views in normative ethics. PhilPapers survey: normative ethics—other: 32.3% - Shows substantial philosophical disagreement beyond the named theories. PhilPapers survey: moral realism: 56.4% accept or lean towards realism - Used to contrast realism with anti-realism in metaethics. PhilPapers survey: moral anti-realism: 27.7% accept or lean - Illustrates that a sizable minority rejects moral realism. PhilPapers survey: other on realism: 15% - Remaining respondents fell outside the realism/anti-realism categories. Prisoner's dilemma sentence outcome: Both confessing yields a medium sentence; both staying quiet yields a low sentence - Explained as the classic cooperation problem where Nash equilibrium differs from Pareto-optimal cooperation. Trolley problem harm tradeoff: 1 person vs 5 people - Canonical example used to contrast utilitarian and deontological intuitions. Broken-leg example: 50 single broken legs vs 1 person with 2 broken legs - Lieben’s counterintuitive example showing where maximin can conflict with intuition.

Pivotal Quotes: "“the robots are going to be doing something and we have to decide what we want it is for them to do”" — Sean Carroll: Introduces the central premise that human designers must choose the ethical rules autonomous systems follow. "“we should prefer the distribution that makes the worst-off person as best-off as possible”" — Derek Lieben: Core statement of Lieben’s Rawlsian/maximin contractarian principle. "“all collisions are bad and we want to avoid them all equally”" — Sean Carroll: Critique of a simplistic industry stance on self-driving car ethics.

Implications: AI and autonomous vehicles cannot be ethically neutral; they need explicit decision rules. Lieben’s view pushes industry toward transparent, data-driven moral design, while raising hard questions about bias, implementability, and whether military autonomy should exist at all.