Algorithms, Algorithmic Discrimination, and Autonomous Vehicles with Caleb Watney

Episode Summary

Executive Summary: Caleb Watney argues that AI and algorithms are most concerning when used by government, where transparency and feedback loops are weaker than in markets. He uses criminal justice and self-driving cars to show that algorithms should be judged against human performance, not perfection, and that open, accountable systems can improve outcomes when paired with human oversight.

Main Topics: Algorithmic bias and Kathy O’Neill’s critique (Priority: 5/5): The discussion begins with O’Neill’s concern that algorithms can hide or amplify bias, especially when people assume machine outputs are neutral. Watney agrees bias is possible but says the bigger issue is where algorithms are deployed and how incentives shape their use. Criminal justice risk assessment (Priority: 5/5): The transcript focuses on sentencing, bail, and recidivism tools such as COMPAS. The key question is whether algorithmic assessments are fairer than human judgment and how to handle proprietary models in life-altering decisions. Public vs. private sector incentives (Priority: 5/5): Watney distinguishes market use from government use of algorithms. Private firms face competition and can be punished for errors, while government systems often lack corrective feedback and therefore need stronger procurement safeguards. Transparency, open sourcing, and due process (Priority: 4/5): A major policy proposal is to make government-used algorithms open source or otherwise publicly auditable so defendants, judges, and researchers can inspect assumptions, data, and outputs. Algorithmic fairness definitions and statistical tradeoffs (Priority: 4/5): The conversation notes that different fairness metrics can conflict. Watney argues that debates over bias often depend on which definition of fairness is chosen and that some disparities can arise from base-rate differences rather than discrimination. Self-driving cars and the human-vs-machine standard (Priority: 4/5): The interview expands to autonomous vehicles to show how algorithms can outperform humans in safety-critical settings. Watney argues the relevant benchmark is flawed human driving, not perfection, and that AI can reduce accidents if incentives align. AI as partnership rather than replacement (Priority: 4/5): The closing theme is that near-term AI is more likely to augment humans than replace them. Judges, truckers, and drivers will increasingly work with algorithms rather than be fully displaced by them.

Key Arguments: Algorithms can encode bias, but the presence of bias is not unique to machines; human systems already contain hidden prejudices that algorithms can help surface. Government use of algorithms is more concerning than private use because public systems often lack market competition and strong feedback loops that would force correction of errors. Proprietary algorithms in criminal justice create due-process problems because defendants cannot challenge the models affecting their liberty. Open sourcing government algorithms, including underlying data sets and methods, would allow outside scrutiny, replication, and faster detection of errors or bias. In criminal justice, algorithms may improve outcomes by making implicit assumptions explicit and by helping judges make more accurate risk assessments. New Jersey’s bail reform is presented as evidence that risk tools can reduce jail populations without increasing crime. Fairness is technically complex: optimizing one fairness metric may worsen another, and some disparities can emerge from differing base rates across groups. Self-driving cars should be judged relative to human driving, which causes the vast majority of accidents; perfection is not the standard. Private firms have strong incentives to avoid catastrophic AI failures because consumers can switch competitors and reputational damage is severe. The likely future is hybrid systems in which AI assists humans in domains like driving and justice rather than fully replacing them.

Data Points: Jail population decline in New Jersey: 20% - Watney cites New Jersey’s 2017 bail reform and risk assessment use as having reduced jail population without increasing crime. Year of Loomis case: 2016 - The Eric Loomis sentencing and appeal controversy is used as a key example of proprietary risk scoring in criminal justice. Criminal sentence in Loomis case: 5 years - Loomis was sentenced in part due to a private risk assessment algorithm. Road fatalities in the U.S.: 37,000 - Watney compares human driving to autonomous vehicles by citing U.S. road deaths in 2016. Share of accidents caused by human error: 94% - He attributes the vast majority of crashes to human error, arguing that AI drivers only need to outperform humans. Average risk scale: 1 to 10 - COMPAS was described as using a one-to-ten risk scale, with debates about how a given score should be interpreted across demographic groups. Autonomy levels: 0 through 5 - The discussion references the SAE autonomy taxonomy for self-driving cars. Deployment area: Phoenix area - Waymo’s level-four driverless cars were described as operating in Phoenix as an early deployment region. Testing mode: Level 4 - Watney distinguishes level-four autonomy (geofenced full autonomy) from level five (anywhere autonomy). Driverless car fatality count: 1 - He notes only one fatality so far in a semi-autonomous Tesla crash, as of the transcript’s timeframe.

Pivotal Quotes: "the relevant standard is what's happening right now, what's happening with humans" — Caleb Watney: He explains that algorithms should be evaluated against current human performance, not an idealized perfect system. "we want to create sort of institutional constraints that make sure that, regardless of kind of what the state-of-the-art science is right now, that they're being applied in responsible ways" — Caleb Watney: He outlines his policy view that procurement and transparency matter more than freezing a specific technical fairness rule. "it's less like the movie Terminator and more like the movie Her" — Caleb Watney: Watney summarizes his view that AI will mainly be a collaborative human tool rather than an apocalyptic replacement force.

Implications: Listeners should expect AI to be most beneficial when it is transparent, auditable, and paired with human judgment. The strongest policy concern is government use, where open procurement and public scrutiny can reduce harm without slowing beneficial innovation.