Are We Too Obsessed With AI Predictions? — With Carissa Véliz

Episode Summary

Executive Summary: Oxford philosopher Carissa Valiz argues that prediction has become overused and underexamined across AI, hiring, lending, justice, surveillance, and prediction markets. She says predictive systems can create self-fulfilling prophecies, reduce accountability, and erode fairness and democracy, while urging more contestable, truth-tracking design and greater skepticism about where prediction is appropriate.

Main Topics: Prediction as a form of power (Priority: 5/5): Valiz frames prediction not as neutral forecasting but as a tool that can steer outcomes, create self-fulfilling prophecies, and shape institutions in ways users often fail to notice. Algorithmic hiring and labor-market filtering (Priority: 5/5): The conversation examines AI resume screening and automated personality/emotion assessment, with concern that these systems exclude qualified but nonconforming candidates and incentivize intrusive workarounds. Predictive lending and fairness (Priority: 5/5): They debate whether mortgage/loan scoring improves banking efficiency; Valiz argues that predictive black boxes can embed bias, lack causal grounding, and make rejected applicants unable to contest decisions. Surveillance, justice, and democratic risk (Priority: 4/5): Valiz links surveillance infrastructure to predictive policing and justice-system scoring, warning that broad monitoring undermines anonymity, accountability, and liberal democracy. Generative AI as prediction and ‘bullshit’ (Priority: 4/5): She argues generative AI is fundamentally predictive and sycophantic rather than truth-tracking, though the host pushes back that tool use and grounding improve usefulness and economic value. Prediction markets and manipulation (Priority: 4/5): The discussion shifts to how prediction markets can be used for signaling power, shaping public perception, or exploiting insider information, not just aggregating wisdom. Humor, art, and preserving human openness (Priority: 3/5): Valiz closes by defending humor and the analog world as antidotes to prediction-heavy culture, arguing that comedy and surprise help keep societies imaginative and democratic.

Key Arguments: Prediction is not merely descriptive; it can actively shape reality by affecting hiring, loans, and legal outcomes, creating self-fulfilling prophecies. In fairness-sensitive domains like justice and lending, predictive systems are problematic because they are hard to contest and can hide discrimination behind statistical confidence. Even when predictions are accurate, they may be socially harmful if the act of prediction changes behavior or opportunities for the people being predicted about. Automated hiring and personality/emotion screening can exclude talented candidates who are quirky, introverted, or unable/unwilling to navigate algorithmic gatekeeping. Loan decisions based on black-box correlations are less rational than decisions based on clear, causal, contestable criteria that tell applicants how to improve. Surveillance is not a neutral tradeoff for safety; it can erode anonymity, protest rights, and democratic freedom, and the safest societies are not necessarily the most surveilled. Generative AI should be understood as part of the same predictive logic as other machine learning systems, and its tendency to flatter users makes it closer to ‘bullshit’ than truth-seeking. Prediction markets may reflect wisdom, but they can also be instruments of influence, manipulation, and conflict escalation rather than pure information aggregation. Humor and the analog world are important cultural defenses against an over-predictive, over-optimized society because they preserve surprise, creativity, and democratic dissent.

Data Points: Mortgage approval threshold example: $10,000 - Valiz uses a clear eligibility rule to contrast verifiable criteria with predictive black-box decisions. Loan scoring example: Green / yellow / red categories - Host describes a machine-learning mortgage system that classifies applicants by repayment likelihood. Prediction accuracy claim: 99.9% accurate - Valiz says companies may tout extremely high accuracy while still creating self-fulfilling outcomes. Loan lawsuit coverage rule: 51% chance of succeeding - Valiz cites an insurance-related example where probabilistic logic determines whether a lawsuit is covered. Prediction market profit example: $900,000 - She cites a case where parties allegedly pressured a journalist over a report because of a large bet. Anonymous prediction-market winnings: $1.2 million - Valiz references six anonymous accounts that profited betting on an attack on Iran. Training time horizon: 1,000 years - She argues predictions far into the future should be treated skeptically; the closer to the present, the more reliable the forecast. Workforce scale example: 1,300 stores - Mentioned in the sponsor/introduction about Ulta Beauty deploying AI across many retail locations. Enterprise adoption scale: Nearly half of the Fortune 500 - Scribe Optimize is described as being used by over 80,000 enterprises including nearly half of Fortune 500 firms.

Pivotal Quotes: ""The future isn't written."" — Carissa Valiz: Core thesis against treating prediction as destiny or a discoverable script. ""Self-fulfilling prophecies are like the perfect crime because it's like a murder weapon that disappears upon striking."" — Carissa Valiz: Her explanation of why predictive systems can create harms that leave no clear error signal. ""Predictions are not facts. They can be defied."" — Carissa Valiz: Closing point on why listeners should treat forecasts as contestable, not authoritative.

Implications: Listeners should treat AI and other forecasts as tools with built-in power, bias, and feedback effects—not neutral facts. Builders should prefer transparent, contestable, truth-tracking systems, especially in hiring, lending, justice, and public policy.