41. The Folly of Prediction

Episode Summary

Executive Summary: This episode argues that prediction is deeply incentivized yet often unreliable, especially in politics, finance, and media punditry. Using Romanian witches, economists, experts, football pundits, and crop forecasters, it shows that bold predictions are rewarded while failed ones are forgotten. The best forecasters are less dogmatic, more self-critical, and more willing to update beliefs—or use markets and systems that price accountability.

Main Topics: Prediction incentives and accountability (Priority: 5/5): The episode opens with Romanian witches facing possible regulation and even jail for false predictions, contrasting that with the lack of consequences for politicians, pundits, and other forecasters who are often wrong. Expert forecasting is systematically weak (Priority: 5/5): Philip Tetlock’s long-running research found that highly educated political experts were only slightly better than chance and often less accurate than simple baseline methods, largely because they were overconfident. Dogmatism vs. self-criticism (Priority: 5/5): Poor predictors cling to ideology and explanations that confirm prior beliefs, while better predictors show cognitive flexibility, revise views when evidence changes, and can say what would change their minds. Media rewards boldness over accuracy (Priority: 4/5): The episode argues that television and opinion culture favor hedgehog-like certainty, extreme calls, and memorable hits, while ignoring the many missed predictions that never get tracked. Prediction in markets and sports (Priority: 4/5): Wall Street forecasts and NFL picks are shown to be only marginally better than simple heuristics, with extreme correct calls often correlated with worse overall forecasting ability. Where prediction works better (Priority: 4/5): Agricultural forecasting and recommendation systems like Pandora can do relatively well because they rely on repeated measurement, structured data, and narrower goals, though they still face randomness and limits. Prediction markets as a possible fix (Priority: 5/5): Robin Hanson argues that prediction markets align incentives by rewarding accuracy and discouraging uninformed opinions, potentially improving forecasts of political and geopolitical events.

Key Arguments: Bold, memorable predictions are disproportionately rewarded, while inaccurate predictions are quickly forgotten, creating a prediction industry with weak accountability. Tetlock’s research showed that experts predicted political outcomes only slightly better than random or simple no-change baselines, and often worse than simple extrapolation. The best predictors tend to be foxes rather than hedgehogs: flexible, pragmatic, and willing to integrate conflicting evidence rather than defend a single theory. Correctly predicting one extreme event does not imply strong overall forecasting ability; in fact, it can correlate with lower general accuracy. Many media pundits prefer vague language like 'could,' which makes predictions hard to audit and protects them from being clearly wrong. Prediction markets may improve forecasting because they create direct financial incentives to speak only when one has information and to be penalized for errors. Agricultural forecasting is more disciplined than political forecasting because it uses large samples, repeated measurement, and physical indicators, though weather still introduces major uncertainty. A central lesson is that people often overestimate how predictable complex systems are and underestimate the role of randomness and timing.

Data Points: Romanian witch jail penalty proposal: 6 months to 3 years - Proposed punishment in Romania for witches whose predictions repeatedly fail Witch regulation law: Passed one chamber of parliament before stalling - Earlier Romanian proposal to regulate and tax witches Tetlock study participants: Close to 300 experts - Political forecasting study participants Tetlock study time span: 20 years - Longitudinal tracking of expert predictions Tetlock prediction count: About 80,000 predictions - Total predictions analyzed in Expert Political Judgment research Wall Street Journal survey participants: About 50 top economists every six months - Survey used by Christina Fang and Yerker Denrell NFL expert accuracy: 36% - Experts’ division and wild-card picks compared with simpler baselines Casual/untrained baseline NFL accuracy: 33% - Accuracy if excluding the worst team in each division Random four-team division pick chance: 25% - If selecting one team out of four at random Pandora song attributes: Up to 480 attributes per song - Music Genome Project breakdown of tracks Pandora library size: More than 1 million songs - Scale of the recommendation database Average American music listening: 17 hours per week - Used to justify Pandora’s cultural relevance USDA farmer survey sample: About 85,000 - Number of farmers/ranchers surveyed in early March USDA cornfield enumeration sample: Roughly 1,900 cornfields in 10 states - Field measurements used for crop forecasting USDA forecast precision claim: Within 5%, often 2-3% - Joe Prusaki’s description of typical monthly forecast accuracy Corn price reaction: 9% spike, then another 6% - Market response to USDA lowering crop estimates Intrade euro bet: 15% chance - Bet on whether any euro-using country would drop the euro by year-end Intrade WMD terror attack bet: 28% chance - Bet on a successful WMD terrorist attack by end of 2013

Pivotal Quotes: "The experts think they know more than they do." — Philip Tetlock: Summary of Tetlock’s findings about overconfidence in expert political judgment "The fox knows many things, but the hedgehog knows one big thing." — Isaiah Berlin (quoted by Philip Tetlock): Used to explain why flexible forecasters outperform dogmatic ones "When there are big rewards to people who make predictions and get them right and there's zero punishment for people who make bad predictions because they're immediately forgotten, then accountants would predict that that's a recipe for getting people to make predictions all the time." — Steve Levitt: Explanation of why forecasting is abundant despite poor accountability

Implications: Listeners should treat confident forecasts skeptically, demand track records, and prefer accountable systems like prediction markets or data-driven methods. In complex domains, humility and updateability matter more than certainty.