233. How to Be Less Terrible at Predicting the Future

Episode Summary

Executive Summary: The episode argues that most experts are poor forecasters because they lack accountability and rely on vague language, while Philip Tetlock’s Good Judgment Project shows forecasting can be measurably improved through training, teamwork, humility, base-rate thinking, and constant feedback. It uses sports, intelligence failures, and political examples to show why numerical probabilities beat “could/might” language.

Main Topics: The problem with expert prediction (Priority: 5/5): The transcript opens with a fantasy-football example showing how pundits often choose consensus picks and miss upset outcomes, illustrating the broader claim that experts are frequently overconfident and unaccountable. Cam Newton and accountability (Priority: 5/5): Cam Newton’s critique of media prediction is used to argue that people without skin in the game can make bold claims without consequences, unlike forecasters whose performance is scored. Philip Tetlock’s research on expert judgment (Priority: 5/5): Tetlock’s long-running empirical work finds that many highly educated experts are not especially accurate and are often dogmatic, changing their minds too slowly in response to evidence. The Good Judgment Project and IARPA tournament (Priority: 5/5): Tetlock describes a large forecasting tournament run by IARPA, where tens of thousands of participants made probability forecasts on geopolitical questions and the Good Judgment Project outperformed expectations. Traits and methods of super forecasters (Priority: 5/5): Super forecasters are described as humble, numerate, open-minded, curious, hardworking, and willing to use outside views, base rates, and frequent updating rather than vague intuition. Why vague verbiage fails (Priority: 4/5): Examples like the Bay of Pigs and Iraq WMD intelligence show that terms such as 'fair chance' or 'could happen' are ambiguous and easily distorted; explicit probabilities communicate risk better. Implications for public debate and leadership (Priority: 4/5): Tetlock argues that forecasting tournaments and numerical accountability could improve intelligence analysis, political decision-making, and public discourse by making people more careful and less partisan.

Key Arguments: Experts are often poor forecasters because they are not held accountable for accuracy and can revise or evade past claims without cost. Probability estimates are more useful than vague language because 'could' or 'fair chance' can mean very different things to different audiences. Forecasting skill is not just innate; it can be improved through training, feedback, and structured practice. Working in teams and using experimental methods improved forecasting performance in the Good Judgment Project. Super forecasters rely on base rates and outside views before adjusting for specific details, reducing overconfidence. Humility and open-mindedness are critical because forecasters must update beliefs quickly when evidence changes. Good forecasting requires effort, curiosity, and continual updating rather than one-time intuition. Public and political institutions would likely make better decisions if they measured and tracked forecasting accuracy more rigorously.

Data Points: Fantasy Football Nerd pundits predicting Seahawks vs. Panthers: 36 picked Seattle; 2 picked Carolina - Example of consensus forecasting failing when Carolina beat Seattle 27-23. Carolina record at prediction time: 4-0 - Panthers were undefeated before upsetting Seattle. Seattle record at prediction time: 2-3 - Seahawks were struggling despite their pedigree. Super forecasting tournament duration: 4 years - IARPA forecasting tournament ran from 2011 to 2015. IARPA questions posed: roughly 500 - Geopolitical forecasting questions across the tournament. Forecast judgments collected: in excess of 1 million - Cumulative individual probability judgments submitted by forecasters. Performance objective: 50% better than the unweighted average of the crowd - IARPA’s fourth-year benchmark for forecasting teams. Top forecasters labeled: top 2% - Best-performing forecasters were elevated to 'super forecasters'. Size of top forecaster group: about 60 people - Top 2% of roughly 3,000 forecasters. Amazon gift certificate compensation: a couple hundred dollars - Incentive paid to volunteer forecasters like Mary Simpson. Effective hourly compensation: about 20 cents an hour - Simpson estimated the volunteer work paid very little relative to time spent. Probability example for Bay of Pigs: about 1 in 3 - Joint Chiefs’ 'fair chance of success' was later interpreted as roughly 33%. Obama bin Laden probability range: 0.4 to 0.95, center around 0.75 - Tetlock describes the probability estimates Obama received about bin Laden’s location.

Pivotal Quotes: "I find all media comical at times. Because I think in you guys' profession, you can easily take back what you say. And you don't get, there's no danger, you know, when somebody says it." — Cam Newton: Used as the episode’s opening illustration of why prediction without accountability is unreliable. "They're not really who they are, Tetlock writes, it's what they do." — Stephen Dubner quoting Tetlock: Summarizes the central claim that super forecasting is a learned set of habits rather than an innate gift. "When you don't have skin in the game and you aren't held accountable for your predictions, you can say pretty much whatever you want." — Stephen Dubner: Transition from sports commentary to the broader critique of expert forecasting.

Implications: Listeners are encouraged to distrust vague expert certainty and value numerically expressed, accountable forecasts. The episode suggests institutions should score predictions, train forecasters, and use probability language to improve policy, journalism, and everyday decisions.