Noisy Decisions

Episode Summary

Executive Summary: The episode explores Daniel Kahneman’s distinction between bias and noise in human judgment. Bias is a systematic error; noise is unwanted variability across judgments that should be similar. Kahneman argues noise is widespread in settings like courts, medicine, and hiring, but can be reduced through averaging and better decision design or “decision hygiene.”

Main Topics: Bias vs. noise (Priority: 5/5): The conversation distinguishes predictable cognitive bias from random variability in judgment, emphasizing that noise is a separate and overlooked problem. Noise in real-world institutions (Priority: 5/5): Examples include judges sentencing defendants, doctors diagnosing patients, and insurance underwriters assessing risk, where similar cases produce widely different outcomes. The size of the problem (Priority: 4/5): Kahneman argues that people underestimate how much experts differ, with noise often far larger than intuition suggests. Averaging as noise reduction (Priority: 4/5): Combining independent judgments can mathematically reduce noise, though it is often too costly or impractical in institutional settings. Decision hygiene (Priority: 5/5): Kahneman recommends structured, step-by-step evaluation processes that delay global impressions to reduce variability in judgment. Structured vs. unstructured interviews (Priority: 4/5): Job interviews are used as a concrete example: evaluating candidates by separate criteria is presented as better than relying on early intuition. Applications beyond hiring (Priority: 3/5): The same structured approach may improve company and investment evaluation by breaking decisions into independent dimensions before forming an overall conclusion.

Key Arguments: Noise is not the same as bias: bias is systematic, while noise is inconsistent variation in judgments that should be similar. There is more noise in expert decisions than people expect, and it appears across many domains wherever judgment is involved. Judges, doctors, and insurance underwriters often disagree substantially on the same or very similar cases, creating lotteries in outcomes. Averaging multiple independent judgments reduces noise reliably, though it can be expensive and operationally difficult. Structured decision processes that postpone intuition until after separate criteria are assessed can reduce noise and improve judgment quality. Early impressions in interviews tend to dominate later evaluation, making unstructured interviews less effective than structured ones.

Data Points: Expected difference in underwriter judgments: 10% - People’s intuitive expectation for how much two underwriters might differ on the same risk. Actual difference in underwriter judgments: 50% or above - Kahneman says the real variability is about five times larger than people expect. Relative size of underwriter variability: 5x expected - The variability in underwriters is described as about five times greater than expected. Average sentence in judge experiment: 7 years - In a U.S. experiment, this was the mean sentence handed down by judges. Difference between two random judges’ sentences: 4 years - Shows the sentencing lottery faced by defendants when different judges hear similar cases. Interview timing of first impression: First 3–4 minutes - Evidence cited that interviewers form impressions very early, with the rest often spent justifying them.

Pivotal Quotes: "Noise is different from bias. It's not a predictable error. Instead, it's a variability in decisions where there shouldn't be variability." — Daniel Kahneman: Defines the core concept of the interview. "The identity of the judge, the state of the judge, makes a very big difference to the sentence." — Daniel Kahneman: Explains noise in sentencing and the resulting lottery for defendants. "Averaging, whenever it is possible, is a way to reduce noise." — Daniel Kahneman: Describes a mathematically reliable but often impractical solution.

Implications: Listeners should expect expert judgment to be noisier than they assume. Institutions can improve fairness and quality by using structured processes, independent assessments, and delayed intuition, especially in hiring, law, medicine, and finance.