Silencing the ‘Noise’ Behind Bad Corporate Decisionmaking

Episode Summary

Executive Summary: The episode centers on Daniel Kahneman and Olivier Sibony explaining “noise” as unwanted variability in human judgment—distinct from bias—and why it is a major, often hidden organizational problem. They argue that decision quality can be improved through decision hygiene, structured processes, and sometimes algorithms, while emphasizing that diversity helps generate perspectives but final judgments should be made consistently and with independence.

Main Topics: Noise vs. bias (Priority: 5/5): Kahneman defines noise as random variability in judgments, contrasting it with bias, which is the average directional error. The key point is that organizations often focus on bias while underestimating noise. The insurance underwriter audit (Priority: 5/5): Kahneman describes a consulting experiment in which multiple underwriters evaluated the same typical cases; the variability was far larger than executives expected, illustrating how hidden and severe noise can be. Noise as an organizational disease (Priority: 5/5): Sibony frames noise as a problem that emerges when organizations need consistency across many people making decisions on behalf of the institution, including companies, courts, and public bodies. Decision hygiene and structured judgment (Priority: 5/5): The discussion explains decision hygiene as a general method for improving judgment, especially through structured interviews, independent assessments, and delaying overall intuition until relevant information has been gathered. Experts, respect experts, and measurable expertise (Priority: 4/5): Kahneman distinguishes experts whose performance can be objectively evaluated from 'respect experts' whose authority is based on reputation or persuasion rather than measurable accuracy. Algorithms, AI, and the future of decision-making (Priority: 4/5): The speakers debate AI-based screening and algorithmic decisions: in principle they can reduce noise, but in practice current tools are often poorly validated; however, algorithms are improvable through feedback. Diversity and consistency in final judgments (Priority: 4/5): They argue that diversity is valuable in generating ideas and perspectives, but final organizational judgments should aim for consistency rather than simply reflecting different opinions.

Key Arguments: Noise is an unwanted variability in judgment, and it can be as important as bias in producing error. Organizations are especially vulnerable to noise because they need consistent decisions across many decision-makers. Executives systematically underestimate the amount of disagreement among experts; actual variation can be far greater than expected. Structured interviews and similar procedures reduce noise by forcing evaluators to assess attributes separately and delay global intuition. Independent evaluation of different dimensions or by different people reduces halo effects and correlated errors. Algorithms can reduce noise in principle, but many current products are poorly designed and not clearly linked to job success. Diversity is useful during deliberation, but final decisions should not simply preserve every viewpoint; they should seek the best judgment. Some fields allow true expertise to be measured against a gold standard, while others only allow 'respect expertise' based on credibility and experience. In high-stakes settings like foreign policy, decision hygiene still matters: leaders should cover all bases and think through consequences, even under pressure.

Data Points: Expected disagreement between two underwriters: 10% - Executives guessed this was the typical percentage difference between judgments. Observed disagreement between two underwriters: 50% - Noise audit result in the insurance underwriting experiment, about five times higher than expected. Relative magnitude of error: 5x larger - Actual underwriter variability compared with executives’ expectations. Number of underwriters: about 50 - Approximate number of underwriters who evaluated the constructed cases in the audit. Time reference: about eight years ago - Kahneman dates the origin of the study to his consulting work at an insurance company. Publication timing: a year ago - Referenced as the time when the book Noise came out.

Pivotal Quotes: "Wherever there is judgment, there is noise, and there is more of it than you think." — Daniel Kahneman: Summarizing the central thesis of the book and the episode. "Noise is a disease of organizations." — Olivier Sibony: Describing why unwanted variability matters most when decisions are made on behalf of an institution. "We want to distinguish between the process of generating a judgment and the final judgment." — Daniel Kahneman: Explaining how diversity is valuable in deliberation but consistency matters in the outcome.

Implications: Listeners should expect far more variability in expert decisions than intuition suggests. For organizations, better structure, independence, and feedback-driven algorithms can improve fairness and accuracy, though current AI tools still need scrutiny.