Episode Summary
Executive Summary: Daniel Kahneman and Olivia Sibony argue that many organizational decisions suffer not just from bias but from noise: unwanted variability in judgments that should be consistent. They distinguish “real experts” from “respect experts,” explain why human judgment is inherently noisy, and propose “decision hygiene” for organizations—especially through structured processes, independent aggregation, and noise audits—while cautioning against over-bureaucratization and noting that creativity should remain diverse rather than standardized.
Main Topics: Expertise vs. “Respect Experts” (Priority: 5/5): The speakers distinguish verifiable expertise from authority based on reputation, confidence, and eloquence. In domains like forecasting or hiring, performance must be measured against objective benchmarks to qualify as true expertise. Noise as a Distinct Decision Error (Priority: 5/5): They define noise as variability in judgments where the correct answer should be the same, emphasizing that it is different from bias and often overlooked because it is harder to identify in individual cases. Decision Hygiene and Organizational Process Design (Priority: 5/5): The conversation explains how organizations can reduce noise through independent aggregation, structured judgments, relative comparisons, and other process controls that resemble handwashing as preventive discipline. Limits and Trade-offs of Standardization (Priority: 4/5): While noise reduction can improve accuracy and fairness, the speakers warn that overly rigid procedures can create bureaucracy, reduce agency, and be inappropriate in contexts requiring creativity or divergent thinking. Creativity vs. Judgment (Priority: 4/5): They draw a sharp line between judgment (aiming for one correct answer) and creative work (where divergence and variation are desirable), using examples like product innovation and Steve Jobs. Measurement, Audits, and Competitive Advantage (Priority: 5/5): Noise can be measured through audits without knowing the correct answer, and organizations that reduce it may gain an edge over equally noisy competitors, especially in hiring, insurance, and investment. Bias, Social Media, and Public Discourse (Priority: 3/5): The speakers suggest that social media and echo chambers amplify bias more than noise, reinforcing partisan views and making error more polarized in public debate.
Key Arguments: Human judgment is often noisy because people vary in how they make the same decision, even when the correct answer should be identical. Some experts are only “respect experts”: they are respected by peers but cannot be validated by performance, unlike weather forecasters, chess players, or investors. Noise reduction is best achieved through “decision hygiene,” meaning disciplined decision processes rather than ad hoc intuition. Independent opinions reduce noise; discussing opinions before aggregation can worsen it by introducing influence and conformity. Organizations should design decision processes, because individuals are less likely to reliably improve their own thinking through self-help alone. Noise audits are practical because variability can be measured even when the correct answer is unknown. Reducing noise is beneficial only when the gains outweigh the costs; some settings do not justify the expense or loss of agency. Judgment and creativity are different: creative work should allow variation, while judgment should aim for consistency and accuracy. Averaging independent judgments reliably reduces noise but does not remove bias. Noise reduction can create competitive advantage when rivals remain noisy, as in investment or hiring. Social media is framed mainly as a bias amplifier through echo chambers, not necessarily as a primary source of noise.
Data Points: Event date: June 2022 - Live conversation at Union Chapel, London. Noise reduction by averaging: Guaranteed to reduce noise by a known amount based on the number of independent observations - Explained as the mathematical basis for averaging independent judgments. Noise audit feasibility: Relatively easy - Speakers say organizations can measure variability without knowing the correct answer. Typical hiring accuracy: Quite poor - Future job performance is inherently difficult to predict, even under the best conditions. Patent office example: 2 officers per patent would effectively double costs - Used to illustrate why averaging may be too expensive in some contexts. Grading example: 2, 3, 5, or 10 professors per essay - Illustrates that some noise reduction methods are not worth the cost in low-stakes settings. Book structure: Part 2 of 3 - The episode is the second part of a multi-part live event conversation.
Pivotal Quotes: "There are judgments that we make, like long term forecasts, which cannot be verified." — Daniel Kahneman: Explaining why some domains produce “respect experts” rather than measurable experts. "When we talk about noise reduction or about decision hygiene, we had in writing the book, we had organizations in mind." — Olivia Sibony: Clarifying that the book is aimed at organizational decision-making, not individual self-help. "A crowd is not necessarily wise because averaging does not necessarily reduce bias." — Daniel Kahneman: Distinguishing noise reduction from bias reduction and challenging simplistic ideas about wisdom of crowds.
Implications: Listeners and organizations should treat inconsistency in judgments as a measurable cost. The practical lesson is to redesign decision processes, not just train individuals, while preserving flexibility for creativity and avoiding bureaucratic overreach.