Gary Klein with Paul Sonkin and Paul Johnson – Conducting Pre-Mortem Analysis (Capital Allocators, EP.109)

Episode Summary

Executive Summary: The episode explores Gary Klein’s research on expertise and decision-making, especially the premortem as a fast, practical way to surface hidden risks before a project starts. Klein, with Paul Johnson and Paul Sonken, explains why expert intuition, psychological safety, and structured dissent outperform rigid checklists, devil’s advocacy, and generic risk reviews—especially in complex, high-stakes settings like investing.

Main Topics: Gary Klein’s expertise research and naturalistic decision-making (Priority: 5/5): Klein traces his work from Air Force pilot training to broader studies of how experts recognize patterns, size up situations quickly, and use mental simulation rather than formal option scoring. Shadowbox as a training tool for expert judgment (Priority: 5/5): The Shadowbox method uses scenarios, ranked options, and expert comparisons to help learners see through experts’ eyes and uncover what they missed. Premortem method: origin, process, and value (Priority: 5/5): The discussion focuses heavily on premortems—imagining a plan has already failed, then generating reasons why—to improve risk assessment, reduce overconfidence, and stimulate candor. How to make premortems work in practice (Priority: 4/5): The guests detail subtle but essential design choices: leader buy-in, the failure framing, independent writing, short time limits, one-item-at-a-time sharing, and follow-on mitigation. Psychological safety and cognitive diversity in teams (Priority: 4/5): They argue premortems unlock suppressed concerns from junior or quieter team members, turning hidden group knowledge into visible decision advantages. Critiques of other risk tools (Priority: 4/5): Devil’s advocates, generic critiques, and broad risk checklists are portrayed as less effective because they can be outsourced, feel mechanical, or fail to reflect context. Applications to investing and organizational decision-making (Priority: 4/5): The speakers connect these methods to portfolio management, security analysis, investment committees, and any environment where a small reduction in errors can have outsized value.

Key Arguments: Expertise is mostly tacit knowledge developed through experience, not rule-following; real-world complexity exceeds what checklists can cover. The traditional rational-decision model is unrealistic because skilled people usually do not generate exhaustive option sets and scoring matrices. Shadowbox improves training by comparing a learner’s judgments against expert judgments within realistic scenarios, making hidden expert cues visible. A premortem works because it changes the frame from debating whether a plan might fail to assuming it already has failed, which lowers defensive bias and surfaces concerns. Two minutes of private writing before discussion is enough to elicit many risks while preserving energy and avoiding groupthink. The leader must set the tone by contributing a serious concern first; otherwise participants will self-censor. Psychological safety is essential: people speak up when the setting makes dissent feel legitimate rather than risky. Devil’s advocates often underperform because the group can discount their views as role-playing, and the responsibility to critique gets outsourced. Premortems are especially useful in investing because avoiding even a small number of mistakes can materially improve outcomes. The technique is powerful because it converts unknown unknowns into known unknowns at very low time cost, without claiming to eliminate all risk.

Data Points: Gary Klein’s career studying expertise: 40+ years - Klein describes the span of his work on expertise and decision-making. Research company size: 37 people - His firm had grown to this size before being sold in 2005. Premortem practice at kickoff: About 30 years ago - Klein says the method was developed roughly three decades earlier. Two-minute writing interval: 2 minutes - Chosen as the sweet spot for premortem idea generation after trial and error. One-minute attempt: Too short - People were still writing when time ran out. Three-minute attempt: Too long - Many finished early and energy dropped. Premortem meeting length target: 20-30 minutes - Klein says a premortem should be brief; an hour is too long. Alternative overly long duration: 2 hours - Used as an example of a bad premortem implementation. Investment batting average cited: About 55% - Illustrates how even small improvements in error rate matter a lot in investing. Project team in example: 27 people - A listener example described a premortem with this many participants. Ideas generated in example: 27 ideas - Each participant wrote down one issue on the first pass. Path-critical issues in example: 5 - Out of 27 ideas, five were deemed path critical. Missed critical issues by leader: 4 of 5 - The team leader had only thought of one of the five major issues. Publishing channel: SSR N / academic paper - The paper is being made available publicly and considered for publication.

Pivotal Quotes: "We don't follow any rules. We just know what to do." — Gary Klein: Fighter pilots explaining that expertise in complex environments is tacit, not rule-based. "I'm looking in a crystal ball, and it's now six months in the future... I see a disaster. This plan has failed." — Gary Klein: Klein describing the core reframing that makes the premortem work. "The premortem allows you to change the mindset as Paul said, it failed. Why did it fail?" — Paul Johnson: Explaining why the failure frame uncovers better risks than generic critique.

Implications: For investors and operators, the episode argues that disciplined dissent, not more analysis, is often the highest-ROI improvement. Premortems and similar tools can cheaply reduce overconfidence, strengthen team candor, and uncover hidden failure modes before capital is committed.