[REPLAY] Gary Klein with Paul Sonkin and Paul Johnson – Conducting Pre-Mortem Analysis (Capital Allocators, EP.109)

Episode Summary

Executive Summary: The episode explores expertise and decision-making under uncertainty, centered on Gary Klein’s methods: recognition-primed decision-making, Shadowbox training, cognitive after-action reviews, and especially the premortem. The guests argue that proper premortems surface hidden risks, improve candor and psychological safety, reduce overconfidence, and make teams smarter—if done with the right framing, structure, and time discipline.

Main Topics: Gary Klein’s research on expertise (Priority: 5/5): Klein explains how his work with Air Force pilots led him to conclude that complex work relies on tacit knowledge and pattern recognition, not just rules or checklists. Recognition-primed decision-making (Priority: 5/5): He describes how experts rapidly size up situations, mentally simulate a likely action, and revise if needed, rather than generating and scoring many options. Shadowbox as expert training (Priority: 4/5): The Shadowbox method uses scenarios, forced rankings, and expert benchmarks to show learners what experts notice, why they decide, and what they miss. Cognitive after-action reviews (Priority: 4/5): The CARGO approach goes beyond outcome evaluation to examine what decision-makers were seeing and interpreting at the time of judgment. Premortem method and proper framing (Priority: 5/5): The discussion focuses heavily on how premortems work when teams imagine a plan has already failed, then write down why, to surface risks before execution. Psychological safety and group dynamics (Priority: 5/5): The guests emphasize that premortems create a safer environment for junior voices, improve candor, and prevent dominant personalities from suppressing diverse insights. Comparisons with devil’s advocates, red teams, and risk checklists (Priority: 3/5): They contrast premortems with other risk tools, arguing that devil’s advocates can be discounted and that checklists/risk factors are often too generic or rigid.

Key Arguments: Expertise in complex domains is largely tacit; people often "just know what to do" because of accumulated pattern recognition and experience, not explicit rules. Recognition-primed decision-making reflects how real experts decide: they quickly diagnose the situation, mentally test an action, and move on if it fails. Shadowbox can accelerate learning by revealing how experts interpret the same scenario, what cues they prioritize, and what warnings they detect. Cognitive after-action reviews should ask not only what should have been done differently, but what the person was perceiving and thinking at the time. A premortem works best when participants are told the plan has already failed; this reframing unlocks more candid and imaginative risk identification than asking what could go wrong in general. Having everyone write independently before discussion reduces production blocking, prevents group influence, and captures more distinct ideas. The leader must go first and show seriousness; without buy-in, participants will hide their real concerns or treat the exercise lightly. Premortems improve psychological safety by turning criticism into a constructive contest to identify the best risks, not a career-threatening objection. Premortems reduce overconfidence more effectively than simply listing pros/cons or critiques. Devil’s advocates can fail because their role is easy to discount and can let the rest of the group outsource critical thinking. Red teams are powerful but often too expensive or slow for frequent, fast-moving decisions like investing. Risk checklists are useful but too mechanical and context-insensitive compared with a context-specific premortem. The ultimate value of a premortem is better decisions, richer group knowledge, and fewer costly mistakes at relatively low time cost.

Data Points: Years of expertise research: 40+ years - Klein’s career studying expertise and decision-making Shadowbox LLC founded: 2005 - Klein’s company for training decision-makers Premortem development: about 30 years ago - Klein says the method dates back roughly three decades Air Force project example timing: 1993 - Premortem used on a complex Air Force software effort Company size: 37 people - Klein’s research company had grown to this size before being sold Scenario-writing time in premortem: 2 minutes - Participants write reasons the plan failed Optimal meeting length: 20-30 minutes - Guests say a premortem should be brief; an hour is too long Alternative timing that was too short: 1 minute - Too little time left people frustrated while still writing Alternative timing that was too long: 3 minutes - Too many people finished early and lost energy Project team size example: 27 people - A listener/host example where 27 risks were generated Ideas generated in example: 27 things - One premortem session produced one idea per attendee Critical risks identified in example: 5 path-critical - Five of the 27 items were deemed critical to project success Confidence effect: premortem reduced confidence more than listing pros/cons or critiques - Klein cites comparative evidence about confidence reduction Investor batting average reference: 55% - Used to argue that even small error reduction has major value in investing Payment disclosure: flat fee - WCM compensated Capital Allocators with a flat fee for the testimonial segment

Pivotal Quotes: "Expertise is not about just incorporating all the rules. Expertise is about building tacit knowledge." — Gary Klein: Explaining what he learned from studying fighter pilots and complex tasks "Now, everybody around the table, take two minutes and write down all the reasons why this plan failed." — Gary Klein: Describing the core premortem instruction and its reframing effect "we're now allowing people to express their concerns that they've had all along that they've been afraid to bring up" — Paul Johnson: On why premortems unlock hidden concerns and improve candor

Implications: For investors and leaders, the premortem is a low-cost way to expose hidden risks, improve team candor, and reduce costly overconfidence. Used correctly, it can materially improve decision quality without slowing execution much.