E24: The AI Revolution in Medicine with Dr. Isaac Kohane of Harvard Medical School

Episode Summary

Executive Summary: The conversation centers on Zach Kohane’s early hands-on testing of GPT-4 in medicine and his conclusion that it is clinically powerful but not autonomous. He argues it should become a ubiquitous “sidekick” for doctors and patients—improving diagnostics, reducing errors, keeping clinicians current, and easing documentation—while still requiring human supervision because it can hallucinate, miss values, and lacks common sense and human grounding.

Main Topics: Early GPT-4 access and clinical astonishment (Priority: 5/5): Kohane recounts being given private early access to GPT-4 and testing it on difficult real-world cases, including a rare endocrine diagnosis it solved step by step. This convinced him the model was far beyond expected capability for its time. Hands-on evaluation over abstract theory (Priority: 5/5): He argues that people should directly test AI in their own domain with real examples rather than rely on speculation. His approach involved hundreds of scenarios spanning diagnosis, management, research, and billing to understand where the model excels and fails. Four paradigms for AI in medicine (Priority: 5/5): The discussion covers Kohane’s framework of trial, trainee, partner, and torchbearer. He rejects simple trial/benchmark models for general-purpose LLMs and sees the near-term best fit as a supervised partner, not an autonomous doctor. Why GPT-4 is powerful but unsafe alone (Priority: 5/5): Kohane emphasizes that GPT-4 is simultaneously smarter and stupider than humans: it can outperform many clinicians in knowledge and reasoning yet still hallucinate, miss context, and fail to understand patient values or tradeoffs, making human oversight essential. Near-term applications: documentation, updates, and system support (Priority: 4/5): He sees immediate value in multimodal assistants that listen to visits, draft notes, flag missed tests, keep doctors up to date, and integrate patient-specific or hospital-specific knowledge via embeddings and retrieval. Longer-term frontier: discovery and multimodal biomedical AI (Priority: 4/5): Kohane is optimistic about AI-assisted drug discovery, protein modeling, and constrained scientific hypothesis generation, but says current models are not yet scientifically supreme. He expects major gains from combining language models with tools like AlphaFold and hospital-scale data. Data ownership and alignment in healthcare AI (Priority: 4/5): He argues that AI should be aligned with patients and that patients should consent to data use. In his view, the ecosystem will likely include AIs for hospitals, insurers, doctors, and patients, but consumer alignment should be primary.

Key Arguments: GPT-4 can solve difficult medical cases at a level that surprised an expert pediatric endocrinologist, including a rare diagnosis that many doctors would miss. The best way to assess frontier AI is by extensive direct use in realistic scenarios, not by relying on benchmarks or theory alone. General-purpose medical LLMs are too broad and dynamic for a traditional randomized controlled trial to cleanly isolate causal effects in healthcare settings. GPT-4 can improve the bottom half of doctors substantially, helping bring average care closer to the top tier even if it remains imperfect. Human supervision is non-negotiable because the model lacks stable common sense, human values, and reliability in edge cases. Near-term value lies in partnership: keeping doctors up to date, reducing documentation burden, remembering patient details, and catching errors. Future discovery gains will likely come from multimodal systems that combine language, imaging, speech, embeddings, and biological modeling rather than from scaling language alone. Patients should control access to their own health data, since alignment with the patient is more trustworthy than alignment with institutions or payers.

Data Points: Early access timing: Fall of 2022 - Kohane received early preview/research access to GPT-4 before ChatGPT’s public release. Diagnostic benchmark scenario: 11-hydroxylase deficiency - GPT-4 independently progressed to this molecular diagnosis in a difficult pediatric case. Doctor comparison: 100 random doctors - Kohane said he would be surprised if even one of 100 random doctors could make that diagnosis. Clinical case volume: Hundreds of scenarios - He tested GPT-4 across hundreds of real-world clinical and research scenarios. Research/diagnostic yield in undiagnosed disease cases: 30–40% - He referenced the Undiagnosed Disease Network, where proper workups and sequencing can solve a substantial minority of cases. Patient-to-clinician interaction length: 10–15 minute meeting - He described the typical primary care visit as too short to answer most patient questions. Hospital integration scale: 800 hospitals - He noted Apple Health’s growing relationships enabling patient data download from many hospitals. Progress rate: Hundreds every month or so - He said Apple’s hospital connections are growing by hundreds per month. Future capability horizon: 5–10 years - He estimated cell simulation and major biomedical AI advances within this timeframe. Specific near-term horizon: Within the next five years - He expects major improvements in drug candidate triage and multimodal biomedical integration.

Pivotal Quotes: "It is simultaneously smarter than and stupider than any person you've ever met." — Nathan LeBenz quoting the book / Zach Kohane’s framing: Used to capture GPT-4’s mixed strengths and failures, especially in medicine. "No doctor should be without it. They should have this sidekick that is meticulous, completely up to date, ever vigilant, and sometimes wrong." — Zach Kohane: Describing how GPT-4 should function as a supervised clinical assistant. "The AI Revolution in Medicine, GPT-4 and Beyond" — Zach Kohane: Title of the book discussed throughout the conversation.

Implications: The episode frames AI as an imminent clinical copilot, not a replacement physician. Expect faster diagnosis support, better documentation, and patient empowerment, but also new standards around supervision, validation, data ownership, and AI alignment in healthcare.