James Zou: Trust is AI’s most critical contribution to health care

Episode Summary

Executive Summary: The episode examines how AI is reshaping healthcare through better diagnosis, treatment support, and clinical-trial design, while stressing that trust, bias, privacy, and regulation remain major barriers. Stanford’s James Tso argues that AI must be evaluated in real clinical workflows, not just on static benchmarks, and shows how real-world data can make trials more inclusive and efficient. The conversation also explores data deletion, model retraining, and the unequal value of patient data.

Main Topics: Trustworthy AI in healthcare (Priority: 5/5): The discussion defines trust as consistent performance across diverse settings and populations, not just strong benchmark results. Tso emphasizes that models must be reliable in real-world clinical environments. Data quality, bias, and evaluation (Priority: 5/5): AI systems can absorb spurious correlations from noisy historical data, so the data pipeline, coverage, and quality control are central to trustworthy deployment. Regulation and FDA readiness (Priority: 4/5): The episode argues that regulators, especially the FDA, are still developing appropriate standards for evaluating medical AI, especially because current approvals often rely on static rather than live clinical evaluation. Human-in-the-loop clinical use (Priority: 4/5): AI is framed as a partner to clinicians—second-reader support, workflow assistance, and decision support—rather than a fully autonomous replacement in medicine. Clinical trials as a bottleneck (Priority: 5/5): Tso explains that clinical trials are the gatekeeper for new treatments and often exclude older, minority, and lower-income patients, making them expensive, slow, and less representative. Trial Pathfinder and trial redesign (Priority: 5/5): A computational platform built with Roche Genentech uses real-world EHR data to simulate trials, relax overly restrictive criteria, and increase eligibility while preserving or improving outcomes. Data privacy, deletion, and model unlearning (Priority: 4/5): The segment on 'Making AI Forget You' explores how to remove an individual’s influence from trained models without full retraining, which matters for privacy law and user control.

Key Arguments: AI in healthcare has high potential because it can improve diagnosis, treatment selection, monitoring, and patient support, but its real-world value depends on trustworthiness and equitable performance. Trust should mean performance that generalizes across diverse locations and populations, not only success on a single institutional dataset or benchmark. The biggest gap in trustworthy medical AI is often the data pipeline: training and evaluation data may be biased, incomplete, or noisy, leading models to learn artifacts rather than clinically relevant signals. Current FDA evaluation methods are not yet fully adapted to AI; medical AI needs standards that assess prospective performance, data coverage, and interaction with clinicians. AI tools in medicine should be designed as part of a team-based workflow, since isolated model performance can differ from team-level patient outcomes. Most approved medical AI systems were evaluated on static datasets rather than in real-time human workflows, which limits confidence in their deployment. Clinical trials are a major bottleneck in translating discoveries into therapies, and overly restrictive criteria can exclude the very populations that will later receive the treatment. Using real-world EHR data to simulate many potential trial designs can reveal how to broaden eligibility, reduce cost, and improve representativeness. Data deletion is difficult because removing a record from a dataset does not automatically erase its influence from a trained AI model. Not all data have equal value; rare-disease, genetic, or highly informative patient records may contribute more to model quality and therefore raise questions about compensation and agency.

Data Points: FDA-approved AI algorithms evaluated in real-time human teams: 3 or 4 out of over 100 - Tso says most approved medical AI systems were validated on static data rather than in live clinical workflows. Patient recruitment cost for oncology trials: over tens of thousands of dollars per patient - Used to illustrate how expensive clinical trial enrollment can be. Eligible patients increased using Trial Pathfinder: more than doubled - Tso reports that trial redesign using real-world data more than doubled the eligible patient pool in cancer trials. Number of data points / scale for trial simulation: hundreds of thousands of patients - Trial Pathfinder uses large real-world datasets from medical records to simulate trial designs. Scale of synthetic experiments: millions of synthetic clinical trials - The platform computationally generates many trial variants by changing inclusion/exclusion rules. Conference reference: NeurIPS - Mentioned as the premier AI conference where benchmark-focused model development is common. Clinical trial screening burden: thousands of patients screened - Explains how restrictive eligibility criteria create high screening overhead for trials.

Pivotal Quotes: "We want to ensure that these AI algorithms... work well across different settings across diverse locations for diverse populations." — James Tso: Defining trust for medical AI beyond a single benchmark or institution. "Clinical trial is really both the gatekeeper and the bottleneck for this entire funnel." — James Tso: Explaining why trial design is central to translating research into patient care. "The algorithm might still remember enough information about me to be able to reconstruct my medical history, or reconstruct my browser history." — James Tso: Describing why deleting data from a dataset is not enough once a model has been trained.

Implications: Healthcare AI will need better data governance, regulatory standards, and human-centered design to earn trust. If successful, AI could broaden trial access, cut costs, improve fairness, and give patients more control over how their data is used.