The future of AI and the legal field

Episode Summary

Executive Summary: Stanford Law’s Julian Nyarko argues AI is already strong at many legal tasks, and the real challenge is using it to cut costs, expand access to justice, and improve legal education without harming learning. He describes his lab’s work on evaluating AI in law, auditing bias, and using agentic AI to accelerate social science research and test claims in legal scholarship.

Main Topics: AI in law as a practical tool, not a future possibility (Priority: 5/5): The conversation frames AI as already useful in legal workflows, especially for drafting, tutoring, evaluation, and research, shifting the key question from capability to responsible deployment. The Lyft Lab’s four research pillars (Priority: 5/5): Nyarko explains his lab’s agenda: evaluation of legal AI, improving legal services, improving legal education, and methods research to adapt AI techniques for law. AI tutoring study in contracts education (Priority: 5/5): A major study compared AI-generated answers to law professors’ answers to office-hour style questions, finding AI matched or exceeded human performance on preference rankings and had lower rates of pedagogically harmful responses. Limits of the study and cautions about learning outcomes (Priority: 4/5): Nyarko emphasizes that the study does not prove AI improves student learning or replace professors; it tested single-turn answers, not interactive tutoring or long-form pedagogy. AI as an accelerator for social science and legal scholarship (Priority: 4/5): The lab uses agentic AI to speed up research, check scholarly claims against evidence, and enable projects that would have been impractical without automation. Auditing bias and disparate treatment in LLMs (Priority: 5/5): Nyarko describes experiments showing name-based disparities in model outputs, plus follow-up work on model editing and distinguishing context-invariant from context-specific bias. Policy and regulatory implications of AI bias (Priority: 4/5): The work suggests some biases are embedded broadly in models while others are deployment-specific, which has consequences for regulation, model safety, and downstream accountability.

Key Arguments: AI is already good enough to perform many legal tasks, so the pressing issue is how to deploy it effectively rather than whether it can do the work. Legal evaluation is hard because law lacks a clear universal definition of a “good” answer; this makes contract quality and legal-service quality difficult to benchmark. AI tutoring can be responsible if used as an always-available teaching assistant, but the evidence does not yet show it improves learning outcomes or can replace human instructors. Single-turn answer quality is not the same as interactive legal coaching; office-hours-style dialogue may require human-like negotiation and clarification that the study did not test. AI can substantially speed up social science by automating data gathering, matching, and analysis, and can enable novel research questions that were previously infeasible at scale. Agentic AI can be used to audit scholarly and legal claims, helping authors, editors, and researchers verify whether broad statements are actually supported by evidence. Bias in LLMs can be measured through controlled name-based experiments, showing disparate treatment that may persist across contexts or vary by deployment. Some model bias appears context-invariant and some highly context-dependent, suggesting that both model-level fixes and deployment-level oversight are necessary. AI may reduce the cost of legal services and expand access to justice, but measuring real-world impact on courts, regulation, and institutions remains difficult.

Data Points: Number of law professors in AI tutoring study: 16 - Professor participants who ultimately agreed to the study of AI answers versus human professor answers Student-style questions per professor: 40 - Each professor generated office-hours or after-class questions for the contracts study AI preference rate: 75% - Across instructors, AI answers were preferred over professor-generated answers in pairwise rankings Pedagogically harmful rate for instructors: 10–12% - Instructors flagged each other’s answers as harmful at this rate Pedagogically harmful rate for LLM answers: 1–2% - LLM-generated answers were flagged at low single-digit rates as harmful Context-invariant share of bias: ~50% - In the bias study, roughly half of measured bias was not dependent on context Context-dependent share of bias: ~50% - The other half of measured bias varied significantly by domain or deployment context Corpus size in early contracts research: 500,000 contracts - Nyarko’s PhD-era NLP work used SEC material contracts at large scale Year of initial NLP work: 2018 - He says he started working with NLP before AI became widely popular Question type tested: Single-turn - The tutoring study evaluated one question followed by one answer, not multi-turn dialogue

Pivotal Quotes: "AI is good at doing law." — Julian Nyarko: His core takeaway about legal AI and the main theme of the episode "What kind of contract do we want it to write for us?" — Julian Nyarko: Discussion of evaluation and the challenge of defining quality in legal drafting "The question is: how do we best use it to decrease costs and increase justice?" — Russ Altman: Opening framing of the episode’s focus on AI in the legal system

Implications: Listeners should expect AI to reshape legal services, legal education, and legal research. The big challenge is evaluation: defining quality, preventing harm, and pairing model-level and deployment-level safeguards to improve access to justice.