How Better Feedback Can Revolutionize Education (with Daisy Christodoulou)

Episode Summary

Executive Summary: Russ Roberts and Daisy Christodoulou explore why educational feedback is often ineffective, arguing that prose feedback rarely tells students how to improve because improvement requires a “model of progression” and tacit practice, not just comments. They then assess AI’s promise and limits for grading and feedback, highlighting hallucinations, position bias, and the need for human-in-the-loop systems that use AI where it is strongest: transcription, synthesis, consistency, and scaling.

Main Topics: What feedback is for (Priority: 5/5): Christodoulou frames feedback as a thermostat rather than a thermometer: good feedback should change performance by closing the gap between current and desired states, not merely measure it. Why written feedback often fails (Priority: 5/5): They argue that comments like “confusing” or “infer more insightfully” are too vague or abstract to drive action; prose is often not optimized for producing the next improvement step. Tacit knowledge, progression, and practice (Priority: 5/5): Using Polanyi and sports analogies, Christodoulou says many skills require structured practice and intermediate activities that do not resemble the final task, such as vocabulary work for better inference or gym work for marathon training. AI’s promise and failure modes (Priority: 5/5): The conversation turns to large language models in grading and feedback: they can generate polished text, but hallucinate, behave inconsistently, and show position bias, making direct automation risky. Human-in-the-loop assessment systems (Priority: 4/5): Christodoulou explains how her organization uses AI to augment rather than replace humans—e.g., transcribing teacher audio comments, combining feedback, and routing difficult cases to humans. Assessment as the price mechanism of education (Priority: 4/5): Roberts and Christodoulou agree that assessment is essential, not optional; it operationalizes the curriculum, motivates students, and reveals where instruction must change. AI as tutor vs. AI as shortcut (Priority: 4/5): They discuss the tension between AI as a learning aid for advanced students and as a shortcut that may weaken memory, self-quizzing, and deep learning, especially for younger or novice learners.

Key Arguments: Written feedback often tells students what is wrong but not what action to take next; effective feedback must specify or enable a better progression step. Many educational improvements require tacit knowledge and structured practice that cannot be fully conveyed in prose. “Read more” and “write more” are insufficient as universal prescriptions; students often need targeted sub-skills such as vocabulary, sentence structure, or guided drills. AI models are attractive because they are fast and cheap, but hallucinations and non-deterministic behavior make direct automation dangerous in high-stakes contexts. LLMs are better used to assist humans than replace them: transcribing, summarizing, clustering themes, and routing uncertain cases can improve efficiency without sacrificing judgment. Comparative judgment is stronger than absolute grading for humans, but LLMs introduce their own biases, including position bias, so they need careful calibration. Assessment should not be abolished; it is the mechanism that makes educational goals concrete and helps teachers and students know whether learning is happening. For younger learners especially, technology cannot simply substitute for human teaching because learning depends on stability, structure, and guided interaction.

Data Points: EconTalk annual poll rank: Top 10 episodes of 2024 were announced; two episodes tied for first place - Opening remarks by Russ Roberts Human comparative judgments processed: About 40 million - Christodoulou discussing comparative judgment performance Human left/right bias in comparative judgment: 50.5% left vs. 49.5% right - Used to show humans have minimal position bias LLM position bias range: 10% to 25% - Christodoulou describing observed model instability in comparative judgment Marathon training example: 26 miles - Illustration that training should not mimic the end goal in every session Time horizon mentioned for training an AI/education system: 2 years of experimentation - Christodoulou describing how long hallucination/feedback issues have been investigated Expected feedback delay example: November to February - Student at an expensive independent school still waiting for work to be returned Educational age range discussed: 5 to 18 years old - Christodoulou notes her work spans primary through secondary education Selective cohort at university: Older, more expert students - Roberts contrasts university students with younger learners

Pivotal Quotes: "Prose is not optimised for action." — Daisy Christodoulou: Central thesis explaining why written feedback often fails to change student performance "Assessment is the price function of education." — Daisy Christodoulou: Her rebuttal to the idea that ideal education would have no assessment "This is the paradox of automation." — Daisy Christodoulou: Explaining why humans monitoring imperfect AI may perform worse than humans doing the task themselves

Implications: Schools should treat AI as a support tool, not a wholesale replacement for teachers or assessment. The best use cases are structured, human-supervised workflows that improve efficiency while preserving judgment, motivation, and real learning.