Episode Summary
Executive Summary: Burr Settles explains how Duolingo uses machine learning across three pillars: building high-quality content aligned to proficiency standards, personalizing learning by modeling what users know, and improving engagement through targeted messaging. He traces Duolingo’s evolution from crowdsourced translation to a freemium model centered on the Duolingo English Test and subscriptions, while emphasizing AI’s role in democratizing education at global scale.
Main Topics: Burr Settles’ path from NLP research to Duolingo (Priority: 5/5): Settles describes a career that moved from machine learning and biomedical NLP into active learning, then into Duolingo, where he saw an opportunity to invert active learning by using machines to teach people rather than the other way around. Duolingo’s mission and business model evolution (Priority: 5/5): The conversation covers the company’s original crowdsourced translation model, its shift away from that business, and the current model built around a free app, the Duolingo English Test, and subscription features. AI for curriculum quality and CEFR alignment (Priority: 5/5): Duolingo uses machine learning to map vocabulary and lessons onto CEFR proficiency levels, helping curriculum developers create content that is appropriate for learners at different stages and across multiple languages. Personalization, spaced repetition, and learning science (Priority: 5/5): Settles explains how Duolingo’s half-life regression replaced older flashcard-like scheduling systems to better predict forgetting and optimize review timing, improving retention and reducing wasted practice. Assessment and adaptive testing at scale (Priority: 5/5): The Duolingo English Test uses ML-generated or ML-ranked items, adaptive item selection, and psychometric fairness checks to deliver a low-cost, secure, internet-based proficiency exam. Engagement optimization through notifications and bandits (Priority: 4/5): Duolingo applies machine learning to choose push-notification content and optimize engagement, including bandit methods that account for eligibility constraints and novelty effects. Speech, LLMs, and future interactive feedback (Priority: 4/5): Settles notes Duolingo is moving more speech capabilities in-house and is exploring language-model-based features for more spontaneous production and interactive feedback rather than simple rote exercises.
Key Arguments: The best scalable substitute for a one-on-one tutor is AI, because it can help deliver content, personalization, and feedback that most learners otherwise cannot access. Machine learning is most valuable in education when it helps model what a learner knows, what they forget, and what content is at the right difficulty level. Duolingo’s CEFR tools let curriculum teams align content to proficiency levels across languages, even when high-quality labeled resources do not exist for every language. Older spaced-repetition systems can be improved by learning from data; Duolingo’s half-life regression outperformed the prior Leitner-style scheduler and increased retention. The Duolingo English Test uses adaptive ML-based item selection and massive item variation to make an online, low-cost exam more secure than traditional fixed-form tests. Fairness is essential in automated scoring, especially for speech, because differences in accents, register, or gender could otherwise produce biased outcomes. For engagement, simple heuristics can be enough when they are understandable, but ML becomes useful when rules become too complex or when random selection leaves performance on the table. AI should not replace teachers, but it can democratize access to high-quality learning experiences that would otherwise be unavailable to most people.
Data Points: Languages spoken among employees: about 30 - Settles says Duolingo employees collectively speak roughly 30 languages. Languages taught by Duolingo: about 30 - He notes the company teaches about 30 languages, though the overlap with employee languages is not exact. Duolingo English Test price: $49 - Settles says the test is low-cost and can be taken online anytime, anywhere. Retention lift from half-life regression: 12% boost - He cites an A/B test showing the new spaced-repetition model improved retention (app return the next day). Effect of the new test on beta graduation time: about 5 weeks vs. about 6 months - New language courses like Latin, Arabic, and Scottish Gaelic graduated from beta much faster after the report-prioritization model was introduced. Report volume: about half a million per week - Users submit reports when they think an exercise is marked wrong; these are prioritized by a machine learning model. Duolingo English Test adoption during lockdown: 2000% increase - Settles says test usage surged when traditional test centers closed during lockdowns. Item exposure rate: less than 0.5%, maybe less than 0.1% - He says the adaptive English test has very low repeated-item exposure because every test administration draws a different set of questions. Human-learner efficacy finding: checkpoint five ≈ 4 or 5 college semesters - A white paper reported that English speakers learning French or Spanish who reached checkpoint five performed at that level in listening and reading. Course-stage scale: 6 CEFR levels (A1 to C2) - Settles explains the CEFR framework used to map content difficulty and learner proficiency.
Pivotal Quotes: "Instead of figuring out how to best use people to teach machines, how can we best use machines to teach people?" — Burr Settles: He explains the conceptual shift that led him from active learning research to Duolingo. "The best way that you can scale that kind of experience is with AI." — Burr Settles: He argues AI is the only practical way to provide tutor-like personalization to large numbers of learners. "Our strategy in combating that was to make it a computer adaptive test with a huge number of items." — Burr Settles: He describes how the Duolingo English Test achieves security and scalability without test centers.
Implications: Duolingo shows how AI can move beyond content delivery into personalized pedagogy, assessment, and engagement. For edtech, the model suggests that adaptive systems plus rigorous measurement can make high-quality learning more accessible, scalable, and fair.