Jeremy Howard: fast.ai Deep Learning Courses and Research

Episode Summary

Executive Summary: Jeremy Howard argues that deep learning should be made radically more accessible to domain experts through practical tools, better abstractions, and smaller, faster workflows. He criticizes academia’s incentives, Python’s limitations, and oversized compute culture, while emphasizing transfer learning, active learning, and human-in-the-loop systems for medicine, research, and society.

Main Topics: FastAI’s mission: empowering domain experts (Priority: 5/5): Howard explains FastAI as a free, practical effort to put deep learning into the hands of people who already know the problem space, rather than relying on AI specialists to solve every domain. Programming languages and hackability (Priority: 5/5): He contrasts languages and environments—Access, Delphi, J, Perl, Python, Swift—arguing that the future should be more hackable, expressive, and conducive to experimentation, especially for data-centric work. Compute, GPUs, and compiler technology (Priority: 5/5): Howard argues that deep learning research is constrained by low-level GPU programming complexity, and that MLIR, Halide-inspired abstractions, and eventually Swift could make tensor computing far more accessible. Practical deep learning over academic incentives (Priority: 5/5): He criticizes much of deep learning research as detached from real-world utility, praising transfer learning and active learning as impactful but under-studied areas that directly help practitioners. Medicine, privacy, and data governance (Priority: 4/5): He sees deep learning as especially valuable in medicine for triage and diagnosis in underserved regions, while stressing that patients should control their medical data and that regulation/lawyers often impede beneficial use. Training speed, DawnBench, and small-data experimentation (Priority: 4/5): Howard recounts how FastAI won DawnBench by using smaller images, higher learning rates, and better practices, emphasizing that most breakthroughs should be feasible on a single GPU. Learning, teaching, and persistence (Priority: 4/5): He describes learning through repeated experimentation, teaching, spaced repetition, and tenacity, arguing that almost anyone can succeed if they keep going and work on problems they care about.

Key Arguments: Deep learning should be taught and used in practical settings by domain experts, not gated behind AI specialists. The best way to improve deep learning is to focus on transfer learning, active learning, and better use of existing data rather than chasing ever larger datasets and compute. Python is a major bottleneck for research because it is slow and unhackable in key low-level areas like RNN kernels and GPU code. A better stack would combine expressive high-level languages (Swift/J-like ideas) with compiler infrastructure (MLIR/ Halide-style optimization) to target GPUs efficiently. Most academic deep learning research is misaligned with real-world impact because publication incentives favor safe incremental work. Medical AI is most useful for triage and decision support in low-resource settings, where human expertise is scarce and should be amplified, not replaced. Privacy concerns are often best addressed by giving users control over their own data and by doing more with less data instead of aggregating everything. Single-GPU workflows and smaller datasets can often achieve state-of-the-art or near-transferable results, making AI more accessible and innovative. Success in startups and deep learning alike depends heavily on persistence, low costs, and working on a problem you truly understand. Ethical AI requires considering feedback loops, human appeals, and the societal consequences of high-leverage models.

Data Points: FastAI courses: Free - Howard says all FastAI courses and materials are provided at no cost. Time to complete FastAI courses: 2 months to 2 years - He says completion depends on background and time commitment. Competent coder workload: 70 hours per course - A strong coder can work through a course relatively quickly. FastMail initial server cost: $70/month - He used very low operating costs to bootstrap the company. Early email storage pricing: $10/year for more than 10 MB - FastMail monetized by charging users once storage exceeded a small free allotment. Optimal Decisions scoping projects: $50,000 to $100,000 - He used paid prototype/scoping projects to fund the consultancy startup. Developing-world doctor shortage: 10x shortage - Howard describes a large global shortage of doctors, especially outside wealthy regions. Estimated time to train enough doctors: 300 years - He uses this estimate to argue for AI-assisted medicine. Africa pediatric radiologists: 5 pediatric radiologists outside South Africa - He highlights extreme scarcity of specialists across the continent. ImageNet training threshold in DawnBench: 93% accuracy - The benchmark required very high accuracy while optimizing for speed/cost. DawnBench ImageNet result: 18 minutes - Howard says FastAI later reached the top of the leaderboard with a distributed setup. DawnBench target: Under 12 hours - Their goal for ImageNet training speed during the competition. Image size reduction: 64x64 instead of 224x224 - Smaller images helped them train faster and still transfer well. Learning rate effect: 10x higher learning rate -> 10x faster training - He cites superconvergence work as dramatically accelerating training. Medical data access example: Catapult Health, LabCorp, Walgreens downloads to phone - He references Doc AI-style tools that let patients control and share records. Model fine-tuning time: 5 minutes - He says FastAI can fine-tune an ImageNet model very quickly for a new domain. ImageNet subset training: 10 minutes on a single GPU - He describes smaller subsets like ImageNet French/ImageWolf for fast experimentation. Lesson one example: ~100% accuracy - His teddy bear / grizzly / brown bear classifier reached near-perfect performance. Historical flashcard lapse: 3 years - He says spaced repetition resumed successfully even after a multi-year break.

Pivotal Quotes: "“The goal is to use the people that you have better, and the data you have better, and the computation you have better.”" — Jeremy Howard: He contrasts FastAI’s philosophy with big-tech approaches that rely on massive compute and data. "“Most things don’t work very well, even though it works better than anything else out there.”" — Jeremy Howard: He describes deep learning as powerful but still primitive, leaving lots of room for practical innovation. "“Train lots of models.”" — Jeremy Howard: His core advice for beginners: learn by doing, fine-tuning, and experimenting with real data.

Implications: Listeners should expect AI progress to come from practical, low-cost, human-centered experimentation—not just giant models. The near-term opportunity is to apply deep learning in real domains, especially with better tools, smaller data, and stronger ethics.