Episode Summary
Executive Summary: Jeremy Howard argues that 2018’s biggest deep learning shift was making models faster, cheaper, and more accessible through transfer learning, dynamic training tricks, and better tooling. He is bullish on practical progress in NLP, vision, and medical imaging, but skeptical of overhyped adversarial methods, GANs, and much of RL. He also warns that generative text will intensify disinformation and that the field needs more reproducibility and open data/code.
Main Topics: Accessibility and efficiency of deep learning (Priority: 5/5): Howard frames 2018 as a year focused on making deep learning usable for normal people solving real problems, emphasizing faster, cheaper training and lower barriers to entry. Training innovations: OneCycle, progressive resizing, AdamW (Priority: 5/5): He credits dynamic training methods and optimizer improvements with dramatic gains in speed and stability, including large reductions in training time and cost. Transfer learning as a dominant paradigm (Priority: 5/5): Howard argues transfer learning is still underappreciated but has become central across vision and NLP, enabling strong performance with much less data. Skepticism toward adversarial methods, GAN hype, and RL overuse (Priority: 4/5): He says adversarial attacks are often not practically useful, GANs are overhyped though now more usable, and RL is frequently applied to problems better solved with other methods. Interpretability of optimization and loss landscapes (Priority: 4/5): Howard highlights work showing smoother loss surfaces from ResNets, DenseNets, batch norm, spectral norm, and weight norm, helping explain why these methods work. Tools and frameworks: PyTorch, FastAI, TVM, Swift for TensorFlow (Priority: 4/5): He discusses 2018’s tooling progress, praising PyTorch’s developer experience, TensorFlow’s reaction, and compiler/runtime efforts that could make low-level optimization more accessible. Education, inclusion, and global access (Priority: 5/5): Howard says FastAI has helped people worldwide enter the field, especially underrepresented communities, and that lowering coding barriers is key to expanding participation.
Key Arguments: Deep learning should be easier, cheaper, and faster so domain experts can solve real-world problems without needing heavy ML infrastructure. Training methods like OneCycle scheduling, reduced momentum at high learning rates, AdamW, and progressive resizing materially improve speed and results. Transfer learning remains one of the most important but underused ideas in ML, especially for small datasets and NLP. GANs and adversarial methods are often overapplied; many tasks can be solved better by designing better loss functions or using simpler augmentation approaches. RL is valuable in constrained physical domains like robotics, but is often misused for tasks that are really prediction or classification problems. Loss landscape and normalization research is clarifying why ResNets, batch norm, spectral norm, and weight norm improve optimization. The field needs stronger reproducibility norms: publish code and data, use strong baselines, and perform ablation studies rather than relying on theory alone. Open, domain-specific model sharing may be more valuable than centralizing data in domains like medical imaging and radiology. Generative text will likely become a major disinformation weapon, and defending against it will be harder than building it. Education efforts like FastAI are expanding access globally and helping underrepresented groups participate in ML research and practice.
Data Points: DawnBench ImageNet training time: 18 minutes - FastAI reached first place using Amazon AWS commodity hardware DawnBench ImageNet training cost: $40 - FastAI’s AWS-based result on DawnBench ImageNet training on specialized hardware: 7 minutes - Howard cites related work by other researchers on specialized equipment NLP data reduction with transfer learning: 100 times less data - FastAI sentiment classification results using transfer learning PCA variance in training trajectory: 40% to 90% - A study found most gradient-update variation lies in just two PCA directions Scale of model discussed: Over 100 million weights - ImageNet-scale network used in loss landscape visualization work Image size in progressive resizing: 64x64 to 288+ - Howard describes training mostly on small images, then finishing at larger sizes GAN training time: About 1 hour on a single GPU - FastAI’s more reliable GAN training workflow FastAI student geography: Lagos is the second biggest FastAI community outside the US - Howard uses this as evidence of global growth in access Conference trend frequency: Every second paper - Howard jokes that at NeurIPS many papers are about adversarial methods or reinforcement learning Conference scale: 4,000 submissions - He references the size of NeurIPS when discussing publication standards Training time improvement claim for RL paper: 2x better training times - Howard cites Sergei Levine’s mention of the TD3 paper Public dataset access issue: Almost no publicly available labeled text datasets - Howard criticizes licensing restrictions in NLP Expected publication scale: 1,000 papers - He cites the volume of papers being published and the need for code/data requirements
Pivotal Quotes: "the stuff that I care about is increasing accessibility of deep learning for normal people to solve normal problems" — Jeremy Howard: Summarizing his 2018 focus and FastAI’s mission "I think my view is probably GANs are a little overhyped, and adversarial attacks are a little overhyped." — Jeremy Howard: His skeptical assessment of two major 2018 research trends "As long as we rely on people knowing how to code, we're missing out on something like 99.9% of the global population." — Jeremy Howard: Explaining why low-code or no-code ML tools matter
Implications: The near future likely favors practical ML: better transfer learning, simpler training, more open tooling, and domain-specific models. But generative text will also amplify abuse, making reproducibility, open datasets, and non-code interfaces increasingly important.