429: 2020's Biggest Data Science Breakthroughs

Episode Summary

Executive Summary: Kirill Aramenko and John Crohn review 2020’s biggest technical breakthroughs: AlphaFold’s leap in protein structure prediction, the accelerating role of GPUs in deep learning, and GPT-3’s transformer-based language capabilities. They balance optimism about AI’s scientific and practical impact with caution about limits, compute costs, and current models still lacking true reasoning.

Main Topics: AlphaFold and protein folding: John explains how DNA encodes amino-acid chains that fold into 3D protein structures, why predicting that fold is a long-standing biology challenge, and how AlphaFold 2 dramatically improved CASP performance while enabling new scientific applications. GPUs, TPUs, and the compute arms race: The discussion covers why GPUs are essential for deep learning, recent memory increases in consumer and enterprise hardware, when it makes sense to buy on-premise versus use cloud compute, and how model parallelism across multiple GPUs works. GPT-3 and transformer architectures: They break down GPT-3’s generative pre-training and self-attention, contrast its massive scale with earlier models, and explore applications like code generation, translation, and text completion alongside its limitations. AI as a scientific accelerator: Both speakers emphasize that machine learning is moving beyond commercial use cases into core research domains such as biology, chemistry, and medicine, with AlphaFold as a strong proof point. Compute, scale, and cost tradeoffs: John gives practical examples of GPU pricing, memory constraints, and retraining frequency to show how infrastructure decisions affect real-world ML deployment and economics. Limits, skepticism, and responsible expectations: While Kirill is highly optimistic about AI’s trajectory, John stresses that current models are not reasoning agents and that progress toward general intelligence will require new approaches beyond larger transformers.

Key Arguments: AlphaFold is a landmark because it solved a 50-year-old protein-folding challenge with machine learning, turning a massive biology bottleneck into a computational problem. The jump in CASP scores shows stagnation followed by breakthrough: progress from 40 to 60 to 92.4 indicates a major step change in capability. GPUs matter because deep learning depends on highly parallel matrix operations; more memory and compute directly expand model size and training feasibility. For some workloads, buying local GPU hardware is cheaper than cloud compute if models are retrained frequently; for very large, short-lived jobs, cloud scaling is better. Transformer models use self-attention to capture long-range dependencies in text, enabling better language generation and sequence understanding than older recurrent approaches. GPT-3’s impressive outputs are real but misleadingly human-like; it predicts likely next words rather than reasoning or understanding. AI is increasingly useful in science and could help unlock discoveries in biology, chemistry, physics, and climate-related applications. Current AI progress is limited by data, compute, and architecture; future leaps may require combining differentiable learning with non-differentiable memory or knowledge systems.

Data Points: CASP best score (2008): 40 - John describes the state of protein-fold prediction before recent breakthroughs. CASP best score (2016): 40 - Shows a period of little progress in protein structure prediction. CASP score for AlphaFold in 2018: 60 - DeepMind’s AlphaFold made a major leap in the 2018 competition. CASP score for AlphaFold 2 in 2020: 92.4 - AlphaFold 2 achieved near-experimental accuracy in protein structure prediction. Moderately difficult protein score vs other algorithms: 90 vs 75 - AlphaFold 2 averaged about 90 on moderately difficult proteins, while other algorithms averaged 75. Known protein structures: 170,000 - The current catalog of experimentally known structures used for training and validation. Estimated known proteins in nature: 180 million+ - John notes this as the approximate broader universe of proteins still to be characterized. Potential protein shape space: 10^300 possible conformations - Used to illustrate why brute-force search is infeasible for protein folding. Training hardware for AlphaFold 2: 128 TPUs - John says the model was trained on a large cluster over several weeks. Training data size for AlphaFold 2: 170,000 known proteins - Used as supervised structure data for protein folding prediction. Consumer GPU memory (GTX 1080 Ti): 11 GB - John’s earlier deep-learning setup used this card as a baseline example. Consumer GPU memory (RTX 3090): 24 GB - Newer consumer GPU with more than double the memory of the 1080 Ti. Enterprise GPU memory (A100): 80 GB - NVIDIA’s high-end GPU aimed at large-scale training. DGX workstation GPU memory: 320 GB total - Four A100s in one DGX box, described as a “fridge” workstation. DGX workstation price: $200,000 - John estimates the cost of the four-GPU DGX system. Build cost for a personal rig: $2,500-$4,000 - John compares home-built multi-GPU rigs versus cloud usage. Payback period for on-prem setup: About 3 months - Untapped’s hardware investment reportedly paid for itself quickly due to retraining frequency. BERT parameter count: 340 million - Used to illustrate scale in transformer language models. DistilBERT parameter count: 66 million - A smaller transformer with comparable performance. GPT-2 parameter count: 1.5 billion - Predecessor to GPT-3 and a major scale-up from BERT. GPT-3 parameter count: 175 billion - Highlights the massive jump in model size. Open-source comparable model: 8.3 billion parameters - John cites Facebook’s Megatron as the largest open alternative mentioned. GPT-3 training compute: 1,000 petaflop/s-days - Illustrates the extreme compute requirements to train the model. GPT-3 output pattern: Next-word probability prediction - John explains the model as statistically generating the most likely next token rather than reasoning.

Pivotal Quotes: "You can have the biggest impact by creating products, data-driven products in particular, and automate things, make people's lives easier, have a good impact on the environment." — John Crohn: John explains why he shifted from neuroscience toward commercial data science and product development. "AlphaFold 2 obtained a score of 92.4." — John Crohn: John highlights the scale of the breakthrough in protein folding compared with prior years. "It doesn't have an understanding of anything. It's just a large number of model weights." — John Crohn: John tempers enthusiasm about GPT-3 by stressing its lack of true reasoning or comprehension.

Implications: Listeners should see AI as a rapidly maturing tool for science and business, but not as human-level intelligence. Progress will hinge on compute, data, and better architectures, with major near-term impact likely in biology, automation, and language tooling.