#490 – State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI

Episode Summary

Executive Summary: Lex Friedman’s conversation with Sebastian Raschka and Nathan Lambert maps the AI landscape across model quality, scaling laws, open vs. closed models, post-training breakthroughs, and the future of software, education, and robotics. The core takeaway is that progress remains rapid, but it is increasingly driven by systems, data, inference-time compute, and product design—not just bigger pretraining runs.

Main Topics: State of the AI race: US vs. China and model leadership (Priority: 5/5): The guests argue there is no permanent winner, but current leadership shifts by domain: OpenAI/Google/Anthropic dominate consumer and coding use cases, while Chinese labs dominate open-weight releases and are influencing global adoption through accessibility and licensing. Scaling laws, pre-training, and compute economics (Priority: 5/5): They distinguish pre-training, mid-training, post-training, and inference scaling, arguing that scaling still works but the best gains now often come from post-training and inference-time compute, while pre-training is increasingly constrained by cost and deployment economics. Post-training and RLVR as the major breakthrough (Priority: 5/5): Reinforcement learning with verifiable rewards (RLVR) is presented as the big technical unlock of 2025, especially for reasoning, tool use, math, code, and step-by-step problem solving, with RLHF still important for style and usability. Open-weight ecosystems and the rise of Chinese labs (Priority: 4/5): DeepSeek catalyzed a wave of open-weight competition in China (Qwen, Kimi, Minimax, Z.ai, etc.), while US/European open efforts (AI2 OLMO, Hugging Face, NVIDIA, Stanford) are trying to close the gap and preserve scientific openness. AI coding, agents, and the future of software (Priority: 5/5): Both guests say AI is already transforming programming through coding assistants, repo-aware agents, and fast iterative debugging. They think full autonomy is still far off, but software creation will become increasingly spec-driven and human-in-the-loop. Education, learning, and human agency (Priority: 4/5): They repeatedly emphasize that struggle is part of learning and that LLMs should augment—not replace—deep study. The advice is to build from scratch, read narrowly, and use models as a second-pass tutor or research partner. Safety, social impact, and long-term societal shifts (Priority: 4/5): The discussion expands to AI safety, mental health, job displacement, misinformation, and the need for trust, verification, and human agency. They expect more slop, more consolidation, and more value on physical, in-person, and human-authored experiences.

Key Arguments: Model leadership is fragmented: today’s winner depends on use case, brand, and product quality rather than raw intelligence alone. Chinese open-weight models are strategically important because they drive global adoption, influence, and pressure US labs to release better open models. Pre-training still matters, but the biggest marginal gains now often come from better data quality, post-training, tool use, and inference-time compute. RLVR is a major step function because it teaches models to reason through verifiable tasks like math, coding, and tool use. RLHF remains essential for tone, safety, and user experience, but it is less scalable for raw capability than RLVR. The apparent architectural changes from GPT-2 to today are often incremental; most progress comes from training recipe, systems, and compute scaling rather than wholly new architectures. AI coding will not eliminate humans soon, but it will drastically reduce the amount of human labor needed and shift work toward system design and specification. Learning still requires struggle; if users rely on LLMs too early, they may lose the opportunity to build expertise and taste. Open-source/open-weight models matter for education, research reproducibility, and national competitiveness because they let more people inspect, modify, and build on frontier techniques. The biggest unresolved problems are tool use, continual learning, memory, long context, and robust evaluation without contamination.

Data Points: DeepSeek R1 timing: January 2025 - Used as the reference point for the so-called DeepSeek moment that surprised the AI field. OpenAI average compensation: Over $1 million in stock per employee per year - Mentioned to illustrate the economic power and competitiveness of frontier labs. AI2 OLMO-3 pretraining cluster cost: About $2 million - Estimate given for renting cluster time and dealing with training issues and multiple runs. DeepSeek pretraining cost (cloud market rates): About $5 million - Cited as a famous low-cost pretraining figure for a frontier model. OpenAI/closed lab pretraining scale: Around 1 trillion parameters (rumored lower later) - Used to illustrate that frontier-scale pretraining is extremely expensive and may be getting more efficient. Pretraining data scale: Trillions of tokens - Discussed as the typical size of modern pretraining corpora. Quen data scale: Up to 50 trillion tokens - Example of very large documented training data scale for a leading open-weight model family. Closed lab data scale rumors: Up to 100 trillion tokens - Mentioned as a rumored scale for some proprietary frontier models. Consumer chatbot usage: 90,000+ businesses - Quo/OpenPhone was described as serving over 90,000 businesses. Finn customer service adoption: 6,000+ customer service leaders - Used to describe the scale of the AI customer-service product adoption. Survey size: 791 professional developers - Referenced in the coding/AI adoption discussion, with professional defined as 10+ years of experience. Developer code generation: Around 50% or more of shipped code for many respondents - Survey result discussed as evidence that AI-generated code is already common in production. Model context growth: From ~8K to ~32K context in one example - Used to illustrate that context extensions can require significant extra compute. Long-context roadmap: Potentially 2M–5M context this year - Speculative forecast for continued context expansion. RLVR example improvement: 15% to 50% accuracy in ~50 steps - Nathan described training Qwen 3 base on Math-500 with RLVR as a striking example of rapid capability unlock. Anthropic legal settlement: $1.5 billion owed to authors - Mentioned in the discussion of copyrighted book data and legal risk around training corpora. AI2 NSF grant: $100 million over 4 years - Cited as a major U.S. open-model funding effort. Magic number for model training: 10^25 FLOPs - Referenced as a rough benchmark scale from U.S. government discussions about training frontier models. Work culture shorthand: 996 = 9 a.m. to 9 p.m., 6 days a week - Used to describe intense AI lab/startup work culture and burnout risk.

Pivotal Quotes: "I don't think nowadays, 2026, that there will be any company who is, let's say, having access to a technology that no other company has access to." — Sebastian Raschka: On why model technology diffuses quickly across labs and talent moves between organizations. "The biggest one from 2025 is learning this reinforcement learning with verifiable rewards." — Nathan Lambert: Summarizing what he sees as the central post-training breakthrough driving reasoning models. "You're trying to always build an LLM that's going to fit on one GPU." — Sebastian Raschka: Explaining the educational value of building models from scratch and keeping the implementation tractable.

Implications: AI progress is broadening from raw scaling to systems, tools, and product integration. Expect stronger coding agents, more open-weight competition, better specialized models, and continued pressure on education, labor, and trust/verification.