Emergency Pod: Reinforcement Learning Works! Reflecting on Chinese Reasoning Models DeepSeek-R1 and Kimi k1.5

Episode Summary

Executive Summary: The episode analyzes DeepSeek R1 and Moonshot’s Kimi reasoning model as evidence that simple reinforcement learning on top of strong base models can rapidly produce highly capable, sometimes strange reasoners. The host argues this narrows the West-China AI gap, challenges compute-governance assumptions, accelerates open-source diffusion, and signals that superhuman reasoning may be arriving faster than expected.

Main Topics: DeepSeek R1 as a breakthrough reasoning model (Priority: 5/5): The host frames R1 as a top-tier reasoning model built from DeepSeek’s efficient V3 base model using reinforcement learning, with performance approaching OpenAI’s o1 while remaining open source and cheap to run at scale. Pure reinforcement learning and emergent reasoning (Priority: 5/5): R1-Zero/R10 is presented as the key scientific result: a base model trained with only rule-based accuracy rewards (no human demos or preference data) develops longer chains of thought, reflection, and self-correction on its own. Productizing reasoning: R1 versus raw reasoning behavior (Priority: 4/5): The episode contrasts the rough, sometimes unreadable behavior of the pure RL model with the more human-friendly R1 product model, which adds supervised fine-tuning, formatting constraints, and broader assistant training. Chinese AI progress, open source, and strategic implications (Priority: 5/5): The host argues the releases show China is now at the global frontier, that open-sourcing may be strategic or de-escalatory, and that Western narratives of an AI war are misguided and potentially irresponsible. Limits of compute governance and frontier gating (Priority: 4/5): Because strong reasoning models can be trained and distilled efficiently, the episode questions whether compute restrictions can meaningfully prevent frontier capability diffusion or harmful applications. Distillation into smaller models and local use (Priority: 4/5): DeepSeek’s reasoning abilities are shown to transfer into smaller Llama and Qwen models, making high-level reasoning available on laptops and via cheaper APIs, which will pressure incumbent business models. Comparison with OpenAI, Google, and Anthropic (Priority: 4/5): The host repeatedly compares DeepSeek/Kimi to o1/o3, Gemini Flash Thinking, and Anthropic’s internal work, concluding that several leading labs appear to share the same simple, scalable reasoning paradigm.

Key Arguments: A very simple RL setup—rewarding correct answers and penalizing wrong ones—can bootstrap strong reasoning without human preference data or process supervision. Longer chains of thought emerge naturally during training because they improve answer accuracy; the curve for thinking length keeps rising and does not appear to have flattened. Reasoning gains can be distilled into smaller models, making PhD-level or near-PhD-level performance accessible on local hardware. The new models shrink the practical and strategic gap between U.S. and Chinese frontier AI labs. Compute restrictions alone are unlikely to prevent advanced AI capabilities from proliferating, because efficient training and inference lower the barrier dramatically. The “AI war” framing is rejected as dangerous and unnecessary; the host argues for de-escalation and broader diffusion of beneficial AI. Censorship appears to be layered on top of the model in product deployments, rather than baked into the base weights, likely to preserve the model’s internal coherence. Reasoning models may become powerful but harder to interpret, with odd behaviors such as language switching, steganographic tendencies, and human-like “aha moments.” OpenAI and other Western labs likely use similar reasoning paradigms, suggesting a convergent technical direction rather than a unique Chinese approach. The long-term path toward AGI looks increasingly concrete: reasoning, memory, multimodality, and scalable RL all now have credible demonstrations.

Data Points: DeepSeek V3 pretraining cost: single-digit millions / about $6 million - Used as the base model before reasoning RL; highlighted as far cheaper than Western frontier training runs. DeepSeek V3 size: 671 billion parameters total, about 37 billion active - Mixture-of-experts base model underlying R1/R1-Zero. Context window: 128,000 tokens - Approximate context size for the model discussed, compared with larger Gemini contexts. RL training steps: 8,000+ steps - Chain-of-thought length and performance improved over the course of RL training. Thinking tokens growth: ~500 to ~10,000 average thinking tokens - The reasoning trace length expanded substantially during training. Benchmark improvement (pass@1): 15% to 70%+ - R1-Zero/R10 single-shot performance on a reasoning benchmark rose dramatically during RL. Benchmark improvement (pass@16): 25% to ~85% - Majority-vote performance across 16 samples improved even more strongly. R1 API cost on Hyperbolic: $2 per million tokens - Illustrates the low-cost inference advantage of DeepSeek R1. OpenAI o1 output cost: $60 per million output tokens - Used as a comparison point showing roughly an order-of-magnitude cost gap. Distilled model benchmark: Llama 70B distilled ~65% on GPQA Diamond - Shows strong reasoning distilled into smaller open models. OpenAI o1-mini GPQA Diamond: ~75% - Used as a comparison for distilled model capability. GPT-4.0 GPQA Diamond: ~50% - Benchmarked as a reference point below the distilled open model. Reasoning/task mix in second phase: 600,000 reasoning examples and 200,000 non-reasoning examples - Used in the broader supervised fine-tuning / reinforcement-learning phase for the R1 product model. Model availability: R1 weights open source; Kimi model not yet publicly available at time of discussion - Important difference between the two China releases.

Pivotal Quotes: "we've now seen no less than Alex Wang, the CEO of Scale AI, take out a full page ad in the newspaper calling the current situation an AI war, which I honestly totally hate and think is wildly irresponsible." — Host: Critique of the escalating geopolitical framing around AI competition. "the simplest possible reinforcement learning setup, where you just give it problems and reward it for being right, works and it makes huge, huge leaps in reasoning capability." — Host: Core thesis of the episode on why R1 matters scientifically. "this is the first model that has come out that I think merits the same level of drop everything and just immerse yourself in this model and really try to understand it" — Host: Call to researchers and listeners to experiment directly with R1/R10.

Implications: Reasoning-model progress now looks rapid, cheap, and widely reproducible, accelerating open-source diffusion while weakening arguments that compute controls alone can safely gate frontier AI. Expect pressure on incumbent pricing, more Chinese-U.S. convergence in methods, and faster movement toward broadly capable AI.