Episode Summary
Executive Summary: Aditi Raghunathan argues that current LLM progress is overly benchmark-driven and often fails at adaptability, safety, and creative problem solving. The conversation covers her lab’s findings that overtrained models can become harder to fine-tune, unlearning is brittle because knowledge isn’t neatly localized, and creativity may require training objectives that encourage structured exploration rather than pure next-token prediction.
Main Topics: Benchmark performance vs real-world robustness (Priority: 5/5): The discussion opens with a critique of optimizing mainly for benchmark scores. Raghunathan argues that models can look excellent on static tests yet fail under slight distribution shifts or deployment conditions, revealing a gap between measured capability and practical reliability. Catastrophic overtraining and fine-tuning brittleness (Priority: 5/5): A core research result is that more pretraining can eventually make models worse starting points for fine-tuning or post-training. She describes U-shaped behavior in which additional compute improves some metrics but eventually hurts adaptability, including under quantization. Data-to-parameter ratio and model updateability (Priority: 4/5): The conversation explores how token-to-parameter balance influences whether a model remains flexible. Larger models can absorb more tokens before becoming brittle, but smaller overtrained models may be less adaptable and therefore less useful as bases for downstream tasks. Unlearning and the limits of localized knowledge (Priority: 5/5): Raghunathan explains why unlearning harmful or private information is hard: current models do not naturally isolate facts into neat neurons or subspaces. Her lab’s memorization sinks work tries to enforce a training bias that separates shared knowledge from document-specific memorization. Creativity beyond next-token prediction (Priority: 5/5): The final major theme is that LLMs can mimic creativity when constrained, but struggle with open-ended, structured novelty. Her work proposes alternatives like multi-token prediction and random-prefix conditioning to encourage global planning and diverse, structured generation. Designing better training objectives and inductive biases (Priority: 4/5): Across overtraining, unlearning, and creativity, the common thread is that training objectives shape what models learn and what they can later do. Raghunathan argues for architectural and objective-level interventions rather than relying on post hoc fixes.
Key Arguments: Benchmark optimization alone is insufficient because it does not measure whether a model remains useful under realistic distribution shifts or deployment changes. More pretraining can become harmful: in realistic settings, later checkpoints can be strictly worse starting points for fine-tuning than earlier ones. The ratio of tokens to parameters is an important proxy for when brittleness emerges, though the exact threshold depends on model size and downstream distribution. Fine-tuning performance depends on how much the model must move; if distributions are close, the overtraining effect weakens or disappears. Current unlearning methods fail partly because knowledge is not naturally disentangled into isolated neurons or subspaces in standard training. Memorization sinks attempt to create that disentanglement by design, assigning document-specific information to dedicated subspaces while preserving shared information elsewhere. Creativity should be understood as structured exploration: finding novel but meaningful connections, not just generating random outputs. Next-token prediction encourages local completion rather than global leaps of thought, so objectives like multi-token prediction or diffusion-style masking may better support creative reasoning. Randomness should be introduced at the beginning of generation, not at every token, so the model can commit to a diverse idea and then follow it consistently. Better base-model design could improve downstream fine-tuning, unlearning, retrieval use, and RL because these all depend on the model’s starting point and sampling behavior.
Data Points: ICML paper award: Outstanding paper award - The paper discussed, "Roll the Dice and Look Before You Leap," won an outstanding paper award at ICML 2025. Pretraining scale example: 3 trillion tokens - She cited an OLMO 1B checkpoint trained on 3 trillion tokens that became worse for downstream fine-tuning than a model trained on fewer tokens. Model size example: 1 billion parameters - Used as the small-model example in the catastrophic overtraining discussion. Fine-tuning threshold example: ~2.5 trillion to 3 trillion tokens - She suggested that for the 1B model, checkpoints beyond roughly this range may no longer be desirable as fine-tuning starting points. Quantization behavior: U-curve - She described quantization experiments where more pretraining eventually led to worse post-quantization performance after an initial improvement period.
Pivotal Quotes: "does that actually solve the task if we just test it in a slightly different way that is also meaningful from a deployment perspective?" — Aditi Raghunathan: Framing the core critique of benchmark-centric evaluation. "more compute is actually kind of make your model worse, not just that it saturates, but it's actively worse." — Aditi Raghunathan: Describing catastrophic overtraining and degraded fine-tuning utility. "we really want this sort of structured exploration from these models." — Aditi Raghunathan: Summarizing her view of creativity as a balance between novelty and constraint.
Implications: For practitioners, base-model choice should consider adaptability, not just benchmark score. For researchers, the field needs training objectives and architectures that improve fine-tuning, unlearning, and structured creativity. For industry, this suggests future gains may come from better inductive biases, not only bigger models.