Episode Summary
Executive Summary: Rashab Agarwal describes his path into reinforcement learning, his shift toward research on building upon existing agents, and the backstory of his NeurIPS 2021 award-winning paper on statistical uncertainty in RL evaluation. The conversation centers on how few-seed benchmarks can produce misleading conclusions, and proposes practical alternatives: performance profiles, confidence intervals, and better aggregate metrics to improve rigor and reproducibility.
Main Topics: Path into reinforcement learning: Agarwal explains how Atari results, game-playing, and early research failures drew him into RL and kept him there despite exploring other areas. Research on building on existing agents: He argues that real-world RL systems usually improve on prior policies rather than starting from scratch, and that research should reflect this incremental setting. From benchmark debugging to statistical precipice: The award-winning paper emerged from unexpectedly high variance in Atari 100K results, where increasing random seeds revealed instability and bias in published comparisons. Why few-seed RL results can be misleading: Agarwal explains that median-based benchmark reporting can hide large uncertainty, produce biased estimates, and even flip method rankings depending on seed count. Proposed evaluation tools and metrics: The paper recommends performance profiles, confidence intervals, probability-of-beating-baseline measures, and interquartile mean as better ways to summarize RL results. Incentives and reproducibility in ML research: He discusses the tension between rigor and publishability, and argues that conferences and tooling should make uncertainty reporting easier and more standard.
Key Arguments: RL appealed to Agarwal because agents learn from their own mistakes, unlike static supervised learning, and because game-like environments are motivating. Research should not assume tabula rasa training; most practical systems build on existing policies, models, or data and should be evaluated that way. Atari 100K exposed severe seed sensitivity: the same method could look weak or state-of-the-art depending on which seeds were used. Using only a few random seeds can create biased estimates of benchmark performance; the paper found the bias could be large enough to materially change conclusions. Most RL benchmark papers report point estimates that hide uncertainty; researchers should report intervals around results, not just single numbers. Performance profiles are useful because they expose the full distribution of task/run scores and let readers recover mean, median, and percentiles. Confidence intervals are preferable to p-values for this setting because they communicate effect size and uncertainty more directly and avoid arbitrary significance thresholds. An interquartile mean can be a better aggregate than mean or median because it balances robustness with sensitivity to all tasks. Better evaluation can improve reproducibility and prevent overstating progress, even if it sometimes makes results look less impressive. Community adoption is helped by low-friction tooling, so the team released an open-source library to generate plots and statistics from raw results.
Data Points: Atari 100K training budget: 100K interactions - The benchmark used in the paper trains agents for roughly 100,000 environment interactions. Human gameplay equivalent: ~2–3 hours - 100K interactions correspond to about two to three hours of human gameplay on Atari 100K. Seeds tried during investigation: 3 to 30 seeds - Agarwal increased the number of random seeds repeatedly because results remained highly variable. Bias in aggregate estimates: up to 30% - The paper found that the estimator bias from limited-seed evaluation could be as large as 30% of performance. Hardware/evaluation correlation: approximately -0.2 to 0.2 - He reported very low correlation between two evaluations of the same 100 seeds after a GPU/randomness change. Atari benchmark scale: 57 tasks - He referenced Atari 56/57-task style benchmarks when discussing aggregate reporting and tables. Compute cost for one result: ~1000 GPU hours - For Atari 56/57-style evaluation across tasks and seeds, he estimated roughly 1000 GPU hours for a single aggregated result. Wall-clock evaluation time: 3–4 days - Even with parallelization, a single multi-task result with five seeds could take several days. OpenAI Dota training time: ~10 months - Used as an example of why one cannot restart from scratch in expensive, changing environments. Additional retraining cost if restarted: 5–6x more time - He cited OpenAI’s claim that training from scratch would have taken five to six times longer. ImageNet seed variation: ±1% - He cited an example that even ImageNet training can vary by about one percentage point depending on seed.
Pivotal Quotes: "Your case was interesting because you have done a lot of research and you have failed." — Rashab Agarwal quoting Geoff Hinton: Hinton’s comment on why early failures can be a positive indicator for a researcher. "We should be reporting interval estimates of things." — Rashab Agarwal: Core takeaway from the paper: report uncertainty, not just point estimates. "Why are we doing that in research?" — Rashab Agarwal: His critique of the common practice of starting RL experiments from scratch instead of building on existing agents.
Implications: RL and broader ML benchmarks should move from single-number claims to uncertainty-aware reporting. Better statistics, tooling, and conference norms could reduce false progress, improve reproducibility, and make comparisons more honest.