⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data

Episode Summary

Executive Summary: The discussion centers on OpenAI’s decision to move beyond SWE-Bench Verified as a coding benchmark because it is now saturated and contaminated, making it less useful for measuring frontier model progress. Mia and Olivia explain how Verified was created, why its limits emerged, and why newer benchmarks like SWE-Bench Pro better capture harder, more diverse, less contaminated coding and agentic tasks. They also connect coding evals to OpenAI’s preparedness framework and broader real-world impact measurement.

Main Topics: Why SWE-Bench Verified is being retired as a primary benchmark (Priority: 5/5): The speakers argue that Verified once served as a North Star for coding progress, but frontier models have now pushed it into saturation and contamination, so small score gains no longer reflect real capability improvements. How SWE-Bench Verified was built and validated (Priority: 5/5): Olivia and Mia describe the extensive human-data effort behind Verified: expert software engineers reviewed tasks, specs, tests, and solutions to produce a curated 500-task benchmark with stronger quality control. Contamination and benchmark fairness problems (Priority: 5/5): They explain that because tasks came from open-source repos, models may have seen them in training data, and some failures were due to unfair or overly narrow tests rather than model weakness. Why SWE-Bench Pro is a better next benchmark (Priority: 4/5): SWE-Bench Pro is described as harder, more diverse, and less contaminated, with larger tasks and more headroom for measuring real coding ability beyond shallow issue-solving. What an ideal future coding benchmark should measure (Priority: 4/5): The conversation shifts to evaluating design taste, maintainability, long-running tasks, end-to-end product creation, and realistic workflows that go beyond short GitHub issue patches. Preparedness framework and model autonomy (Priority: 4/5): OpenAI’s preparedness framework is framed as the reason coding evals matter for tracking frontier risks, especially research automation and model autonomy, which require more advanced and realistic measurements. Future directions: real-world and monetary proxies (Priority: 3/5): They discuss alternative benchmarks based on time, money, task complexity, and real-world usage, while emphasizing that public, reproducible evals still matter for the field.

Key Arguments: SWE-Bench Verified was valuable historically, but it no longer measures frontier coding capability well because performance is now near-saturated. The benchmark is contaminated because many tasks originate from open-source repositories that likely appeared in training data. A significant portion of the remaining failures came from unfair tests, overly narrow specifications, or hidden implementation assumptions rather than actual model deficiencies. SWE-Bench Pro is preferable because it is harder, more diverse, and shows less evidence of contamination. Future coding evals should assess realistic software engineering work: longer tasks, design decisions, maintainability, and end-to-end product creation. Human review remains essential for high-quality evals, especially when correctness depends on domain knowledge and task context. Public benchmarks are still important because they allow the field to compare progress quickly and consistently without expensive human grading every time. Preparedness work needs coding benchmarks because coding is a core component of model autonomy and research automation risk tracking.

Data Points: Verified benchmark task count: 500 tasks - The curated SWE-Bench Verified set was described as a ~500-task benchmark created after extensive review. Expert reviewer count: almost 100 - OpenAI reportedly hired nearly 100 real-world software engineers to review benchmark tasks and tests. Independent reviews per task: 3 reviews - Tasks were independently reviewed by multiple experts to assess fairness and correctness. Benchmark saturation estimate: ~80% - The speakers say frontier models are now around the point where further gains on Verified are not very meaningful. Tasks suitable for less than an hour: roughly 90% - They say most SWE-Bench Verified problems were estimated to take an expert software engineer less than an hour. Hard contamination case solved by GPT-5.2: 31 problems - They mention GPT-5.2 solving 31 tasks that were believed to be very hard to solve without contamination. Human-intensive review scope: over half - In the deeper dive into failed tasks, over half of investigated problems had some issue, often with test fairness or specification narrowness. Preparedness categories tracked: 3 - OpenAI’s preparedness framework is described as tracking bio-risk, cybersecurity, and research automation/model autonomy. White-collar professions in GDPval: 15-16 - GDPval is cited as covering roughly 15 to 16 white-collar professions.

Pivotal Quotes: "SweetBench Verified has been one of the North Star coding benchmarks that the field has looked at to measure coding progress. But recently, we've seen that progress has kind of stalled." — Olivia: Explaining why OpenAI is moving away from SWE-Bench Verified as a primary measure. "The eval is effectively saturated and also highly contaminated." — Olivia: Core thesis of the blog post and the reason Verified is no longer trusted as a progress metric. "We think that this is because the eval is effectively saturated and also highly contaminated." — Olivia: Reinforcing the main argument for shifting to SWE-Bench Pro and other benchmarks.

Implications: Listeners should expect OpenAI and the field to de-emphasize SWE-Bench Verified and move toward harder, more realistic, less contaminated evals. The broader trend is toward measuring real-world coding impact, autonomy, and agent usefulness rather than small benchmark gains.