870: OpenAI’s “Deep Research”: Get Days of Human Work Done in Minutes

Episode Summary

Executive Summary: John Crohn highlights OpenAI’s Deep Research as a major leap in AI-assisted literature review and synthesis: it can break complex questions into sub-tasks, search and synthesize hundreds of sources, cite claims, and adapt its research path. He argues it outperforms competing offerings from Google and Perplexity, points to strong benchmark results, and shows its practical value through a real workshop-planning example.

Main Topics: What Deep Research is and how it works: Explains that Deep Research automates deep-dive research by decomposing queries, browsing the web iteratively, synthesizing findings, and producing cited reports with images and graphs. Benchmark performance on Humanities Last Exam: Emphasizes Deep Research’s standout performance on a difficult new benchmark, suggesting AI is making real progress on tasks once thought resistant to automation. Practical workflow benefits and personal use case: Shares a detailed example of using Deep Research to create a workshop syllabus, title, and abstract, saving hours and improving output quality. Comparison with Google and Perplexity: Compares OpenAI’s version with competitors: Google’s research is broad and structured, Perplexity is fast and free, but OpenAI is portrayed as more methodical and reliable. Limitations, hallucinations, and cost: Notes the risk of hallucinations and misinformation, and highlights the high price and query limits of OpenAI Pro as the main tradeoff. Broader implications for data science and autonomous agents: Argues that deep research will democratize expert-level research and eventually integrate with AI agents to automate more knowledge work.

Key Arguments: Deep Research automates what would normally take human researchers hours or days, making it a powerful productivity tool for literature reviews and synthesis. Its multi-step reasoning and iterative search process make it more capable than standard LLM chat, especially for complex, multi-source tasks. OpenAI’s reinforcement-learning training on browsing and reasoning tasks likely explains its superior benchmark results and practical usefulness. The Humanities Last Exam benchmark suggests AI is beginning to tackle previously difficult academic-style reasoning tasks, with Deep Research far ahead of other models mentioned. A concrete workshop-planning example shows that the tool is not just impressive on benchmarks but also directly useful for real professional work. Competitor products from Google and Perplexity are useful, but OpenAI’s approach is described as more thorough, transparent, and higher quality for mission-critical research. The main downsides are hallucination risk and cost, though the speaker believes these issues are manageable and likely to improve over time.

Data Points: Episode number: 870 - The podcast episode is introduced as episode 870 on Deep Research. Humanities Last Exam scope: 3,000 questions across 100+ subjects - Described as a comprehensive benchmark spanning many domains from rocket science to linguistics. OpenAI o1 accuracy on Humanities Last Exam: 9.1% - Used as a comparison point for earlier reasoning models. DeepSeek R1 accuracy on Humanities Last Exam: 9.4% - Another comparator model on the same benchmark. OpenAI Deep Research accuracy on Humanities Last Exam: 27% - Presented as the standout result, far above competing models. GPT-4.0 accuracy on Humanities Last Exam: 3% - Cited as a non-reasoning baseline model performance. Claude 3.5 Sonnet accuracy on Humanities Last Exam: ~4% - Mentioned as another low-scoring comparator. Gemini Thinking accuracy on Humanities Last Exam: ~6% - Google’s effort in the comparison table. OpenAI o3 Mini accuracy on Humanities Last Exam: 13% (high compute setting) - Referenced as a stronger reasoning model before Deep Research. Deep Research research session: 3 minutes and 8 sources - The speaker’s workshop-syllabus example used this amount of time and source breadth. OpenAI Pro subscription price: $200/month - The speaker says he pays for Pro to access Deep Research and finds it worth it. Deep Research query limit: 100 queries/month - Described as roughly a little over three queries per day.

Pivotal Quotes: "It remarkably well automates deep dive literature reviews and synthesizes hundreds of online sources into a coherent, well-cited report." — John Crohn: Defines the core value proposition of OpenAI Deep Research. "OpenAI's deep research absolutely crushing all of its competitors" — John Crohn: Summarizes the speaker’s view of Deep Research’s benchmark advantage. "The implications are profound." — John Crohn: Used repeatedly to emphasize the broader impact on research, productivity, and AI-driven automation.

Implications: Deep Research could reshape research-heavy work by making expert-level synthesis fast and accessible. For listeners, it signals a near-term future where AI assistants handle much of the information-gathering burden, though cost and verification still matter.