Episode Summary
Executive Summary: Josh Tobin argues that the next wave of useful AI agents will come less from hand-built workflows and more from models trained end-to-end to perform multi-step tasks, recover from errors, and use tools effectively. He discusses OpenAI’s agentic products—Deep Research, Operator, and Codex CLI—as early examples, and emphasizes that trust, context, and tool governance are the biggest remaining hurdles.
Main Topics: From hand-built workflows to trained agents (Priority: 5/5): Tobin contrasts 2023–2024 agent building approaches—human-designed multi-step workflows with LLM calls—with models trained end-to-end via reinforcement learning to perform real processes more robustly. Why foundation models changed the ML infrastructure market (Priority: 4/5): He explains that pre-ChatGPT infrastructure assumed every company would train its own models, but foundation models made off-the-shelf capability far cheaper and faster, reducing the need for bespoke training stacks. OpenAI’s agentic product portfolio (Priority: 5/5): The conversation covers Deep Research, Operator, and Codex CLI as examples of agents that act over longer time horizons and interact with the real world or local codebases. Context, tools, and reasoning in coding agents (Priority: 4/5): Tobin describes Codex CLI as a context-light 'superhuman intern' that explores codebases using standard command-line tools, and argues that tool exposure plus reasoning are central to agent performance. User trust and high-stakes actions (Priority: 5/5): A major gap in the field is determining how agents should request permission, especially for sensitive actions like spending money or using credentials, and how risk should be shared among users, model providers, and platforms. The future of software development and learning (Priority: 4/5): He predicts more code will be written by AI, while human engineers shift toward product judgment, validation, architecture, and reviewing model output; learning with ChatGPT, O3, and Deep Research will reshape skill acquisition. MCP and the agent tool ecosystem (Priority: 3/5): Tobin sees protocols like MCP as essential for exposing tools to models in a way that supports both training and real-world use, enabling more capable and customizable agents.
Key Arguments: General-purpose foundation models made it impractical for most businesses to train their own models; the valuable work now happens on top of them. The main limitation of early agents was compounding error across many steps, not just per-step intelligence. Models trained end-to-end with reinforcement learning can learn to recover from failures better than hand-designed workflows. Agentic products are already useful today, especially for deep research, coding help, and long-running browser tasks, even if they are not fully autonomous yet. Large models help with agentic tasks because they generalize better and can dynamically allocate reasoning effort to harder steps. Tool access is a prerequisite for useful agents, but trust boundaries around sensitive tools are still unresolved. Software development is shifting from manually writing every line of code to directing, validating, and supervising AI-generated work. Open source and ecosystem integration are important for Codex CLI because they let the community extend it into CI, issue triage, and other workflows.
Data Points: Time since last conversation: over 5 years - Sam notes that he last spoke with Josh in Vancouver around the Neurops era. Gantry founding year: 2019 - Tobin says he left OpenAI in 2019 and co-founded the ML infrastructure startup Gantry. OpenAI return date: September (year not specified in transcript) - He says he rejoined OpenAI in September and now leads the agents research team. Deep Research use cases: business, scientific research, travel, shopping, programming - Tobin lists common ways people use Deep Research. GitHub stars for Codex CLI: more than 20,000 - He cites adoption metrics for the open-source coding agent. Codex CLI contributors: around 100 - He notes community contributions to the open-source project. Workflow accuracy example: 90% per step; 10 steps compounds failure - Used to illustrate why multi-step agents break down even when individual steps are strong. Alternative per-step accuracy example: 95% per step - Another example of compounding error in multi-step workflows. Model version mentioned: O3 - He says O3 is better than older models at following complex instructions and using tools over multi-step trajectories. Model version mentioned: O1 - Compared with O3, O1 is described as reasoning for roughly the same amount of time every time. Model version mentioned: o4 mini - He mentions people using o4 mini or o3 to help craft better deep research prompts.
Pivotal Quotes: "when models are able to learn how to do the process by being rewarded for succeeding at the process, they're able to figure out in many cases something that's better than you could easily sit down and design yourself" — Josh Tobin: He explains why end-to-end training can outperform human-designed agent workflows. "my mental model for giving a task to the codex CLI is it's kind of like you're giving a task to a superhuman intern who has never seen your code base before" — Josh Tobin: He describes how Codex CLI operates and why it needs to explore codebases on its own. "the future that everyone is picturing is they'll just ask ChatGPT to go book a vacation" — Josh Tobin: He uses this example to discuss trust, permissions, and high-stakes actions for agents.
Implications: Agents are becoming practical, but the biggest blockers are reliability, tool access, and trust. Expect faster adoption in research and coding first, then broader autonomy as models improve and governance around sensitive actions matures.