Episode Summary
Executive Summary: Kevin Weil, OpenAI’s CPO, describes a company operating amid rapid model improvements where product strategy must stay flexible, eval-driven, and model-maximalist. He explains OpenAI’s internal cadence, why chat is a durable interface, how product and research increasingly co-build, where startups can still win, and why AI will reshape work, creativity, education, and consumer behavior over the next few years.
Main Topics: OpenAI’s pace and product philosophy (Priority: 5/5): Weil says OpenAI operates under extreme uncertainty because model capabilities improve every few months, so planning must be lightweight, bottoms-up, and iterative. The company emphasizes launching early, learning in public, and trusting teams to move quickly without waiting on heavy approvals. Evals as a core product skill (Priority: 5/5): He defines evals as tests or benchmarks for model capability and argues they are essential because product design now depends on whether a model is 60%, 95%, or 99.5% reliable on a task. Product teams must create custom evals, use them to fine-tune models, and continually measure progress. How OpenAI builds products with models (Priority: 5/5): Weil explains that OpenAI increasingly uses ensembles of models, fine-tuning, and specialized prompts for different tasks. He sees future product teams including researchers and ML-minded builders who can adapt models to company-specific data and workflows. ChatGPT as the default interface (Priority: 4/5): He argues chat is not a temporary UI but a universal interface matching how humans communicate. Chat works across intelligence levels and use cases, while more prescribed interfaces can still be better for narrow, high-volume tasks. AI’s impact on work, startups, and careers (Priority: 4/5): Weil believes many startups should build in areas OpenAI won’t cover because data and workflows are domain-specific. He expects more fine-tuned models, more product teams with research talent, and major workflow changes such as vibe coding, AI-assisted prototyping, and AI support automation. Future benefits: education, creativity, and reskilling (Priority: 4/5): He is highly optimistic about AI in tutoring, creativity, and reskilling, pointing to personalized education and tools like image/video generation as transformative. He says AI can let people create beyond their own manual skills and could dramatically improve learning outcomes globally. Libra as a lost opportunity (Priority: 3/5): Weil reflects on Facebook’s Libra project as his biggest career disappointment, arguing the product should exist because cross-border money transfers are expensive and slow. He thinks the idea remains valid and may be more feasible today given crypto-friendly policy and Meta’s different reputation.
Key Arguments: Model capability is moving so fast that any AI product should assume the current model is the worst one users will ever have. Because model quality changes continuously, roadmaps must be directional rather than fixed; planning is useful, but plans are not. Evals are the equivalent of unit tests for AI and are necessary to know whether a use case is safe to ship and how the model should be improved. AI products need custom workflows, custom data, and often fine-tuned or ensemble model stacks rather than one generic model call. OpenAI will not and should not build every AI product; there are enormous opportunities for startups in vertical or company-specific use cases. Chat is a powerful interface because it mirrors human communication and provides maximum flexibility across many task types. The best AI products will increasingly include researchers embedded in product teams, not just in centralized model groups. AI should significantly improve education through personalized tutoring and reskilling, with broad societal upside despite near-term labor disruption. Creative work will be expanded, not replaced, by AI: models like Sora and image generation help people explore more ideas and produce better final outputs. A successful AI company should iterate in public, ship early, and expect capabilities to change underneath the product. Fine-tuning and specialized prompts will become standard because models can be made much better for specific tasks than they are out of the box. Prompt engineering should matter less over time as models get easier to use, but today examples in prompts still materially improve outputs.
Data Points: OpenAI API developers: 3 million - Weil says OpenAI serves 3 million developers through its API and wants to power many more use cases. Weekly active users: 400 million+ - He references OpenAI’s massive ChatGPT user base when discussing customer support and feedback volume. Image model internal rollout: A couple of months - He says ImageGen had been in internal testing for a couple months before launch and generated strong internal buzz. Reasoning model latency: 25–30 seconds - He describes the wait time for a reasoning response as long enough to feel different, but short enough to keep the user in-session. Deep research runtime: 25–30 minutes - He says deep research can work for this long, doing work that might otherwise take a week. Model cadence: Every 2 months - He says models can gain new capabilities roughly this frequently. Reasoning model cadence: Roughly every 3–4 months - He says the O-series of reasoning models is improving on this pace. Cost reduction: ~100x cheaper - He says the original GPT-3.5-era model was around a hundred times the cost of GPT-4o mini today in the API. Libra remittance fees: Around 20% - He cites the cost of sending money home across borders as an example of a broken payments system Libra aimed to fix. Interview delay: 9 days - He recounts waiting nine days after interviewing for OpenAI before getting a hiring response. Company PM count: About 25 - He estimates OpenAI has roughly 25 product managers, emphasizing a lean PM organization. Children’s ages: 10, 8, 8 - He mentions having one 10-year-old and eight-year-old twins while discussing AI-native kids and education.
Pivotal Quotes: "The AI models that you're using today is the worst AI model you will ever use for the rest of your life." — Kevin Weil: He frames the exponential improvement curve of AI and why product teams should keep building near the frontier. "Writing evals is going to become a core skill for product managers." — Kevin Weil: He explains that PMs will need to define tests for model behavior in order to ship reliable AI products. "The models are not perfect. They're going to make mistakes... our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are." — Kevin Weil: He describes OpenAI’s model-maximalist shipping philosophy and why they avoid over-scaffolding around current model weaknesses.
Implications: AI product teams should build around fast-changing capabilities, invest in evals and fine-tuning, and expect more research/product blending. For users, AI will increasingly become a universal work, creation, and learning layer—especially through chat, agents, and personalized tools.
About Lenny's Podcast
Lenny Rachitsky interviews world-class product leaders and growth experts about building products and growing careers.