Emergency Pod: OpenAI's new Functions API, 75% Price Drop, 4x Context Length (w/ Alex Volkov, Simon Willison, Riley Goodside, Joshua Lochner, Stefania Druga, Eric Elliott, Mayo Oshin et al)

Episode Summary

Executive Summary: The episode is an emergency developer-focused discussion of OpenAI’s June update, centered on the new Functions API, cheaper embeddings, and a larger GPT-3.5 context window. Panelists debate whether this is a major platform shift or an incremental improvement, weighing gains in reliability and developer ergonomics against unresolved issues like hallucinations, prompt injection, and lock-in. The group also explores implications for agents, retrieval, local embeddings, and future AI UX.

Main Topics: OpenAI June update overview (Priority: 5/5): The hosts frame the release as a developer-centric emergency pod covering the Functions API, embedding price cuts, and GPT-3.5 16K context expansion. Functions API and structured tool use (Priority: 5/5): Panelists discuss how the new API formalizes tool/function calling, adds a function role, and makes structured outputs easier than prompt-engineered JSON. Pricing and context window changes (Priority: 4/5): The conversation highlights major reductions in embedding and GPT-3.5 costs, plus a 4x larger context window for GPT-3.5, with debate over real-world impact. Prompt injection and security (Priority: 5/5): Several speakers warn that easier function calling increases the attack surface and that prompt injection remains unsolved, requiring human approval and reversible actions. Agents, retrieval, and architecture tradeoffs (Priority: 4/5): The panel contrasts LLM-centric agent designs with code-centric systems, discussing recursion, tool routing, retrieval, and whether functions push the field toward more structured agent architectures. Open source, local models, and client-side embeddings (Priority: 3/5): Participants compare OpenAI’s ecosystem to local/browser embeddings and open-source alternatives, emphasizing privacy, cost, and reduced vendor dependence. Future product requests and AI UX (Priority: 3/5): The group asks for widgets, better retrieval/knowledge graphs, vision APIs, and non-chat interfaces, arguing that chat alone is not the right UX for many tasks.

Key Arguments: The Functions API is effectively a formalized plugins/tool-calling interface, making it much easier to build reliable LLM workflows without elaborate prompt hacks. Cheaper embeddings and larger context windows reduce friction and cost, but long-context performance may not scale linearly with token count. Prompt injection becomes more dangerous when models can take actions in the world; security must assume adversarial inputs and require reversible or human-approved operations. OpenAI’s release may increase ecosystem lock-in by making its models the easiest place to build tool-using applications and retrieval pipelines. The field may be shifting from LLM-core/code-shell systems toward code-core/LLM-shell systems, where code handles more of the reliable logic and the model is constrained to smaller tasks. Structured outputs and function schemas are useful, but they do not fully solve hallucination or reliability problems; they mainly reduce developer burden. Local/client-side embeddings remain relevant for privacy, latency, and experimentation, even as API embeddings become dramatically cheaper. The most important long-term gains may be lower prices and larger context windows rather than the functions feature alone.

Data Points: Embedding price reduction: 75% - OpenAI cut embedding prices again, following a prior major reduction months earlier. GPT-3.5 price reduction: 25% - The June update also lowered GPT-3.5 pricing. GPT-3.5 context window increase: 4x - The new GPT-3.5 model expands context from 4K to 16K tokens. Estimated cost to embed the whole internet before update: $50 million - A tweet cited during the discussion estimated the cost before the new pricing cut. Estimated cost to embed the whole internet after update: $12.5 million - A follow-up estimate after the price reduction. Embedding dimensionality mentioned for OpenAI: 1,536 dimensions - Zenova contrasted OpenAI embeddings with smaller local/browser embeddings. Local/browser embedding dimensionality mentioned: 768 dimensions - Used as a comparison point for client-side embedding approaches. Audience size: 1,400 people - The hosts said about 1,400 people tuned in to the prior discussion. Hackathon prize pool: $100K - Roy announced Pinecone’s virtual hackathon prize pool. Hackathon dates: June 19th to June 26th - Pinecone’s virtual hackathon schedule. GPT-4 Vision access mentioned: 10% of Bing users - Riley cited a Bing update indicating partial access to GPT-4 Vision.

Pivotal Quotes: "This is effectively the chat to beauty plugins API being released to us." — Sean: He describes the Functions API as a formalization of the plugin/tool-calling pattern developers had previously hacked together. "Prompt injection is kind of, doesn't matter if you're just playing with a chatbot that can't actually do anything. It only becomes dangerous when you hook them up to functions that let them do things in the world." — Simon: He warns that the new API lowers the barrier to dangerous real-world actions. "I think the real thing that might be more important in the end is just this march towards lower prices and bigger context windows." — Riley: He argues that cost and context improvements may matter more than the Functions API itself.

Implications: Developers can build more structured, tool-using AI systems with less prompt engineering, but they must treat security and reversibility as first-class concerns. The release accelerates agent and retrieval workflows while intensifying debates over lock-in, open-source alternatives, and the future of AI UX beyond chat.