Episode Summary
Executive Summary: Simon Willison discusses how generative AI has transformed coding, prototyping, and data analysis, especially through ChatGPT Code Interpreter, Claude, and vision models. He frames LLMs as powerful but uneven tools best used with examples, iterative QA, and human oversight. The conversation centers on practical workflows, retrieval/search, local vs hosted models, and the future of tool-using AI for journalists and developers.
Main Topics: AI as a productivity multiplier for coding and prototyping (Priority: 5/5): Willison describes LLMs as a dramatic boost for software development, especially for exploratory coding, rapid prototypes, and repetitive implementation work. He treats models like a fast, tireless intern that can generate and test code. ChatGPT Code Interpreter and tool use in a loop (Priority: 5/5): He repeatedly emphasizes Code Interpreter as the standout capability because it can write, execute, debug, and iterate on code, enabling workflows like testing Python or compiling C extensions directly in the model environment. Workflow preferences: Claude, ChatGPT, voice, and IDEs (Priority: 4/5): Willison explains his day-to-day stack: Claude as default, ChatGPT for Code Interpreter and voice, Copilot mainly for autocomplete, and occasional use of custom CLI tools. He favors mainstream interfaces over niche tooling. Vision models and structured extraction from images/PDFs (Priority: 5/5): The discussion explores GPT-4 Vision, Claude vision, and Gemini vision features such as OCR, bounding boxes, document extraction, and converting photos of flyers or scanned forms into structured data. Retrieval, search, and RAG limitations (Priority: 4/5): Willison argues that good RAG requires classical information retrieval as much as embeddings, and notes that vector search alone can return irrelevant but plausible answers. He highlights the complexity of production-grade retrieval systems. Local/open models vs hosted frontier models (Priority: 4/5): He is enthusiastic about open models and local execution, but says hosted models like Claude 3.5 Sonnet still outperform laptop-run models in most tasks. He sees open models as improving quickly and enabling cheaper, more controllable deployments. Agents, loops, and research assistants (Priority: 4/5): Willison is supportive of tool-use loops and repo-level automation like GitHub Copilot Workspace, but skeptical of vague 'agent' hype. He wants reliable research assistants that can visit multiple sites and synthesize data without hallucinations.
Key Arguments: LLMs are most valuable when paired with tools that let them run code, inspect results, and iterate; Code Interpreter is the clearest example of this. For coding, hallucinations are less dangerous because bad code fails visibly when run, making QA and iterative testing central to effective use. The best prompting technique is often to provide examples rather than long instruction blocks; his prompting style has become shorter and more example-driven over time. Vision models are underexplored relative to text models, but they already enable powerful tasks like OCR, table extraction, bounding boxes, and document understanding. RAG systems often fail not because retrieval is impossible, but because the hard part is combining retrieval with the right constraints, ranking, and user intent handling. Vector databases are less compelling as standalone products than vector indexes inside normal databases like SQLite or Postgres. Open models are improving fast and are important for pricing competition and control, but hosted frontier models remain the most capable for daily work. The most useful AI workflows are often practical and narrow—prototyping, extraction, search, and coding—rather than broad AGI speculation.
Data Points: Length of dog-walk coding sessions: 1 hour - Willison says he walks his dog for an hour while coding via voice mode. ChatGPT Code Interpreter access timing: early last year - He got beta access to Code Interpreter early and immediately saw its data-analysis potential. Usage share of coding in his work: ~70% - He estimates about 70% of his LLM usage is programming-related. Typing as share of software-engineering work: ~10% - He estimates only about 10% of his software engineering job is typing code. Claude context window: 200,000 tokens - He cites Claude’s standard context length. Claude high-end context window: 500,000 tokens - He mentions a paid Claude variant with a much larger context window. Gemini context window: 2 million tokens - He notes Gemini’s very large context window, enough to handle long video inputs in frames. Llama 70B memory usage: ~64 GB RAM - He says running Llama 70B on his Mac nearly consumes his entire 64 GB system memory. Availability of Gemini bounding boxes: Flash and Pro - He says Gemini bounding-box support is available in both Flash and Pro models. OpenAI/Google model timing: less than 1 year old - He describes vision-capable LLMs as very new, with GPT-4 Vision only around since last November.
Pivotal Quotes: "I thought I was going to spend the next like 10 years of my life figuring out how to give people the tools to analyze data. And this thing just does it, like right out of the box." — Simon Willison: Reflecting on how ChatGPT Code Interpreter changed his perspective on his Dataset project and data analysis tools. "I call it my weird intern because it really does feel like you've got this intern who is screamingly fast and they've read all of the documentation for everything." — Simon Willison: Describing how he thinks about LLMs when using them for production code and iterative development. "I hate the word agent because I think every everyone you talk to about agents has a different idea of what it means." — Simon Willison: Explaining his skepticism about the ambiguity of the term 'agent' and his preference for precise descriptions of tool-use loops.
Implications: Listeners should expect AI workflows to become more tool-rich, iterative, and multimodal. The biggest near-term gains are in coding, prototyping, extraction, and search—provided users pair models with evaluation, examples, and human review.