Why you should write your own LLM benchmarks — with Nicholas Carlini, Google DeepMind

Episode Summary

Executive Summary: Nicholas Carlini argues that LLMs are genuinely useful today, but mostly as practical assistants for coding, learning, debugging, and evaluation—not as magical autonomous agents. He emphasizes grounded, verifiable use cases, custom benchmarks tied to real workflows, and a security mindset that treats current models as helpful but imperfect tools with serious risks.

Main Topics: Practical AI use in daily work (Priority: 5/5): Carlini explains how he uses LLMs to generate boilerplate, kickstart projects, explain unfamiliar tools, debug errors, and handle one-off tasks faster than manual work. Why he wrote the 'How I Use AI' post (Priority: 5/5): He wanted to counter both hype and dismissal by documenting only personally verified, real-world uses of models, especially from the perspective of a security researcher. Custom benchmarks over generic benchmarks (Priority: 5/5): He argues that people should build benchmarks from their own tasks so model evaluations reflect actual utility rather than leaderboard optimization or vibes-based testing. AI security and model/data extraction (Priority: 5/5): The conversation covers his research on adversarial ML, including data poisoning, model stealing, and training-data extraction from production systems. Prompting, agents, and model limits (Priority: 4/5): Carlini is skeptical of overemphasizing prompt engineering, but accepts that current models often need iterative back-and-forth and that better interfaces may reduce prompting burden. Writing, distribution, and independent thinking (Priority: 3/5): He describes writing as a tool for sharing ideas rather than enjoyment, and says he avoids social media to prevent distraction and online arguments. Fun side projects and Turing completeness (Priority: 3/5): He discusses playful technical projects like Game of Life CPUs, printf-based Turing completeness, and data storage on paper as examples of keeping engineering enjoyable.

Key Arguments: LLMs are useful when the task is verifiable and the user can check the output, even if the model is imperfect. The most valuable use of AI is often to handle boring, repetitive, or context-switching parts of work so humans can stay focused on the hard part. People should create benchmarks from their own real tasks because generic benchmarks often measure the wrong thing. Security research should focus on real systems and real attacks, not only theoretical worst-case scenarios. Prompting matters less than many people think for everyday use; if a model only works with elaborate prompting, it may not be useful for that user. Current models can help non-experts and experts alike, but the biggest risk is blind trust and deployment in unsafe settings. Model and data extraction attacks are important because they can force providers to change system behavior for security reasons. Writing and publishing are worthwhile when they create a durable artifact that others can inspect and learn from.

Data Points: Time spent writing the AI usage post: less than 10 hours - Carlini said he wrote the long blog post in one pass and did not spend much time editing. Model output length for app-building prompt: 30,000 words - He referenced a prior effort to get GPT-4 to build a 'Can GPT-4 solve this?' style app. Benchmark question count: 20 questions - He mentioned having a 20-question evaluation in his benchmark for fun. Disclosure period: 90 days - He said vulnerable model/API providers were notified and given 90 days to patch issues before publication. Recovered training-data match length: at least 50 words in a row - He described a case where ChatGPT repeated verbatim training data after being prompted to repeat a word forever. Paper storage estimate: about 1.44 megabytes - He estimated the amount of data that can be encoded on a single printed page with an off-the-shelf printer/scanner setup. Image dataset scale: LAION-400M - He discussed a poisoning attack involving expired domains in the LAION-400M image dataset. Model layer theft: one layer - He clarified that the discussed OpenAI extraction attack stole one layer, with prior work showing how to extend to more layers.

Pivotal Quotes: "I wanted to write something that just says: look, let's sort of ground reality and what we can actually do with these things." — Nicholas Carlini: Explaining the motivation for his 'How I Use AI' post. "The most underrated thing is the ability of it to solve the uninteresting parts of problems for me right now." — Nicholas Carlini: Describing the practical value of LLMs in research and programming workflows. "I want to make sure that we have exhausted the space of possible attacks so that it's not going to be the case that someone else comes up with a very bad thing." — Nicholas Carlini: On why he publishes security attacks and focuses on real-world adversarial research.

Implications: For users, LLMs are most valuable as verifiable assistants that reduce friction, not as fully trusted agents. For industry, the message is to build task-specific evals, improve safety, and avoid blind AI integration. For research, real-world security attacks and multi-turn benchmarks matter more than hype.