Episode Summary
Executive Summary: Riley Goodside explains how his unconventional entry into AI via Twitter led him to prompt engineering, starting with early GPT-3/Codex experiments and then deeper study of what LLMs can and cannot do. The conversation frames prompting as sculpting model behavior through structure, examples, and task decomposition, while emphasizing three mental models: pretraining distribution, RLHF, and autoregressive sampling failure.
Main Topics: Riley’s path into prompt engineering (Priority: 5/5): Goodside describes moving from data science to prompt engineering during a 2022 sabbatical, inspired by Codex and early experiments with GPT-3 in Jupyter notebooks. Unexpected capabilities and hallucination discovery (Priority: 5/5): Early experiments showed models could handle obscure patterns like hash values and could be induced to answer trick questions or play along with absurd prompts, suggesting broad latent capabilities but also hallucination risk. Mental models for understanding LLM behavior (Priority: 5/5): He emphasizes three lenses: pretraining data as a space of possible text, RLHF as shaping outputs toward approved responses, and autoregressive inference as a source of compounding errors and derailment. Prompting as scaffolding and problem decomposition (Priority: 5/5): Goodside argues effective prompting is less about clever wording and more about structuring tasks, using checklists, K-shot examples, and representations that make the model’s job easier. Limits of zero-shot prompting and the role of fine-tuning (Priority: 4/5): He cautions that prompts are often only scaffolding and that many real tasks require moving toward curated examples, retrieval, or fine-tuning rather than relying on text alone. Advanced prompting workflow and practical techniques (Priority: 4/5): He discusses using edge-case-rich examples, invented examples, data restructuring, and approaches like Copilot-style retrieval/static analysis to improve reliability. Resources and community learning (Priority: 3/5): Goodside recommends learnprompting.org and Twitter/X as the most useful places to track the fast-moving prompt engineering field.
Key Arguments: LLMs have broader latent capabilities than early users assumed; obscure text patterns like MD5/SHA-1 hashes revealed surprising memorization and generalization. Trick-question behavior was not proof of “understanding nothing”; it may instead reflect the model selecting a sarcastic/joking text manifold. RLHF substantially changed model behavior by making responses more answer-seeking, compliant, and less obviously improv-like. Prompting works best when it constrains the model’s space of possible continuations rather than relying on elegant wording alone. K-shot prompts can help, but they also introduce bleed-through and ambiguity; they are not universally superior to zero-shot prompting. Many problems are better solved by decomposing them into steps, checklists, or structured formats rather than asking for a final answer directly. Prompt engineering should often be viewed as a temporary scaffolding layer on the path to data collection, retrieval, or fine-tuning. Output reliability is limited by autoregressive sampling; once a generation drifts out of distribution, it can compound into nonsense. The model’s inability to pause and think independently explains why chain-of-thought, tool use, and other external scaffolds matter. Good prompting often means understanding the model’s weaknesses—calculation, counting, regex, ambiguous references—and reshaping inputs accordingly.
Data Points: Timeline: Beginning of 2022 - Riley says his deeper work with LLMs began during a sabbatical after leaving Grindr. Model release referenced: Codex - He cites Codex as the release that motivated him to start developing with LLMs. GPT-3 version referenced: text-davinci-002 - He says early hash-memorization observations were from this early GPT-3-era model. GPT-3 version referenced: text-davinci-003 - He notes this RLHF’d model, alongside ChatGPT, improved trick-question handling. Question set performance: All but one - He says ChatGPT answered all but one of Hofstadter/Bender’s trick questions correctly. Failure case: World record for walking across the English Channel entirely on foot - The one question ChatGPT failed; Riley researched it extensively and found obscure historical evidence. Paper prompt result: 10-shot outperformed by zero-shot - He describes a French-to-English translation prompt where a flattering zero-shot prompt beat a 10-shot prompt. Prompt pattern: Masterful French translator flawlessly translates this sentence in English - The exact style of prompt he says improved translation performance. Pause-token effect: Constant amount of floating point operations per token - He argues chain-of-thought is partly inevitable because each extra token buys more compute time. Sequence corruption scheme: epsilon - He references a paper where random token corruption with a backspace token improved behavior.
Pivotal Quotes: "Prompting is usually scaffolding." — Riley Goodside: He summarizes his view that prompts are often a temporary mechanism for shaping or collecting behavior, not the final solution. "The model doesn’t talk to us, it freestyle raps." — Riley Goodside: He uses this metaphor to explain why autoregressive generation can drift and hallucinate without pause or self-correction. "By telling the model it’s an expert, it becomes an expert." — Riley Goodside: He describes how flattering zero-shot prompts can improve outputs by narrowing the model’s expected context space.
Implications: For practitioners, better prompting means structuring tasks, exploiting model strengths, and knowing when to move to retrieval or fine-tuning. For the industry, RLHF and tool scaffolds matter as much as prompt wording, and future gains may come from better sequence-control methods.