Ignore Previous Instructions and Listen To This Interview with Sander Schulhoff, CEO of Learnprompting.org

Episode Summary

Executive Summary: LearnPrompting.org founder Sander Schulhoff discusses practical and advanced prompt engineering, the growth of his open-source guide, and the shift toward agents. He also explains a global prompt-hacking competition that exposed persistent vulnerabilities: prompt-only safeguards can be bypassed, including across chained models. Developers should limit agent permissions, protect prompts, test adversarial inputs, and avoid treating prompt instructions as a security boundary.

Main Topics: LearnPrompting.org and making prompt engineering accessible: Schulhoff describes creating a comprehensive, approachable guide from scattered online prompting advice. The resource remains free and open source, while paid courses support enterprise and professional training. Practical and advanced prompting techniques: Core practices include supplying relevant context, providing examples, and eliciting reasoning. Clear labels and output formats can help, while role prompting is most reliable for style rather than accuracy. Contrastive chain of thought and model-generated examples are further techniques. Prompt evaluation, debugging, and model choice: Small wording changes, model versions, and answer formatting can materially affect results. Schulhoff describes systematic experimentation but emphasizes that benchmarking is difficult, implementations vary, and leaderboards provide an incomplete picture. From prompt engineering to agents: Agents can inspect files, use tools, and take actions across multiple steps. Schulhoff expects organizations to need people who can design job-specific agents, including managing tool use, context, and debugging—not just write prompts. Prompt hacking competition and attack taxonomy: The competition tested whether users could override developer instructions in varied prompt setups. Participants found attacks involving obfuscation, context switching, indirect requests, and chained models; one defense layer could be induced to attack the next. Security limits and mitigation: Schulhoff argues that prompt-based defenses alone do not reliably prevent injection. He recommends obscuring prompts, restricting agent permissions, and testing with adversarial examples, while stressing that these measures are imperfect and capability limits matter.

Key Arguments: Prompt engineering remains valuable even as models become easier to use: Schulhoff reports relying on many of the same techniques as a year earlier and says model choice and wording can still change outcomes. The most broadly useful prompting fundamentals are relevant context, few-shot examples, and prompting for reasoning; role prompts can shape style but may fail to improve—or even hurt—accuracy. Prompting results require systematic evaluation. Benchmark outcomes depend not only on the data set but also on prompt ordering, answer extraction, formatting, and model version. Agents extend prompting into multi-step action and tool use, creating greater utility but also more complicated design and security problems. The competition demonstrates that prompt injection can bypass developer instructions in diverse ways, including by manipulating one model to produce an attack against a second model. Prompt-only defenses are not a dependable security boundary. Developers should limit what an AI can access or execute and should treat filters and prompt secrecy as partial, bypassable safeguards.

Data Points: LearnPrompting.org users: About 2 million - The open-source guide attracted users worldwide, including researchers and everyday readers. Authors on prompting survey paper: About 20 - Schulhoff said the prompting survey effort included authors from OpenAI, Google, and Microsoft. Prompt-hacking competition data set: About 600,000 prompts - Schulhoff said the collected data set could be used to test systems and defenses. Direct attack transfer from GPT-3 to GPT-4: Almost 40% - The team reported that prompts developed against GPT-3 transferred directly to GPT-4 in their transferability study. Competition funding and prizes: $40,000 in cash and other prizes - Schulhoff described this as the support assembled for the competition; he also recalled an initial $2,000 in credits from Scale and about $7,000 from another sponsor. Prompt-hacking competition duration: About one year overall - Schulhoff estimated the process—from sponsorship outreach through competition, review, publication, and conference recognition—took roughly a year. Expected focus for 2024: “The year of agents” - Schulhoff predicted agents would be a major focus as systems gained the ability to inspect information and take actions.

Pivotal Quotes: "The most important: context, few shot, and thought." — Sander Schulhoff: His concise summary of core prompting practices for everyday users. "Prompt-based defenses do not work, period." — Sander Schulhoff: His warning that instructions telling a model to resist malicious inputs are not sufficient security. "And that's the key component of agents these days: taking actions." — Sander Schulhoff: His distinction between ordinary prompting and systems that use tools and act across steps.

Implications: Users can improve results with context, examples, and iterative testing, but organizations should not rely on prompts to enforce security. Agent builders need least-privilege access, adversarial testing, and safeguards outside the model; prompt injection remains an open architectural challenge.