Episode Summary
Executive Summary: Garrett Peterson describes a long-running experimental economics paper, "Instructions," that unexpectedly shifted from studying memory sophistication to studying how experimental instructions are understood. The core finding is that many participants misunderstood the rules despite multiple interventions, prompting broader lessons for experimental economists: use clearer, user-friendly instructions, quizzes, feedback, repetition, and better documentation of procedures.
Main Topics: Origin of the Paper and Memory Study (Priority: 5/5): The project began as a test of whether people are sophisticated about their own forgetfulness: whether they anticipate memory lapses and take preemptive action. Poodle Jump vs. Slider Task Experimental Design (Priority: 5/5): Participants played a distracting game while deciding when to perform a one-time slider task whose payoff varied over time, creating a test of whether they would wait, act immediately, or forget entirely. Unexpected Misunderstanding and Pivot (Priority: 5/5): The original memory hypothesis broke down because many participants behaved as if they did not understand the instructions, including mashing the button immediately even when waiting would pay much more. Instruction Treatments and Their Effects (Priority: 4/5): The team tested quizzes, answer feedback, payments for correct quiz responses, paper instructions, and repeating instructions; these reduced but did not eliminate misunderstanding. Meta-analysis of Experimental Economics Papers (Priority: 5/5): The authors reviewed 260 lab experiment papers across top journals to see how instructions are delivered and documented in the field, finding large variation and substantial reporting ambiguity. Broader Lessons for Experimental Design (Priority: 4/5): The episode concludes with recommendations for more user-friendly instructions, repeated prompts, clearer language, pilot testing, and stronger attention to participant comprehension.
Key Arguments: Participants often do not read or process experimental instructions carefully, so apparent behavior may reflect confusion rather than true preferences or strategic reasoning. The original memory-study design failed because many subjects misunderstood that the slider task could only be done once and at a specific optimal time. Quizzes with correct answers, paper instructions, and repeated instructions improved comprehension, but no treatment eliminated misunderstanding completely. Paying participants for correct quiz answers did not produce additional improvement beyond quiz-and-answer feedback, suggesting incentives alone do not guarantee understanding. Experimental economists should treat instruction delivery as a substantive design choice, not a trivial administrative detail, because it can materially affect outcomes and replication. The literature lacks consistent reporting about how instructions are delivered; in the reviewed sample, many papers were unclear about this basic procedural detail. Behavior that looks irrational in lab data may sometimes be caused by participants testing the software, expecting to learn by doing, or missing key details in the instructions.
Data Points: Co-authors: 4 authors total - The paper "Instructions" was co-authored by Dave Freeman, Eric Kimbrough, Garrett Peterson, and Han Tong. Journal: Journal of the Economic Science Association (JEZA) - The paper was conditionally accepted after revise-and-resubmit. Original distraction task payoff: 25 cents per minute - Participants earned this while playing the poodle jump game during the memory experiment. Early slider-task payoff in main treatment: 20 cents for the first 21 minutes - In the higher-payoff condition, the slider task was available early at a lower value. Peak slider-task payoff: $7 in minute 22 - This was the optimal time to perform the slider task if participants remembered the instructions. Approximate earnings in optimal treatment: Around $21 - Estimated total earnings when waiting until the optimal time before doing the slider task. Approximate earnings if done immediately in high-payoff condition: About $19 - Participants could earn slightly less by doing the slider task right away and then continuing the distraction task. Share of participants misunderstanding: About one-third - In the initial treatments, roughly a third of participants behaved in ways indicating misunderstanding. Reduced misunderstanding after interventions: Closer to one-fifth - Quiz feedback, paper instructions, and repeated instructions lowered but did not eliminate misunderstanding. Papers coded in meta-analysis: 260 papers - The authors reviewed lab experiment papers from five top journals over five years. Paper-only instructions: 124 papers - Most common instruction format in the literature review. Computer-only instructions: 48 papers - A substantial minority of papers used computer-delivered instructions only. Both paper and computer instructions: 11 papers - Some studies used multiple instruction formats. Unclear instruction format: 56 papers - Many papers did not clearly specify how instructions were delivered, even after checking supplements. Neither paper nor computer instructions: 21 papers - These were cases where instructions were delivered orally or in very simple field-style setups.
Pivotal Quotes: "people don't always read things super clearly" — Garrett Peterson: Main takeaway from the experiment and its instruction-comprehension failures. "maybe participants in Vancouver are just well stupid" — Referee report: A referee’s blunt remark about whether the unexpected behavior reflected participant quality rather than instruction problems. "make your experiment more user-friendly, give prompts and reminders during the experiment, use plain language" — Garrett Peterson: Final advice to experimental economists based on the paper’s findings.
Implications: The episode argues that experimental outcomes can be badly distorted by misunderstood instructions. Researchers should improve clarity, add checks for comprehension, and report procedures precisely to strengthen validity and replication.
About Economics Detective
Economics Detective Radio is a podcast about markets, ideas, institutions, and all things related to the field of economics. Episodes consist of long-form interviews and are generally released on Fridays. Topics include economic theory, economic history, the history of thought, money, banking, finance, macroeconomics, public choice, business cycles, health care, education, international trade, and anything else of interest to economists, students, and serious amateurs interested in the scienc...