Adam Mastroianni on Peer Review and the Academic Kitchen

Episode Summary

Executive Summary: Russ Roberts interviews Adam Mastriani about his critique of academic peer review. Mastriani argues the system is modern, costly, slow, and usually does not verify truth; instead it filters for novelty and status while often leaving errors and fraud undetected until after publication. He proposes a more open, diverse, reader-driven model of scientific communication.

Main Topics: What peer review is assumed to do versus what it actually does (Priority: 5/5): Roberts and Mastriani contrast the common belief that peer review thoroughly checks correctness with the reality that referees often skim for interest, novelty, and fit rather than verify data or rerun analyses. Peer review as a recent historical invention (Priority: 5/5): Mastriani explains that modern journal-based peer review is not an ancient feature of science but a mid-20th-century centralization of a much more varied communication ecosystem. Costs, delays, and incentives in the publication system (Priority: 5/5): The discussion emphasizes how long reviews take, how much unpaid labor they consume, and how publish-or-perish incentives push scholars toward safer, smaller, and sometimes less meaningful work. Replication crisis and failure to catch errors (Priority: 5/5): They discuss evidence that reviewers miss major errors, fraud is usually detected only after publication, and many findings—especially in psychology and related fields—do not replicate. Peer review may be worse than nothing (Priority: 4/5): Mastriani argues the system creates unjustified trust: readers infer reliability from publication status even though journals often do not meaningfully validate claims, which can distort public understanding. An alternative model: open, readable, diverse science (Priority: 5/5): Mastriani describes publishing work directly online with code, data, and plain language, allowing voluntary public feedback rather than gatekept, end-of-process approval. Science as a strong-link rather than weak-link problem (Priority: 4/5): He argues science should not be optimized to suppress the worst work at the expense of the best work, because the best ideas drive progress while bad work typically fades away.

Key Arguments: Peer review is widely misunderstood as a rigorous correctness check, but in practice it usually does not verify data, rerun code, or fully assess the underlying analysis. The current system is historically recent and was not the dominant form of scientific communication for most of science’s history. The publication process imposes large time costs on authors, referees, and the broader research enterprise, diverting effort from actual discovery. Most referees do not detect major errors reliably; experimental studies found only about a quarter of inserted errors were caught on average. Fraudulent papers are almost never exposed at the review stage; discovery usually happens after publication through replication attempts or independent investigation. Because publication status confers credibility, peer review can mislead readers into over-trusting papers and thereby become worse than a transparent, no-pretense system. Many published findings are either not important enough to matter or are unlikely to be robust; replication efforts reveal that a substantial share do not hold up. A more open model—plain-language writing, public data/code, and post-publication feedback—may better fit how scientific truth actually advances. Science should tolerate variance: suppressing some low-quality work risks suppressing high-quality work that could be transformative. The right standard is not whether every claim survives a journal gate, but whether the broader ecosystem can identify and amplify the best ideas over time.

Data Points: Review time cost: 15,000 person-years per year - Estimated annual time scientists spend reviewing papers, cited as a major social cost of peer review. Publication lag: Often 1 year or more - Mastriani says a one-year gap from submission to publication is considered doing well, and multiple years is common after rejections and resubmissions. Error-detection rate in referee studies: About 25% - British Medical Journal studies inserting errors into papers found referees caught only about a quarter on average. Randomization error detection: About 50% - For the specific error that a randomized trial was not actually randomized, only about half of reviewers noticed it. Non-citation / non-use estimate: About 30% never cited again in social sciences - Mastriani cites a rough estimate that many papers are never meaningfully used after publication. Alternative estimate mentioned: 80% never looked at again - Russ Roberts notes another number he had seen, suggesting an even larger share of papers are never revisited. Impactful citations in one’s own paper: 12% to 15% - Mastriani says only a small share of citations in his own papers were critical to the argument; most were not foundational. Original study sample size: N of 18 - Used by Roberts when discussing the Lancet autism-vaccine paper as an example of weak evidence gaining prestige through publication. Public response to essay: ~215,000 views - Mastriani reports his essay on peer review became his most widely read piece. Age of guest: 31 years old - Roberts asks Mastriani’s age near the end of the conversation, framing the risk of his critique as a young scholar.

Pivotal Quotes: "published and true are not synonyms" — Brian Nosek, Jeffrey Spies, and Matt Model: Roberts cites this line to capture the central replication-crisis insight that publication does not guarantee truth. "What I'd like to do is make a historical claim, which is this is new and weird, an empirical claim, which is this doesn't seem to do the thing that we intend for it to do" — Adam Mastriani: Mastriani summarizes his case against peer review as both historically contingent and empirically ineffective. "The radical idea that people aren't stupid" — Adam Mastriani: Mastriani describes his broader philosophy that ordinary readers can understand, evaluate, and benefit from openly shared science.

Implications: Listeners should be more skeptical of “peer-reviewed” as a stamp of truth and more attentive to transparency, replication, and direct evidence. The episode suggests science may advance better through open, diverse, post-publication critique than through closed journal gatekeeping.