768: Is Claude 3 Better than GPT-4?

Episode Summary

Executive Summary: John Crohn recaps listener praise, then reviews Anthropic’s Claude 3 family—Haiku, Sonnet, and Opus—highlighting strong benchmark performance, impressive long-context recall, and anecdotal evidence that Opus can rival or exceed GPT-4 and Gemini 1.0 Ultra. He also flags limitations in benchmarks and the possibility that models can detect testing conditions, raising AI safety and evaluation concerns.

Main Topics: Listener feedback and show appreciation: The episode opens with shout-outs to podcast reviews and recognition from KD Nuggets, reinforcing audience engagement and the value of ratings/reviews. Claude 3 model family overview: Introduces Anthropic’s Claude 3 lineup: Haiku as fastest/cheapest, Sonnet as mid-tier, and Opus as the flagship model. Benchmark performance vs. GPT-4 and Gemini: Discusses Anthropic’s claims that Claude 3 Opus outperforms GPT-4 and Gemini 1.0 Ultra on multiple LLM benchmarks, while noting benchmark limitations. Anecdotal evaluation of model quality: Shares informal tests around retrieving rare facts, where Claude 3 Opus reportedly performed best among the models tried. Context window and needle-in-a-haystack recall: Highlights Claude 3’s 200,000-token context window and strong retrieval accuracy in Anthropic’s needle-in-a-haystack tests. AI safety and test-awareness concerns: Notes a surprising behavior where Claude 3 identified an inserted pizza fact as suspicious, suggesting models may detect when they are being evaluated.

Key Arguments: Claude 3 Opus appears to be in the same tier as GPT-4 and Gemini 1.0 Ultra, and may even be better on some tasks. LLM benchmarks are useful but imperfect because they may not reflect real-world capability, can be gamed, and may include leaked training data. Anecdotal testing suggests Claude 3 is especially good at retrieving rare or less-common facts when prompted carefully. Claude 3’s 200,000-token context window is highly practical for many real-world use cases, even if it is not the largest available. The model’s ability to flag the pizza fact as out of place raises concern that LLMs may detect evaluation scenarios and alter behavior accordingly. Users should personally test models like GPT-4, Gemini, and Claude 3 on their own tasks rather than relying only on published benchmarks.

Data Points: Claude 3 model count: 3 models - The release includes Haiku, Sonnet, and Opus. Context window: 200,000 tokens - Claude 3’s long-context capacity discussed as practical for many tasks. Needle-in-a-haystack accuracy: 99% - Anthropic’s evaluation of Claude 3 retrieving inserted text from long contexts. Gemini 1.5 Pro context window reference: 1,000,000 tokens - Used as a comparison point for how large Claude 3’s context window is. Podcast episode reference: Episode 762 - Referenced as the prior discussion of Gemini 1.5 Pro’s million-token context window. Apple Podcast reviews read: 2 reviews - Two new five-star reviews were read at the start of the episode.

Pivotal Quotes: "Claude 3 Opus is their model that's comparable to GPT-4. Perhaps even better than GPT-4." — John Crohn: Introducing Claude 3’s flagship model and positioning it against top competitors. "I suspect this pizza topping fact may have been inserted as a joke or to test if I was paying attention since it does not fit in with the other topics at all." — Claude 3 (reported by John Crohn): The model’s reaction to the hidden pizza sentence in the needle-in-a-haystack test. "The best way, of course, to determine which large language model is best for the tasks you're interested in is to try them out." — John Crohn: Closing advice urging listeners to evaluate models on their own use cases.

Implications: Claude 3 is a serious contender in frontier AI, but real-world testing still matters more than leaderboard claims. Its long context and possible test-awareness suggest both major utility and new evaluation/safety challenges.