778: Mixtral 8x22B: SOTA Open-Source LLM Capabilities at a Fraction of the Compute

Episode Summary

Executive Summary: John Crohn highlights Mistral’s Mixtral 8x22B, a new open-source mixture-of-experts LLM that improves on prior Mistral models and other open-source competitors across reasoning, language, coding, and math benchmarks. He emphasizes its efficiency at inference, permissive Apache 2.0 licensing, 64k context window, and native function calling, positioning it as a major practical release for developers and data scientists.

Main Topics: Mixtral 8x22B release: Introduction of Mistral’s newest open-source model and why it matters as a state-of-the-art upgrade. Mixture-of-experts architecture: Explains how Mixtral routes prompts to specialized expert submodels, using only a subset of parameters per query. Benchmark leadership: Claims the model sets new highs across major benchmarks, including general reasoning, knowledge, coding, math, and multilingual tasks. Open-source licensing and accessibility: Notes the Apache 2.0 license and emphasizes unrestricted use, unlike more restricted 'open' models. Practical developer features: Highlights the 64,000-token context window and native function calling for integrating LLMs into software workflows. Efficiency and production cost: Argues that sparse expert activation reduces inference cost and latency relative to dense models of similar total size.

Key Arguments: Mistral’s Mixtral 8x22B is a major step up from its earlier 8x7B MoE model and earlier dense 7B model. The mixture-of-experts design allows only a fraction of parameters to be used per request, improving cost and speed at inference time. Mixtral 8x22B reportedly outperforms all other open-source models on key benchmarks, including MMLU, coding, math, and several European languages. Apache 2.0 licensing makes the model broadly usable without the limitations found in some other 'open' models. A 64,000-token context window and native function calling make the model especially useful for real software applications. Although benchmarks have limits, early user feedback suggests the model’s performance matches the hype.

Data Points: Total parameters: 141 billion - The full Mixtral 8x22B model contains eight 22B experts, for a total of 141B parameters. Active parameters per inference: 39 billion - Only about 39B parameters are used for a given response, according to the episode. Estimated efficiency gain: About 75% cost savings and 75% time savings - Claimed benefit of using only a subset of experts at inference instead of the full model. Earlier dense model size: 7 billion parameters - Mistral previously released a standalone dense model. Earlier MoE model size: 8 x 7 billion parameters - Mistral’s prior mixture-of-experts release. Benchmark referenced: MMLU - Named as the most popular benchmark and used to compare Mistral models against competitors. Context window: 64,000 tokens - Mixtral 8x22B’s stated long-context capability. License: Apache 2.0 - Described as the most permissive open-source license. Languages mentioned: French, German, Spanish, Italian - The model is said to outperform other open-source models on these language tasks.

Pivotal Quotes: "Mixtral 8x22B uses only 39 billion of its 141 billion total model parameters, saving about 75% of the cost and 75% of the time" — John Crohn: Explaining the practical inference efficiency of mixture-of-experts models. "It also outperforms all other open source models on coding and math benchmarks as well." — John Crohn: Summarizing the model’s performance across technical tasks. "Mixtral is actually releasing the Mistral 8x22B under Apache 2.0 license, which is the most permissive open source license." — John Crohn: Highlighting the model’s unrestricted open-source availability.

Implications: Mixtral 8x22B raises the bar for open-source LLMs, giving developers a fast, capable, and permissively licensed option for multilingual, coding, and math-heavy applications. It may accelerate adoption of open models in production.