801: Merged LLMs Are Smaller And More Capable, with Arcee AI's Mark McQuade and Charles Goddard

Episode Summary

Executive Summary: Mark McQuaid and Charles Goddard of RCAI explained how model merging and related methods can combine the strengths of multiple LLMs into one model without increasing parameter count, enabling cheaper, faster, and more private enterprise AI. They also discussed Spectrum for more efficient training, mixture-of-experts vs. mixture-of-agents, sparse upcycling, and why small specialized models may outperform massive foundation models for many business tasks.

Main Topics: Model merging as next-gen transfer learning: Charles described model merging as combining pretrained weights from multiple neural networks into one model that inherits useful capabilities from each. The hosts framed it as a natural evolution of transfer learning that can reduce cost and simplify deployment. Enterprise monetization and RCAI's business model: Mark explained how RC monetizes open source technology through a secure platform for training, merging, and deploying customer-owned models in private environments, while also supporting an open source ecosystem. He highlighted rapid early revenue and enterprise interest. Evolutionary model merging and MergeKit: Charles detailed evolutionary model merging, where CMA-ES optimizes merge parameters to maximize task performance. This is exposed in MergeKit via merge-kit evolve, making sophisticated merges accessible without extensive manual experimentation. Mixture of experts, sparse upcycling, and mixture of agents: The conversation distinguished sparse MoE architectures from mixture-of-agents workflows. Charles explained how sparse upcycling can transform a dense model into a sparse expert system, while Mark and Charles positioned mixture-of-agents as a more practical and emerging approach. Spectrum for efficient training: Mark introduced Spectrum, RC's efficient training method that targets specific modules and freezes others to reduce training cost and time. It was presented as complementary to merging: train efficiently, then merge for better final performance. Small language models for enterprise AI: Both speakers argued that smaller, specialized models are often a better fit for enterprise use than giant foundation models. They cited lower latency, lower cost, private deployment, and strong performance on narrow tasks as key advantages. Future of AI commoditization: The episode closed with reflections on AI as commoditized intelligence. The speakers suggested that access to capable models will reshape industries broadly and that current capabilities likely underestimate near-future impact.

Key Arguments: Model merging lets teams combine strengths from multiple models into one model of the same parameter size, reducing inference cost and operational complexity. Open source checkpoints can be repurposed to avoid rebuilding broad capabilities from scratch; organizations only need to train for their specific task and then merge in general-purpose strengths. RCAI's platform is commercially viable because enterprises want private, secure, customizable models that keep data inside their own environment. Evolutionary optimization can automate the search for good merge configurations, making model merging more practical for users who lack deep merging intuition. Mixture-of-experts architectures are structurally different from a naive 'one expert per task' mental model; they are more like sparse matrix factorization than discrete subject specialists. Sparse upcycling is powerful but compute-intensive and generally suited to organizations with massive datasets and GPU budgets. Spectrum improves training efficiency by freezing parts of the network and focusing learning on selected modules, lowering cost and time without degrading performance. Small, specialized models can outperform giant foundation models on enterprise-specific tasks because most business use cases do not need broad general intelligence. Owning and deploying a customized model on your own infrastructure can produce substantial cost savings and create an IP moat. The market is increasingly ready for 'everybody does a merge' as model merging becomes normalized in production workflows.

Data Points: Series A funding: $24 million - RCAI announced a Series A round led by Emergence Capital. Seed funding: Just over $5 million - Mark said RCAI raised its seed round in September. Revenue: Nearly $2 million - Mark said RCAI was close to hitting $2M in revenue shortly after launching the product in November. Model size example: 7 billion parameters - Used repeatedly to describe RC Spark, customer models, and example merged models. Model size example: 8 billion parameters - Charles used Llama 3 8B as a concrete example of merge size staying constant after merging. Cost reduction: 40% to 50% faster training - Mark described Spectrum as enabling more efficient continual pre-training. Cost reduction: 10k - Mark claimed RCAI can help train a model for around $10,000 in some scenarios. Cost comparison: Half a million dollars - Mark referenced Mosaic's claim about training a GPT-3/3.5-level model for about $500K as a prior benchmark. AWS GPU cost: About $1/hour - Mark estimated a G5.2XL / A10 instance price on AWS. Annual hosting cost: About $15K/year - Mark estimated running a 7B model on that AWS instance could cost around $15K annually. Inference activation in Mixtral: 2 of 8 experts per token - Charles explained Mixtral uses top-two expert selection. Inference savings: Roughly 25%–30% of activations used - Derived from top-two-of-eight routing plus triage overhead. Open source company milestone: RC Spark outperformed GPT-3.5 on certain evals - Mark described RC Spark as a 7B model that beats GPT-3.5 on some benchmarks. Open source company milestone: RCAGENT 7B beat closed-source models on specific evals - Mark introduced RCAGENT as a 7B function-calling model with strong benchmark results.

Pivotal Quotes: "Model merging is a family of techniques that let you take the pre-trained weights of neural networks and combine them into a single network that captures some of the strengths or abilities of all the networks that you threw into that melting pot." — Charles Goddard: Charles defining model merging early in the episode. "Why have your 200, 300, 500 billion parameter model for your HR use case, as an example, or for your customer support LLM? It's just not needed." — Mark McQuaid: Mark arguing that specialized small models are often enough for enterprise use cases. "I think every organization will want to own and customize their own LLM at some point soon." — Mark McQuaid: Mark explaining the business rationale behind RCAI's private deployment and platform strategy.

Implications: The episode suggests enterprise AI is shifting toward smaller, privately owned, task-specific models trained efficiently and improved through merging. For practitioners, the opportunity is lower cost, lower latency, and more control; for the industry, it points to a future where model composition becomes standard.