97% Cheaper, Faster, Better, Correct AI — with Varun Mohan of Codeium

Episode Summary

Executive Summary: Varun Mohan explains how ExaFunction was built to solve GPU inefficiency in deep learning infrastructure and how that expertise led to Codium, a code-completion product optimized for low latency, high correctability, and strong unit economics. The conversation covers model serving, evaluation, scaling laws, context retrieval, and why domain-specific, infrastructure-heavy AI products may outperform generic approaches.

Main Topics: ExaFunction’s origin: GPU infrastructure for deep learning: Varun describes founding ExaFunction after seeing how hard it was to manage and serve deep learning workloads at scale, especially on heterogeneous hardware. The company focused on virtualizing GPUs, improving utilization, and reducing cost for enterprise ML teams. Why Codium emerged from infrastructure work: Codium was born when the team realized their serving stack and compiler optimizations could support a consumer-facing code completion product with strong performance and economics. Early internal use of Copilot on C++/CUDA showed the value of low-latency, high-utility completions. Product design: latency, quality, and correctability: Varun argues that successful LLM products need the right balance of speed, output quality, and how easily users can correct mistakes. He emphasizes that code completion works because users can quickly accept, edit, or ignore suggestions, unlike broader multi-file generation tasks. Model scaling, Chinchilla, and serving constraints: The discussion covers scaling laws, compute-optimal training, and why bigger models are not always practical if they cannot be served efficiently. Varun notes that latency, memory footprint, and hardware fit often matter as much as raw model quality. Context retrieval and the future of LLM systems: Varun discusses Retro-style retrieval, embedding databases, and the idea that future models will dynamically fetch context from external stores. He expects context augmentation and retrieval systems to become central to LLM applications. Evaluation, data quality, and vertical specialization: He stresses that teams must build their own evals and data-cleaning pipelines because generic benchmarks can be contaminated or misleading. He believes high-quality, task-specific data and evaluation are essential for reliable vertical AI products. AI product strategy and future expansion: Varun sees code completion as only the first step and believes startups should focus on one thing well before expanding into other domains like legal or broader enterprise workflows. He is bullish on specialized AI apps but cautious about overextending too early.

Key Arguments: GPU utilization is a major hidden inefficiency in ML infrastructure, and solving it can dramatically reduce costs. Most enterprises should not reinvent model architectures; they should use strong base models and focus on deployment, latency, and workflow fit. Code completion is a strong LLM use case because it has low latency, high correctability, and clear user value. Many LLM products fail because they generate too much at once, making errors hard to correct and eroding trust. Bigger models are often better, but only if they can be served within memory and latency constraints. Future LLM systems will likely rely on retrieval and external context rather than only longer prompts. Teams need custom evals and data cleaning for their specific workload; generic benchmarks are insufficient and can be contaminated. Vertical AI products will win by solving a narrow task extremely well rather than trying to be general-purpose from day one.

Data Points: User count: over 10,000 users - Codium’s reported adoption at the time of the interview Daily active users: thousands of daily actives - Codium usage and retention Growth rate: around 4% to 5% day over day - Codium’s growth, driven mainly by word of mouth Cost reduction: 97% cost cut - A customer reportedly went from 30 clients to one GPU using dynamic multiplexing Client consolidation: 30 clients to 1 GPU - Example of improved GPU utilization at ExaFunction Copilot price: $10 per month - Referenced as a benchmark for code completion pricing Copilot code retention estimate: 60% to 70% - Mentioned as the share of generated code that ends up checked into repos Model size: 12 billion parameters - Copilot size referenced in the scaling discussion Model size: 16 billion parameters - Salesforce CodeGen size referenced in the scaling discussion Model size: 175 billion parameters - GPT-3 size referenced in the scaling discussion Compute-optimal ratio: about 20x parameters - Chinchilla-style scaling rule discussed for compute-optimal training Training bandwidth: 3200 gigabits/second - Inter-node bandwidth cited for H100 systems Memory bandwidth comparison: 400 gigabytes/second - Equivalent inter-node bandwidth cited as roughly double CPU memory bandwidth Optimizer state size: 14x the size of the model - Used to explain why large models require heavy sharding during training Embedding database scale: 10x the number of training tokens - Retro-style retrieval example with a very large external embedding store Enterprise contract size: 3,500 lawyers - Referenced as a large legal AI deployment by Harvey and Allen & Overy

Pivotal Quotes: "I think it comes down to for most people. One of the things about CPUs that's really nice is with containers, right? ... It's not really the same with GPUs." — Varun Mohan: Explaining why GPU utilization is fundamentally harder than CPU/container utilization "The reality is, you know, if I were to take a product like Copilot or Codium, the latency is very low. The quality I think is good enough for the task, but the correctability is very easy." — Varun Mohan: Summarizing the three key product criteria for successful LLM applications "I think the most important thing to take away is there's probably a lot out there if people continue to tinker." — Varun Mohan: Closing takeaway on AI product opportunity and experimentation

Implications: The episode suggests the next wave of AI winners will be specialized products built on strong infrastructure, custom evals, and retrieval-aware systems. For builders, the lesson is to optimize for real user workflows, not just model size or demo quality.