Episode Summary
Executive Summary: Codium co-founders Varun and Anshul explain how their GPU-infrastructure company Exafunction led them to build an AI toolkit for software developers. Codium combines autocomplete, codebase-aware chat, and search, with an emphasis on smooth IDE workflows and enterprise self-hosting. They argue that private code, broader context, and local fine-tuning can make AI assistance more useful and secure, while stressing that humans remain in control and product bets must match current technical capabilities.
Main Topics: From GPU infrastructure to Codium: The founders' work at autonomous-vehicle company Nuro and later Exafunction exposed the challenges of GPU scarcity, cost, and utilization. Their infrastructure expertise became a foundation for Codium, an AI coding product. Codium as a broader developer toolkit: Rather than focusing only on autocomplete, Codium offers IDE-integrated chat, codebase-aware search, and assistance with tasks such as explaining, refactoring, documenting, and testing code. The goal is to support more of the software-development lifecycle. Enterprise privacy and self-hosting: Codium can be deployed in a customer's environment, including air-gapped settings, so company code does not leave its tenant. Self-hosting also enables personalization using internal code and knowledge. Context and fine-tuning for relevant code suggestions: The guests argue that generic models have limited access to a company's private code and that low-latency autocomplete cannot use vast amounts of context at inference time. They use broader codebase context and local fine-tuning to improve relevance and reduce hallucinations. Product workflow and developer use cases: IDE-integrated controls and actions streamline common tasks by avoiding manual copying between a chat interface and editor. The founders distinguish autocomplete as an accelerator for known tasks from an exploratory aid when developers are unsure how to proceed. Human oversight and careful product bets: The founders see AI as an assistant that increases developer productivity, not as an imminent replacement for developers. They caution against building features around exciting demos before the underlying technology reliably solves real user problems.
Key Arguments: GPU workloads are unusually difficult to virtualize because GPU memory is limited and the hardware is expensive; better utilization can therefore make large-scale AI applications more practical. The founders saw demand for coding assistance alongside a gap in enterprise adoption: many developers liked tools such as Copilot personally but could not use them at work because of company restrictions. Autocomplete is only one part of software development. Integrating chat, search, and code actions into the IDE can reduce workflow friction and cover more developer tasks. Private company code can be more relevant to a company's developers than public training data. Local fine-tuning and access to codebase context can make suggestions more specific while keeping IP inside the company's environment. A free individual product provides broad real-world usage and feedback, while also pushing the company to make its inference infrastructure efficient; those operational learnings can benefit enterprise deployments. Autocomplete's value includes both speeding up tasks developers already understand and helping them explore unfamiliar APIs or approaches; the latter depends on supplying useful context without producing distracting suggestions. AI coding tools are best understood as assistants with humans in control. Companies should prioritize reliable, useful capabilities over features that are impressive in demonstrations but not yet dependable.
Data Points: Exafunction GPU management: Over 10,000 GPUs in one GCP region - Varun said Exafunction's virtualization software managed this scale for enterprise workloads. H100 chip cost: Well over $30,000 - Given as an example of how expensive GPU hardware is compared with CPUs. Codium active developers: Over 100,000 - Anshul described the size of the free individual plan's active developer base. Codium daily processing: Over 10 billion code tokens, or over a billion lines of code - Varun cited the product's daily processing volume as evidence of its infrastructure scale. Autocomplete's share of product usage: More than 90–95% - Varun said autocomplete accounts for the vast majority of Codium usage. Autocomplete context: About 150 lines; Codium says it allows double Copilot's context - Anshul described the constraints of low-latency autocomplete, while Varun said Codium provides twice the context Copilot does. Example enterprise codebase: 10 million lines of code, or about 100 million tokens - Used to illustrate the scale of private company code that can inform local fine-tuning. GPU inference capacity: About 300 teraflops for a modern A100 - Varun used this figure to explain the compute capacity available when GPUs are efficiently utilized.
Pivotal Quotes: "Let's try to build an AI toolkit rather than an AI autocomplete tool." — Anshul: Summarizes Codium's effort to support multiple tasks across software development instead of offering autocomplete alone. "The answer is always no." — Anshul: In response to whether enterprise customers' IP leaves Codium; he said deployments can be fully self-hosted, including air-gapped environments. "You just 100% shouldn't be building a tool when the technology just isn't there yet." — Varun: Explains why the company evaluates feature ideas against current technical reliability and customer usefulness.
Implications: For developers, IDE-integrated AI may help with both routine coding and exploration, but good context and workflow matter. Enterprises can pursue AI assistance while retaining code privacy through self-hosting and personalization. The field's winners will need efficient infrastructure, useful products, and realistic expectations about model capabilities.