Episode Summary
Executive Summary: Zane Asgar explains Gimlet’s approach to heterogeneous inference for agentic AI: breaking workloads into graph components, routing them across mixed hardware, and using compilation plus LLM-assisted kernel optimization to cut cost per token and latency. He argues inference is better served by scalable, commodity-oriented systems than supercomputer-style training racks, and that heterogeneity is becoming a practical advantage in data centers, sovereign clouds, and edge-to-cloud deployments.
Main Topics: Gimlet’s origin and mission (Priority: 5/5): Zane outlines Gimlet’s goal of making AI workloads at least 10x more efficient, starting from edge devices and shifting toward data center inference because the market is hotter and the technology generalizes well. Agentic inference as a heterogeneous systems problem (Priority: 5/5): Agentic workloads combine model calls, CPU work, database access, and orchestration, so Gimlet treats the application as a full dataflow graph rather than isolated model invocations. Workload disaggregation and hardware right-sizing (Priority: 5/5): The system partitions workloads into performance-critical and less critical pieces, placing them on the most appropriate hardware to optimize cost, memory bandwidth, and latency. Compilation and LLM-assisted kernel optimization (Priority: 4/5): Gimlet compiles workloads to target hardware using MLIR/Torch-MLIR, then uses LLMs in a hardware-in-the-loop loop to synthesize faster kernels while validating correctness offline. Kubernetes, DRA, and orchestration across heterogeneous clusters (Priority: 4/5): The product runs on Kubernetes and uses dynamic resource allocation concepts to slice GPU capacity and route workloads based on availability, caching, and model placement. Economic case for heterogeneous inference (Priority: 5/5): Zane argues that mixing hardware generations and vendors can significantly improve TCO, especially when older or cheaper accelerators can absorb non-critical work while premium GPUs handle the hot path. Inference vs. training hardware philosophy (Priority: 5/5): He distinguishes inference from training, saying training is trending toward vertically integrated supercomputer-like racks, whereas inference should scale out on commodity hardware with interoperability across vendors.
Key Arguments: Inference workloads should be treated as end-to-end applications, not just model calls, because orchestration overhead and heterogeneous compute dominate real-world performance. The biggest gains come from workload disaggregation and smarter scheduling: putting critical paths on top-tier accelerators and filling in with cheaper hardware elsewhere. Heterogeneity matters both within a vendor and across vendors because compute, memory bandwidth, memory capacity, networking fabric, and caching all change the cost/performance tradeoff. Kubernetes is a good substrate for this problem; Gimlet adds the missing resource and routing intelligence rather than rebuilding cluster management from scratch. LLM-generated kernels can be useful for optimization, but only when verified offline with profiling and numerical tests due to correctness and floating-point concerns. The most valuable use of LLMs today is to automate repetitive systems-engineering work, not to make live, unguarded control decisions. Older or less-optimized hardware can deliver major TCO wins when paired with a heterogeneous scheduler, letting expensive GPUs focus on the highest-value tasks. Inference will likely diverge from training: training keeps becoming more supercomputer-like, while inference benefits from scale-out, commodity, heterogeneous infrastructure.
Data Points: Target efficiency improvement: 10x - Gimlet’s original mission for AI workload efficiency Revenue: eight figures - Zane says Gimlet is already doing this much revenue Public cloud launch timing: Q1 - Planned usage-based pricing launch for the cloud product GPU utilization pain point: 30% utilized - Example of wasted GPU capacity on underutilized clusters H100 kernel-level gains: single-digit percentage improvements - Observed on highly optimized Hopper/H100 hardware RTX 6000 / B40 / B200-class gains: 20–40% improvements - Reported on less-explored or newer GPU targets Mac/AMD/Intel kernel gains: up to 2x - Potential improvement versus relatively unoptimized code paths Rack power in generic data centers: ~20 kW per rack - Typical power budget limiting dense GPU deployment Back-of-rack cooling threshold: 25–30 kW+ - Where rear-door heat exchangers or liquid cooling become necessary Direct liquid cooling threshold: 100 kW+ racks - High-density rack environments for latest-generation systems Next-generation NVIDIA rack power: ~600 kW - Kyber rack example described as almost a data center in one rack Early access cloud customers: handful - Public cloud product is not broadly launched yet Deployment model: two vendors, maybe three at most - Typical number of vendors seen in customer data centers
Pivotal Quotes: "We're building these like fully vertically integrated systems. I'm not sure that's the route for inference." — Zane Asgar: Arguing that inference infrastructure should favor scale-out heterogeneity rather than supercomputer-style integration "The vast majority of benefits come in because knowing what hardware is available and how to split it up, you can go ahead and start packing these things in pretty tightly into the different resources." — Zane Asgar: Explaining why workload disaggregation drives most of the cost and performance gains "Inference is much better served as a large-scale workload where you can utilize a bunch of relatively commodity hardware and be able to scale out efficiently." — Zane Asgar: Contrasting inference architecture with training architecture
Implications: The conversation suggests inference infrastructure is shifting toward heterogeneous, cost-aware orchestration across mixed hardware. For builders, the winning stack may combine Kubernetes, compiler tooling, and guarded AI automation to reduce cost per token without sacrificing reliability.