AI Inference: Good, Fast, and Cheap, with Lin Qiao & Dmytro Ivchenko of Fireworks AI

Episode Summary

Executive Summary: Fireworks AI co-founders Lynn Chow and Dimitro Ifchenko explain how their inference platform helps developers deploy generative AI with lower latency, better quality, and lower total cost of ownership. The discussion covers their Meta/PyTorch background, the economics of inference, hardware and parallelization trade-offs, LoRA fine-tuning, automatic optimization, and why their competition is ultimately OpenAI/Anthropic rather than other infra vendors. They also discuss their Stability AI partnership for Stable Diffusion 3 and the differences between text and image inference.

Main Topics: Fireworks AI’s mission and market positioning (Priority: 5/5): Fireworks is positioned as an inference platform that abstracts hardware complexity, speeds experimentation and production deployment, and optimizes for low latency, high quality, and low TCO rather than subsidized pricing. Lessons from Meta and PyTorch (Priority: 5/5): The founders’ prior work on PyTorch at Meta shaped their philosophy: prioritize developer UX, hide backend complexity, and build dedicated infrastructure services rather than assuming a library swap is enough. Inference optimization and hardware trade-offs (Priority: 5/5): The conversation dives into GPU/ASIC trade-offs, CUDA/Triton/Tensor Core programming, quantization, and why new hardware generations continuously force re-optimization of inference stacks. Fine-tuning and LoRA as a product strategy (Priority: 5/5): Fireworks emphasizes fast, lightweight fine-tuning—especially LoRA—to adapt smaller models to narrow enterprise tasks, with shared-base-model inference as a major production efficiency gain. Quality competition with frontier model providers (Priority: 4/5): The founders argue that open and closed small models will converge in capability over time, but enterprises still need fine-tuning and workload-specific optimization; their real long-term competitors are OpenAI and Anthropic. Automatic optimization and DBMS analogy (Priority: 4/5): Fireworks envisions an optimizer-like system that learns workload patterns and self-tunes deployments over time, similar to how databases evolved from manual DBA tuning to automated query optimization. Stability AI partnership and image-generation inference (Priority: 4/5): In a bonus segment, they discuss Fireworks becoming the exclusive inference provider for Stable Diffusion 3 and how transformer-based image models create partial reuse of text-inference techniques while still requiring unique quality-sensitive optimizations.

Key Arguments: Inference providers must compete on latency, quality, and TCO—not just raw low price—because pricing subsidies are unsustainable. Generative AI applications are interactive products, so time-to-first-token and stable latency are critical, especially for consumer and voice use cases. Fireworks can abstract away the complexity of hardware choice, sharding, and deployment so developers can focus on product development. Open-source and closed-source small models are likely to converge in quality, so the advantage shifts to ecosystem, fine-tuning, and deployment efficiency. Fine-tuning is often the right answer for narrow enterprise tasks because many use cases can be solved with only hundreds of examples. LoRA dramatically lowers training cost and can also reduce inference cost by allowing many adapters to share one base model. Hardware and precision changes force continual kernel and compiler optimization; there is no permanent “solved” inference stack. Fireworks’ strategic competition is frontier model APIs like OpenAI and Anthropic, because they set the quality bar customers care about. Image generation and text generation share transformer-based optimization patterns, but image models require different handling of quality, quantization, and denoising workflows. The company’s long-term vision is an optimizer that automatically chooses the best deployment configuration based on workload data, reducing the need for manual tuning.

Data Points: Meta H100 deployment: more than 600,000 H100 GPUs - Used as an example of hyperscale AI infrastructure scale and power consumption Power comparison: roughly twice the size of San Francisco - Approximate comparison for the power needed to run 600,000 H100s PyTorch production scale: 50 trillion inference per day - Dimitro described Meta’s PyTorch platform scale across multiple data centers PyTorch data centers: 50 data centers - Infrastructure scale supporting Meta’s inference workload LoRA parameter reduction example: 10 million parameters reduced to 7,000 - Illustrative matrix decomposition example used to explain low-rank adaptation Reduction factor in LoRA example: more than 1,000x - Claimed savings from low-rank factorization in the worked example Typical fine-tuning dataset size: hundreds to around 1,000 examples - The hosts discussed that narrow tasks often need only small datasets Token pricing comparison: $0.20 per million tokens - Mentioned as Fireworks pricing for small models up to 16B parameters Older OpenAI pricing comparison: $0.06 per 1,000 tokens - Compared to earlier GPT-3-era pricing Anthropic Haiku input pricing: $0.25 per million tokens - Used as a direct small-model competitor benchmark Image generation default steps: 50 denoising steps - Stable Diffusion 3 default generation setting discussed in bonus segment Image generation turbo steps: 4 denoising steps - Faster but lower-quality variant discussed for image generation Fine-tuning/output ratios: 10:1 input-to-output ratio - A typical workload ratio described for text-generation use cases Batch sizes for generation: 16 to 10 - Generation batch sizes were described as relatively small compared with pre-fill Batch sizes for pre-fill: tens of thousands - Pre-fill can batch far more tokens than generation Model size threshold: up to 16 billion parameters - Fireworks’ pricing/quality discussion referenced a small-model tier up to this range

Pivotal Quotes: "We cannot just directly apply the techniques we learned from text models on image model because it has quality implications." — Dimitro Ifchenko: Explaining why image model optimization cannot simply reuse text-model methods "Our real competition is actually OpenAI and Anthropic." — Lynn Chow: Describing Fireworks’ strategic framing for quality and customer choice "You don't worry about it. We handle it all for you. So you just focus on your product application development." — Lynn Chow: Summarizing Fireworks’ value proposition as an abstraction layer over infrastructure complexity

Implications: Inference is becoming a specialized infrastructure layer with its own optimization stack, business economics, and product design. For teams, the winning pattern is likely workload-specific tuning plus automated deployment optimization, not generic one-size-fits-all models.