S12 E28: The AI Throughput Illusion: Why Splurging on Expensive Models Fails to Ship Code and How to Measure Real Engineering Output with Emilie Schario, Co-Founder & Head of Product & Engineering at Kilo Code

Episode Summary

Executive Summary: Emily Sherio, co-founder and head of product/engineering at Kilo, explains how the company is building an open-source, open-model AI coding harness centered on “model freedom.” The conversation covers Kilo’s fast-moving product strategy, trade-offs like sunsetting App Builder, senior distributed team structure, an incident where Kilo itself went down and exposed operational dependencies, and Sherio’s view that the future of AI coding is multi-model and highly specialized.

Main Topics: Kilo’s mission and model freedom (Priority: 5/5): Kilo is positioned as an agentic engineering platform that lets users choose from 500+ models, unlike tools locked to a single vendor. Sherio emphasizes flexibility as a core differentiator. Product definition in a rapidly changing AI market (Priority: 5/5): The team treats the MVP as a moving target because AI coding expectations evolve quickly; features like loops and model workflows can become table stakes almost overnight. Strategic trade-offs and sunsetting App Builder (Priority: 5/5): Kilo had to narrow focus after broadening its ICP ambitions beyond developers. The company decided to hide and likely sunset App Builder rather than continue supporting a subpar experience. Roadmap and product management in AI (Priority: 4/5): Sherio rejects long-term static planning in favor of a blend of enterprise demand, user feedback, community input, competitive watching, and business-impact prioritization. Team structure, hiring bar, and distributed execution (Priority: 4/5): Kilo’s 35-person globally distributed team is highly senior, product-oriented, and split between product/engineering and go-to-market, with strong emphasis on autonomy and velocity. Operational resilience and the Kilo outage (Priority: 4/5): An outage showed the risk of depending on Kilo itself as an on-call tool. The team had to rethink incident response when the assistant used for troubleshooting was unavailable. Future of AI coding and entrepreneurship advice (Priority: 3/5): Sherio believes the future is multi-model, with specialization by task; she also warns aspiring founders that starting a company is harder and less glamorous than it looks.

Key Arguments: AI coding tools should not be tied to a single model provider; users need the ability to choose the best model for each task. In fast-moving AI product categories, the definition of MVP changes constantly as user expectations and workflows evolve. Companies must make explicit stop-investing decisions when a product area no longer deserves scarce engineering resources. Product roadmaps in AI should be informed by enterprise needs, user feedback, open-source community input, and competitor movements rather than fixed long-range plans. The future of coding assistants is multi-model because specialization is likely to matter for models just as it does for people. A senior, autonomous team can reduce common scaling pains because experienced engineers anticipate problems earlier. Operational tooling should not be assumed to always be available; companies need fallback processes when their own AI systems fail. Starting a company is stressful and often misperceived as glamorous; only the truly committed should pursue it.

Data Points: Company age: About 1.5 years old - Sherio describes Kilo as founded roughly a year and a half ago Team size: 35 people - Current size of the globally distributed Kilo team Model availability: 500+ models - Kilo offers users access to over 500 AI models from multiple providers Engineer average experience: 15 years - Average professional experience of engineers at Kilo Least experienced engineer: 10 years - Sherio says even the newest engineer has 10 years of experience AI outage timing: A couple weeks ago - Sherio references a recent incident where Kilo itself was down CrossFit frequency: 3 times a week - Sherio says she tries to go to CrossFit three times weekly if lucky Children: 3 boys, all five and under - Sherio describes her household and parenting life App Builder status change: A couple of weeks ago / a couple of months ago - She notes uncertainty in timing but says App Builder was hidden from new users and likely headed toward sunset

Pivotal Quotes: "The thing that really differentiates Kilo is what we call model freedom." — Emily Sherio: Explaining Kilo’s main product advantage over single-model coding assistants "The future is definitively multi-model." — Emily Sherio: Describing where she believes AI coding and model usage are heading "My advice every time is a version of don't do it. Don't do it. You don't want to do this. It's way harder than you think." — Emily Sherio: Her answer to a young entrepreneur asking for startup advice

Implications: The episode suggests AI coding platforms will win by being flexible, open, and workflow-native rather than model-locked. It also shows that in AI startups, focus, senior talent, and fallback operations are becoming essential advantages.