Retrieval After RAG: Hybrid Search, Agents, and Database Design — Simon Hørup Eskildsen of Turbopuffer

Episode Summary

Executive Summary: The episode traces TurboPuffer’s origin from a ReadWise recommendation experiment to a database/search company built for AI-era workloads. Simon Eskildsson argues that modern search infrastructure must be object-storage-first, NVMe-accelerated, and query-rich over time. The conversation covers product scope, customer wins at Cursor and Notion, hiring philosophy, pricing, and future expansion beyond vector search into full-text and broader database features.

Main Topics: TurboPuffer’s origin and product definition (Priority: 5/5): Simon explains TurboPuffer as a search engine for unstructured data that supports full-text and vector search, positioned as the external knowledge layer for AI systems. Why a new database architecture is needed (Priority: 5/5): He lays out three conditions for a big database company: a new workload, a new storage architecture, and an expanding feature set over time. TurboPuffer is built around object storage, NVMe SSDs, and minimal state. From Shopify and ReadWise to the founding insight (Priority: 4/5): His decade at Shopify scaling infra and a consulting stint at ReadWise exposed the pain of expensive Elasticsearch/vector workflows, inspiring the low-cost architecture behind TurboPuffer. Early customers and product-market fit (Priority: 5/5): Cursor and Notion served as formative design partners, validating the cost/performance economics and driving early implementation choices, including cross-cloud latency work. Workload evolution in the AI era (Priority: 4/5): The discussion shifts from single RAG queries to agentic, parallel, hybrid search patterns, increasing demand for concurrency, lower query pricing, and mixed retrieval methods. Hiring philosophy and the P99 engineer (Priority: 4/5): Simon describes a talent-dense culture built around obsessive, high-agency engineers who can bend systems to their will, with interviews designed to surface those traits. Future roadmap and expansion (Priority: 4/5): TurboPuffer plans to deepen full-text search, scale to 100B-vector/data-set sizes, improve dashboards, and potentially add adjacent query patterns like OLAP, logs, time series, and graph-style queries.

Key Arguments: AI creates a new universal workload: every company will need searchable, AI-connected unstructured data. A modern database can be cheaper and simpler by putting durable data in object storage and only inflating hot data into NVMe/DRAM. Legacy search systems like Elasticsearch/Lucene often block product ambition because they are hard to scale and hard to adapt. TurboPuffer’s architecture was enabled by cloud primitives that only recently matured: NVMe SSDs in cloud, S3 consistency, and compare-and-swap capabilities. Customers buy TurboPuffer not just for capability but because it improves economics enough to make features or products viable. Agentic workloads are increasing concurrency dramatically, which favors systems designed for many parallel queries per round trip. Hiring should be extremely selective and favor people who have demonstrated unusual technical leverage, not just general competence. The company should stay focused on search first, even while observing adjacent workloads that might justify future expansion.

Data Points: Shopify tenure: Almost a decade - Simon worked on infrastructure at Shopify before founding TurboPuffer. ReadWise infrastructure budget: About $5,000/month - Total infrastructure spend for ReadWise before adding the new recommendation/search feature. Projected feature cost: About $30,000/month - Estimated cost to run embeddings and vector search for one ReadWise feature. Cost reduction at Cursor: 95% - TurboPuffer reportedly cut Cursor’s cost by 95% during migration. TurboPuffer ANN V3 scale: 100 billion vectors - Current ANN version can search this scale with low latency. ANN V3 latency: P50 ~40 ms; P99 ~200 ms - Performance figures given for TurboPuffer’s vector search engine. Query pricing change: Reduced 5x - TurboPuffer is lowering query pricing to support higher-concurrency agent workloads. Cloud storage timing: S3 consistency in Dec 2020 - Simon notes this as a key enabling platform change. Cloud storage timing: NVMe SSDs in cloud around 2017 - He cites this as another prerequisite for TurboPuffer’s architecture. Compare-and-swap availability: S3 late 2024 - Needed for metadata coordination without a separate consensus layer. Latency example: ~14 ms vs ~7 ms - Cross-cloud latency affected Notion deployments and round-trip planning. Fiber cost: About $5,000 - TurboPuffer bought dark fiber to reduce latency between cloud regions. Tea collection: 200 teas - Simon keeps an Airtable tracking teas tried over 15 years. Launch setup: Single 8-core machine - Initial TurboPuffer deployment was a Rust binary running in tmux on one machine.

Pivotal Quotes: "if this doesn't have PMF by the end of the year, like, we'll just like return all the money to you" — Simon Eskildsson: He describes his unusually blunt pitch to investor Locky while raising early money. "why hasn't anyone built a database where you just put everything on object storage, and then you puff it into NVMe when you use the data" — Simon Eskildsson: This was the napkin-math insight that led to TurboPuffer’s architecture. "the default should not be, we're going to hire this person. The default should be, we're definitely not hiring this person" — Simon Eskildsson: He explains the company’s extremely selective hiring philosophy for P99 engineers.

Implications: TurboPuffer reflects a broader shift toward AI-native data infrastructure: cheaper, more parallel, hybrid retrieval systems that unify search and context for agents. If the trend continues, search stacks will become more database-like, and database stacks will become search-like.