The AI Email Assistant I've Been Waiting for, with Andrew Lee of Shortwave

Episode Summary

Executive Summary: Andrew Lee explains how Shortwave evolved from a collaborative email concept into an AI-powered email assistant built on full inbox ingestion, search, retrieval, and drafting. The conversation focuses on the product vision, RAG architecture, model choices, latency/cost trade-offs, and why email is becoming a knowledge base rather than just a messaging tool.

Main Topics: Origin and product thesis (Priority: 5/5): Shortwave began in 2020 as a response to centralized communication concerns, Google Inbox’s shutdown, and Lee’s long history with email infrastructure. The team initially pursued collaborative email but pivoted to individual email use when that proved more viable. AI assistant product experience (Priority: 5/5): The core experience is a sidebar chat assistant that can search email, synthesize answers, check calendar context, and draft responses in the user’s style. It aims to feel like a highly capable executive assistant embedded in email. RAG and inbox indexing architecture (Priority: 5/5): Shortwave imports entire inboxes to its own servers, stores data in Elasticsearch and Pinecone, and uses embeddings plus reranking to answer user questions. This pre-indexing is expensive but enables fast downstream retrieval and richer AI features. Agent workflow and model orchestration (Priority: 5/5): The assistant does not rely on a single end-to-end agent loop. Instead, it runs multiple steps: tool selection, query reformulation, feature extraction, keyword/metadata search, vector search, reranking, and a final GPT-4-based response generation step. Model stack and fine-tuning (Priority: 4/5): Shortwave uses GPT-4, GPT-3.5, fine-tuned GPT-3.5, Mistral, Instructor XL, and a Microsoft-trained cross-encoder. Fine-tuning on email-completion tasks improved formatting, tone, and factual completion for autocomplete. Cost, latency, and product timing (Priority: 4/5): Lee argues the best time to launch is right before a product seems possible. Shortwave is intentionally operating ahead of economic efficiency, spending heavily on inference and infrastructure because the trendline suggests the market will soon justify it. Future of email and collaboration (Priority: 5/5): Lee sees email evolving into a knowledge base that can proactively help users manage work, triage information, and eventually support team-wide intelligence and cross-assistant collaboration. He also expects AI-generated spam to make relationship/context signals more important.

Key Arguments: Email has historically been underutilized because its contents were hard to query beyond basic search; LLMs unlock the latent value of this unstructured corpus. Shortwave’s full inbox ingestion is expensive but strategically important because server-side data enables fast retrieval, embedding, and AI-assisted workflows. A multi-step retrieval pipeline outperforms a single monolithic agent call for this use case because quality and context control matter more than simplicity. General-purpose chat is useful for discovery, but specific UX patterns like autocomplete and drafting need specialized optimization for speed and reliability. Fine-tuning works best when trained on realistic task data; Shortwave found email-completion training on real sent emails generalized well and improved formatting. The product should be launched before economics fully work if the trendline is clear; venture funding can bridge the gap until model and infrastructure costs fall. Email will shift from a pile of messages to a structured organizational memory and operational knowledge base for individuals and teams. AI will increase email volume and personalization, making social context and trusted relationships more important signals for filtering and prioritization.

Data Points: Company start year: 2020 - Shortwave was started at the beginning of 2020 after Lee decided to build a new email product. Inbox import time: From a few minutes to a few days - Depending on inbox size, initial ingestion and reindexing can take this long. Inbox size mentioned: 3–5 million emails - Lee notes that some users have inboxes this large, making indexing a significant process. Production model count: 6 models - Shortwave said it runs six models in production across OpenAI and self-hosted infrastructure. OpenAI models used: GPT-4, GPT-3.5, fine-tuned GPT-3.5 - These are part of the production stack, with GPT-4 used for final assistant responses and GPT-3.5 variants for other tasks. Self-hosted models used: Instructor XL, Microsoft cross-encoder, Mistral - These support embeddings, reranking, and summaries. LLM calls per assistant query: About 10 - Lee explains that each user query triggers multiple model calls across tool selection, rewriting, feature extraction, reranking, and response generation. Serial LLM steps: 5 serial calls - Tool selection, query rewrite, feature extraction, reranking, and final answer generation are described as serial stages. Fine-tuned autocomplete approach: Based on user’s own sent emails - The team trains on emails with sections removed and the missing text as the target completion. Cost dynamics: Loses money on every customer (current stage) - Lee says the subscription price does not yet cover the cost of heavy inference and infrastructure work.

Pivotal Quotes: "The right time to launch a feature or launch a product is right before it seems possible." — Andrew Lee: Lee describes his philosophy for shipping AI products before the economics and market consensus fully catch up. "For the entire history of email clients, the data in that email has been completely inaccessible." — Andrew Lee: He explains why LLMs change email from a basic communication tool into a searchable knowledge base. "What if there was like a human sitting next to you in a chair, watching you do your email, who knew everything about what's on your screen?" — Andrew Lee: He characterizes the Shortwave assistant as an executive-assistant-like copilot for email and calendar work.

Implications: Shortwave shows how AI can turn email into a searchable work memory, but the economics, reliability, and spam arms race are still evolving. The broader lesson: useful AI products may need deep infrastructure, not just model wrappers.