Episode Summary
Executive Summary: The conversation centers on Sphere’s AI-powered compliance system for global sales tax and VAT/GST, and why RAG remains essential when accuracy and exact citations matter. Alex Bocut explains how Sphere uses document ingestion, semantic chunking, dense+sparse retrieval, re-ranking, and reinforcement fine-tuning to help tax experts review and approve determinations faster while preserving legal correctness.
Main Topics: Why RAG is still necessary for high-stakes compliance (Priority: 5/5): Bocut argues that while long-context models and agentic search may work for some tasks, Sphere needs highly reliable retrieval because tax determinations require exact citations and very high accuracy. Sphere’s tax compliance product and TRAM workflow (Priority: 5/5): Sphere helps companies manage sales tax and VAT/GST. TRAM is an internal system that drafts taxability determinations for human tax experts, who review outputs before they are loaded into a deterministic tax engine. Data ingestion across messy legal sources (Priority: 4/5): Sphere ingests legislation, rulings, and bulletins from many jurisdictions in varied formats including HTML, structured PDFs, image PDFs, spreadsheets, text, and Word docs, requiring OCR and format-specific parsing. Retrieval architecture: semantic chunking, dense and sparse search (Priority: 5/5): The team preserves document hierarchy, chunks by legal structure, and combines embeddings with sparse full-text-style retrieval to improve citation accuracy and relevance. Reasoning models and reinforcement fine-tuning (Priority: 4/5): Sphere saw gains from OpenAI’s o1 reasoning model and later used reinforcement fine-tuning with expert feedback on previously missed determinations to improve accuracy further. Human review as legal validation, not labeling (Priority: 4/5): Bocut emphasizes that expert review is more like legal review than data labeling because the outputs are legal claims about taxability and require citations and contextual judgment. Future expansion: taxonomy generation, scraping, and migration tools (Priority: 3/5): Sphere aims to automate product taxonomies, improve ongoing legal scrapes, and help customers migrate from competitors by mapping external classification systems into Sphere tax codes.
Key Arguments: RAG is not dead for Sphere’s use case because tax compliance depends on exact source citations and extremely high accuracy, not just approximate answers. Long-context models help, but they do not yet eliminate the need for carefully tuned retrieval and re-ranking in legal/compliance settings. Semantic chunking based on legal structure outperforms naive fixed-size chunking because it preserves meaning and citation fidelity. Combining dense embeddings with sparse retrieval improves citation-side accuracy, especially when specific legal terms and citations matter. Human tax experts function as legal reviewers, ensuring correctness before results enter the deterministic tax engine. Reasoning models improved performance on complex taxability decisions because they better handle long, nuanced legal context. Reinforcement fine-tuning worked well because Sphere had strong training signals: missed examples, corrected answers, and detailed expert feedback. The biggest business win is reducing review time and eventually getting accurate enough to rely on sampling instead of full human review.
Data Points: Review time per determination: About 9 seconds - Average time for a tax expert to review each model-generated determination Productivity gain from TRAM: Almost 2 orders of magnitude faster - Internal experts moved through tax review work much faster than a fully human workflow Series A funding: Raised last year from Andreessen Horowitz - Growth milestone mentioned by Bocut Model improvement benchmark: Big jump with OpenAI o1 in December 2024 - Reasoning model improved taxability determination quality Human review target: Random sampling instead of every determination - North star is high enough accuracy to reduce full manual review Expansion scope: Support for global jurisdictions including U.S., Canada, and international countries - Sphere aims to scale beyond the U.S. tax incumbents Context usage: Longer context helped but is not enough alone - Bocut says models can handle more tokens now, enabling broader retrieval expansion
Pivotal Quotes: "for us, or at least for this particular problem, because we are so sensitive to accuracy and we're so sensitive to the exact right citation, as of today, I don't think agents are just searching over the file system, grepping over it is at a point where we could switch over" — Alex Bocut: Explaining why RAG remains necessary for Sphere despite larger context windows and agentic search "TRAM allows our internal tax experts to move almost two orders of magnitude faster through this process with less errors than the traditional, just fully human, you know, focused approach" — Alex Bocut: Describing the internal impact of Sphere’s AI-assisted tax review system "We think of it more as like a Legal review" — Alex Bocut: Clarifying that expert oversight is a correctness check on legal claims, not just data labeling
Implications: For compliance, legal, and finance AI, retrieval quality and citation fidelity remain critical even as context windows grow. The likely winning pattern is hybrid: structured ingestion, retrieval, reasoning, and human-in-the-loop validation.