Episode Summary
Executive Summary: Kelly Hong explains Chroma’s generative benchmarking approach for retrieval/RAG: instead of relying on overfit public benchmarks like MTEB, it builds representative eval sets from a user’s own data using chunk filtering, context-guided query generation, and LLM-judge alignment with human labels. The goal is practical, production-like evaluation that reveals real model and retrieval performance differences.
Main Topics: Why public benchmarks are insufficient (Priority: 5/5): Public retrieval benchmarks can be generic, clean, and potentially memorized, so their scores may not reflect production performance or domain-specific use cases. Generative benchmarking workflow (Priority: 5/5): The method creates custom eval sets from a user’s corpus by first filtering chunks for relevance and then generating realistic queries conditioned on application context. LLM-as-judge with human alignment (Priority: 5/5): The team uses an LLM judge to filter/select data, but only after aligning its criteria to human labels; unaligned judges were shown to be unreliable. Representativeness over diversity (Priority: 4/5): Unlike some synthetic eval tools focused on query diversity, this approach prioritizes whether generated queries resemble real production queries and preserve model rankings. Retrieval metrics and model selection (Priority: 4/5): The evaluation emphasizes recall@K over ranking metrics like NDCG, because what matters most is whether relevant chunks are retrieved at all. Practical implications for RAG engineering (Priority: 4/5): The discussion highlights that retrieval quality, distractors, chunking, and contextual rewriting can strongly affect downstream LLM outputs, so engineers should debug retrieval first. Future directions: iterative evals and memory (Priority: 3/5): Kelly points to iterative benchmark refinement using live user queries and mentions upcoming research on agent memory and best practices.
Key Arguments: Public benchmark scores can be misleading because they are often too polished, too generic, and may contain data the model has already seen. A custom eval set built from a system’s own corpus is more useful for debugging real retrieval performance than vibe-checking or benchmark-only model selection. Filtering chunks before query generation is essential because irrelevant or low-value content can distort what a benchmark measures. Context matters: giving the LLM the application’s purpose and example queries helps it generate realistic, production-like queries. The generated benchmark should preserve the relative ranking of embedding models; if rankings match ground truth, the eval set is likely representative. LLM judges are not automatically aligned with human preferences; human-labeled calibration materially improves reliability. Recall@K is the most meaningful retrieval metric for this use case because having the right chunk somewhere in the retrieved set matters more than exact rank order. Retrieval quality strongly influences downstream answer quality, so engineers should inspect retrieved documents rather than only final LLM outputs. Domain-specific corpora are harder than public benchmarks and often expose larger performance gaps between embedding models and retrieval strategies.
Data Points: Alignment of initial LLM judge to human labels: 46% - Before prompt/criteria refinement, the LLM judge’s chunk filtering matched human labels at only about 46%. Alignment after evalgen iteration: Over 70% - Using evalgen to refine criteria and prompting increased judge-human alignment to above 70%. Manually labeled examples: 200 chunks/documents - The team manually labeled 200 chunks to calibrate and evaluate the LLM judge. Corpus size used in one study: ~13,000 chunks - Weights & Biases data used for the research contained around 13,000 pre-chunked items. Chunk size: ~400 tokens - The chunks in the study were approximately 400 tokens each. Public benchmark comparison: Voyage 3 large performed best - On real production data, Voyage 3 large ranked best among tested embedding models. Model comparison anomaly: Gnat AI slightly worse than OpenAI text-embedding-large - This contradicted the MTEB-based expectation, showing benchmark rankings did not match production behavior. Metric emphasized: Recall@K - The team focused on recall@K rather than NDCG or groundedness for retrieval evaluation. Model performance on public benchmarks: Recall@1 was often sufficient - On clean public benchmark sets, recall@1 gave a good sense of performance, unlike harder real-world data.
Pivotal Quotes: "there are so many problems with these public benchmarks" — Kelly Hong: Kelly explains why benchmark scores alone are not trustworthy for choosing embedding models. "we noticed it only got around like 46% alignment, which is pretty bad" — Kelly Hong: She describes the unreliability of an uncalibrated LLM judge before using evalgen to improve alignment. "you don't just want to get high numbers, like, you want something that's more realistic" — Kelly Hong: She emphasizes that representativeness matters more than maximizing synthetic benchmark scores.
Implications: For RAG and retrieval teams, benchmark scores should not be treated as truth. Build evals from your own data, align judges with humans, and optimize retrieval quality first; otherwise, model selection and system tuning may be misleading.