From RLHF to RLHB: The Case for Learning from Human Behavior - with Jeffrey Wang and Joe Reeve of Amplitude

Episode Summary

Executive Summary: Live from Amplitude’s offices, the episode explores how product analytics and AI intersect: using LLMs to make data easier to query, synthesize insights, and eventually help products improve themselves. Jeffrey and Joe describe Amplitude’s AI strategy, the limits of current models, how they instrument and evaluate multi-step LLM workflows, and why first-party behavioral data plus strong metrics matter more than hype.

Main Topics: Amplitude as an AI company: Jeffrey frames Amplitude as naturally aligned with AI because product analytics is fundamentally about turning behavioral data into better product decisions. AI reduces manual analysis and moves the company toward automated insight generation. Origins of Amplitude and product analytics: Jeffrey recounts the company’s pivot from a voice/text startup to analytics after realizing the need for better tools to understand churn and engagement. Gaming companies were early adopters because they already optimized product loops rigorously. LLM product design patterns: Joe outlines three ways to add AI: as a collaborator, as a deeply embedded tool, or as a standalone tool. Amplitude’s first LLM feature is a text-to-chart interface that turns natural language questions into visual analysis. Evaluation, hallucinations, and multi-step inference: The team discusses how LLM workflows break into multiple subqueries, creating harder evaluation problems than a single yes/no metric. They instrument each inference and retry path to understand failures and improve reliability. Metrics, A/B testing, and self-improving products: They argue that AI features should be measured against business outcomes, not vanity metrics like time in app. They envision using A/B tests, retention curves, and multi-armed bandits to optimize AI-generated copy and product experiences. Model choice, embeddings, and infrastructure trade-offs: General models are used for rapid iteration, while smaller or embedding-based approaches are preferred when cost and latency matter. They note that Postgres/pgvector is sufficient for some taxonomy-level use cases, but not for massive event-scale embeddings. Data, privacy, and future behavioral models: The discussion covers first-party behavioral data as a unique asset for training future models, while emphasizing PII stripping and internal-only training. They speculate that a behavioral foundation model could improve prediction, segmentation, and causal analysis.

Key Arguments: Product analytics is an AI problem because the goal is to automatically turn user behavior data into product improvements with less manual work. Amplitude’s early pivot came from realizing analytics tools were more valuable than the original voice/text product, especially for understanding churn and engagement. LLMs are most useful when the problem is decomposed into smaller, well-defined subproblems rather than asking one model to do everything at once. A single user request can trigger multiple model calls, so evaluation must track each sub-inference and failure mode instead of relying on one aggregate score. Business metrics should drive AI evaluation; for many products, time in app is a vanity metric, while retention, conversion, and value creation are better North Stars. Natural language is a great entry point for exploration, but precise tasks like revenue calculations or SQL still require exact semantics and human control. General-purpose models are ideal for prototyping, but smaller or embedding-based systems become attractive once the task is validated and needs lower latency/cost. First-party behavioral data is a strategic asset because it can support models that understand user behavior patterns, segmentation, and likely future actions. Privacy boundaries matter: PII should be stripped, and training should remain internal rather than using raw customer data in external model training. The future of product experiences may be self-improving systems where AI generates options and experimentation layers choose the best-performing variant.

Data Points: Hackathon attendance: over 150 people - Amplitude hosted a weekend hackathon at its offices. Company history: about a decade - Jeffrey said he has worked on product analytics for the last decade or so. Original startup pivot window: first three to six months - The hosts asked about the early challenges after founding Amplitude. LLM workflow decomposition: 7 different questions - Amplitude split one chart-generation request into multiple subqueries such as chart type, events, display, and naming. AI design partner rollout: last week - Jeffrey said the question-to-chart feature started rolling out to AI design partners the week before the recording. Behavioral prediction window: 90 days - Joe described a predictions feature that uses the last 90 days of user behavior to forecast cohort membership. Human energy efficiency: 30 watts - Joe cited the power usage of the human brain as a comparison point for future intelligence systems. Model used at the time: GPT-3.5 - Jeffrey said the team was using GPT-3.5 for everything at that stage. Scale of taxonomy data: on the order of tens of millions - Jeffrey explained that taxonomy-level embeddings are small enough to store in Postgres.

Pivotal Quotes: "Amplitude has always been, you know, a company on the path to AI because, you know, figuring out how to make your product better from data is ultimately an AI problem." — Jeffrey: Explaining why product analytics and AI are deeply connected. "Let humans be good at doing human things. Let machines be good at doing machine things." — Joe: A closing principle for designing useful and safe AI products. "Natural language is a great interface to like get started... once you kind of know what you want, it's very hard to like make it precise." — Jeffrey: Discussing the limits of natural language versus SQL/code for exact analytics.

Implications: The episode suggests AI products will win by combining strong data infrastructure, precise evaluation, and human control. For analytics teams, the near future is natural-language exploration plus rigorous metrics; longer term, behavioral foundation models could automate insight generation and product optimization.