E57: Google's PaLM 2 with Paige Bailey

Episode Summary

Executive Summary: Paige Bailey describes Google’s PaLM 2 effort as a massive, Apollo-like coordination of compute, data, evals, and productization across Alphabet. The conversation covers how frontier models are built, aligned, distilled, and deployed; why model reasoning remains partly statistical and fragile; and why multimodal, code-centric, and efficiency-focused advances will shape the next wave of AI.

Main Topics: Building PaLM 2 as a frontier model at Google: Bailey explains the scale of effort behind PaLM 2: large TPU allocations, cross-team coordination, data curation, continuous checkpoint monitoring, and aligning the model for broad deployment across Alphabet products. What a product manager does for a language model: She frames the role as coordinating researchers, engineers, evals, compliance, and deployment constraints—not traditional PM work—while maintaining technical fluency and hands-on coding ability. Alignment, instruction tuning, and evaluation: The discussion emphasizes defining personas, tuning behavior for downstream use cases, and using rigorous evals for safety and capability (code, math, security, reasoning) without contaminating training data. Reasoning, grokking, and model limits: Bailey argues that apparent reasoning often breaks under slight perturbations, suggesting many successes are pattern-matching rather than robust general reasoning, even as some unexpected capabilities emerge. Efficiency, distillation, and democratizing access: A major theme is making powerful models smaller and cheaper through distillation and efficient serving so frontier AI is not limited to those who can pay for premium accelerators. Code, multimodality, and the next wave of breakthroughs: She highlights code models, execution feedback, multimodal systems like Gemini, and tools for scientists and developers as the most promising areas for future gains. Google’s shipping pace and product surface area: Bailey pushes back on the idea that Google cannot ship, noting that PaLM 2 is already embedded across many products and that the company has long shipped incremental AI improvements, even if they are not always visible.

Key Arguments: Frontier model development is an interdisciplinary systems problem: compute, data, compliance, evals, and deployment all have to be managed together. A product manager for an LLM must understand downstream use cases deeply because pre-training data, instruction tuning, and eval design all affect behavior. PaLM 2’s impact comes not only from model quality but from efficient serving and distillation, which make powerful models usable in real products. Many “reasoning” wins are brittle; models often perform well only on familiar formats and may fail when prompts are slightly changed. Open-ended model capabilities are often discovered after release through user testing, social media, and exploratory probing rather than fully anticipated in advance. Code and math are especially important capability areas because they generalize across many other tasks and can unlock autonomous agent workflows. Multimodal and tool-using systems will likely drive the next leap, especially when combined with execution feedback, retrieval, and static analysis. Google’s large model work is increasingly about turning research into many productized experiences rather than just publishing benchmark-leading papers.

Data Points: PaLM 2 deployments: more than 38 products and features - Bailey says PaLM 2 is now deployed across Alphabet in more than 38 places. Codi researchers and engineers: about 150+ - She cites the size of the code AI effort she is driving at Google DeepMind. Timeline reference for PaLM 1: April 2022 - Mentioned as the publication timeframe for PaLM 1. PaLM 2 announcement timeframe: May (year implied 2023) - Used in the comparison of the compressed development timeline between PaLM 1 and PaLM 2. Model footprint reduction: from hundreds of billions of parameters to a couple dozen - Bailey describes distillation as compressing frontier performance into much smaller models. Efficient serving target: CPU or a single accelerator - She says powerful models should be small enough to serve efficiently on modest hardware. PaLM 2 product scope: text and code based - Bailey contrasts PaLM 2 with Gemini, which is multimodal from day zero. Multimodal explanation example: Malayalam code explanation - An engineer asked the model to generate code and explain each line in Malayalam, which worked unexpectedly well.

Pivotal Quotes: "we don't want to wind up in a world where the only people who have access to these incredibly powerful tools are people who can afford to pay like a premium for accelerators" — Paige Bailey: On why efficiency and distillation matter for access and democratization "it feels an awful lot like working on an Apollo program to be training one of these models" — Paige Bailey: Describing the scale, coordination, and complexity of training PaLM 2 "it hasn't learned anything to that effect. It's just learned kind of pattern matching based on things that it's seen" — Paige Bailey: Her explanation of why many apparent reasoning abilities are brittle

Implications: The AI race is shifting from raw model size to efficient deployment, multimodal capability, and tool-augmented reasoning. For users and builders, the biggest opportunities are in code, science, and productized workflows—not just chatbots or benchmarks.