761: Gemini Ultra: How to Release an A.I. Product for Billions of Users, with Google's Lisa Cohen

Episode Summary

Executive Summary: Lisa Cohen, Google’s Director of Data Science and Engineering for Gemini, Assistant, and Search, explains how Gemini’s rollout works across products, the differences between Nano/Pro/Ultra, and how Google balances quality, safety, accessibility, and privacy at massive scale. The conversation also covers how LLMs are reshaping data science, product development, education, and everyday productivity.

Main Topics: Gemini product family and rollout: Lisa explains the Gemini brand shift from Bard, the product lineup (Nano, Pro, Ultra), and how users access Gemini through the web, app, Google One AI Premium, and Google Cloud. Data science leadership across Assistant, Search, and Gemini: Her team measures quality, latency, reliability, safety, user journeys, and experimentation across products serving hundreds of millions of users, while reusing methods and libraries across teams. Safety, factuality, and responsible launch practices: Google’s launch strategy includes metric-based safety evaluation, ringed rollouts, trusted testers, red teaming, age-based safeguards, and the double-check citation feature to reduce hallucinations. How LLMs change data science and software work: Lisa and John discuss prompting, supervised fine-tuning, and RLHF, and how LLMs raise the level of abstraction so data scientists and developers can focus on design, product thinking, and higher-value work. Globalization, inclusivity, and accessibility: Gemini is being tuned for many languages and regions, with checks for language-specific performance, UI differences, and diverse image generation to avoid bias and improve cross-cultural usability. Education, creativity, and productivity transformation: The episode highlights AI tutoring, creative writing, lesson planning, image generation, and in-flow productivity in tools like Google Workspace as major near-term benefits of generative AI. Org design and data culture at scale: Lisa reflects on centralized vs decentralized data science teams, building data-driven cultures, and using the 80/20 rule and shared context to align technical work with business outcomes.

Key Arguments: Gemini is being renamed and integrated across Google products to create a clearer, more cohesive user experience around the same model family. LLMs make English a programming language for data access, broadening who can query, summarize, and reason over data. Data scientists and developers will increasingly work one level higher, using LLMs for drafting, code generation, reasoning, and collaboration rather than typing every line manually. Prompting, supervised fine-tuning, and RLHF are complementary techniques: prompting shapes outputs, supervised fine-tuning adapts a model to a task, and RLHF aligns it with human preferences. Safety must be built into launch decisions through layered testing, red teaming, and staged releases rather than treated as an afterthought. Localized evaluation matters because multilingual and multicultural users can experience the same product differently, so fairness and quality must be checked by language and region. AI will democratize personalized tutoring and creative assistance, making high-quality support more accessible in education and personal productivity. Centralized data science teams gain scale, shared tooling, and peer learning, but must stay embedded enough to understand product context and business goals.

Data Points: Google Assistant monthly active users: hundreds of millions - Lisa describes the scale of Assistant that her team supports with data science and engineering. Gemini available languages/regions: 40+ languages and 230+ countries and territories - Lisa notes Gemini’s broad international rollout and localization efforts. Gemini Pro availability: made available in all supported languages last month - She explains the latest multilingual expansion of Gemini Pro. Google One AI Premium price: $19.99/month - Pricing for the Gemini Advanced subscription that provides access to Gemini Ultra 1.0. Gemini Advanced trial: 2-month free trial - The premium subscription includes an introductory free trial period. AI-powered workspace adoption: more than 1 million people - Lisa mentions adoption of features like Help Me Write through Duet AI, soon Gemini for Workspace. Image generation diversity requirement: diverse racial and gender representation - Lisa emphasizes training and evaluation for inclusive generated imagery. Cross-product language shift: Bard renamed to Gemini; Duet AI becoming Gemini for Workspace - Google is standardizing product names under the Gemini brand.

Pivotal Quotes: "The new programming language is English." — Lisa Cohen: Lisa explains how LLMs lower barriers to data access and make analysis more accessible to nontechnical users. "We’ve taken a stance at Google of being bold yet responsible." — Lisa Cohen: Her explanation of Google’s launch philosophy for Gemini, balancing rapid innovation with safety. "It’s the same with Gen AI: it’s a new paradigm, it’s a new way of working." — Lisa Cohen: Lisa compares generative AI to prior platform shifts like cloud computing.

Implications: LLMs are becoming a default interface for work, learning, and search. For data and product teams, the advantage will go to those who can combine model capability with safety, localization, and strong product context.