852: In Case You Missed It in December 2024

Episode Summary

Executive Summary: This year-end highlight episode revisits major 2024 themes in AI: Andrew Ng’s advice to prioritize building with the best model before optimizing cost, Ed Donner’s framework for selecting LLMs based on business needs and data constraints, Iman Ibrahimi’s view that agentic systems will complicate data security, and a broader debate about AI’s “wow” moments versus its risks. The episode closes with a philosophical discussion of tech optimism, singularity narratives, and the value of humility and human-centered progress.

Main Topics: Build first, optimize cost later: Andrew Ng argues that teams should not over-focus on model/API cost at the start. The priority is to get something working with the best available model; cost reduction, fine-tuning, or cheaper models come only after the product has proven valuable. How AI engineers choose models and workflows: Ed Donner explains that model choice should follow business requirements, data type/quality, evaluation metrics tied to outcomes, budget, and time-to-market. He recommends starting with a baseline model and often beginning with closed-source models before moving to open source if needed. Open source, privacy, and deployment constraints: The discussion covers when teams should choose open source: proprietary data, sensitive data that cannot leave the premises, lower inference costs, or on-device/offline deployment. It also notes that small language models can be more appropriate for edge use cases. Agentic AI and the future of data security: Iman Ibrahimi argues that agentic systems will span multiple data sources and environments, requiring a more complex security posture. He highlights homomorphic encryption and hybrid architectures as ways to reduce exposure while still enabling useful computation. 2024’s biggest AI wow moments and disappointments: Sadie St. Lawrence and John Crohn compare the year’s biggest AI surprises, with OpenAI’s o1 both disappointing and impressive in different ways, and NotebookLM standing out for its human-like podcast-style output. Waymo and Tesla FSD are discussed as especially tangible demonstrations of AI. Tech agnosticism and the singularity debate: Greg Epstein critiques tech worship and messianic narratives around AGI/singularity. The conversation frames future AI claims as belief-laden, warns against overconfidence, and argues for a slower, more humane approach to technological change.

Key Arguments: Model/API cost is usually not the first bottleneck; the harder problem is making something that works at all. Premature cost optimization can distract teams from product-market fit and real utility. A baseline model, even a simple heuristic or logistic regression, helps establish a measurable benchmark before moving to large LLMs. There is no universally best LLM; the right model depends on business goals, data characteristics, evaluation criteria, budget, and latency/time-to-market. Open-source models are especially valuable when data is sensitive, proprietary, or must stay on-prem, or when inference costs need to be reduced. Agentic AI will increase the complexity of data exposure because different components may run across on-prem, private cloud, and public cloud environments. Homomorphic encryption can help in parts of the pipeline by allowing some computation without exposing plaintext data. NotebookLM impressed because it produced a natural, human-like podcast experience, showing how AI can feel emotionally and socially intuitive. Waymo and Tesla FSD are important because they make AI’s impact visible in the physical world, not just on a screen. Singularity/AGI narratives resemble religious or apocalyptic belief systems and should be treated with skepticism and humility. Society should value uncertainty and gradual progress rather than assume that faster AI development is always better. AI progress may benefit some groups while harming others, so governance and retraining are necessary.

Data Points: Episode number: 852 - In Case You Missed It December episode of the Super Data Science Podcast Referenced episode: 841 - Andrew Ng clip on cost, model choice, and building before optimizing Referenced episode: 847 - Ed Donner clip on AI engineer responsibilities and model selection Referenced episode: 843 - Iman Ibrahimi clip on security, agentic AI, and homomorphic encryption Referenced episode: 849 - Sadie St. Lawrence clip on the year’s biggest AI flop and wow moment Referenced episode: 845 - Greg Epstein clip from Tech Agnostic discussion Referenced episode: 822 - John Crohn references NotebookLM and a podcast episode about his PhD dissertation Cost example: $5 OpenAI bill - Andrew Ng says he sometimes spends a weekend coding and ends with only a $5 bill Cost example: tens of dollars to low hundreds of dollars - Andrew Ng notes some agentic workloads can get into this range Cost reduction: 80% - Mentioned as the scale of price decreases for some model usage options Model example: GPT-4o mini vs GPT-4o - Discussed as a cheaper option versus a more capable model Model example: Llama 3.2 - Mentioned as an example of a small language model for on-device/offline use Time horizon: 5-10 years - Iman Ibrahimi references thinking ahead to future problems when doing research Workforce scale: 100,000 fleet of cars - Waymo is described as having a large fleet operating as a single driver brain Societal scale: Truck driver is the number one job in the United States - Used in a discussion of the broader labor impact of self-driving vehicles Investment figure: $7 trillion - Greg Epstein references Sam Altman’s call for massive AI investment Investment figure: 9 trillion - Mentioned in passing as another huge AI investment number in the broader debate Timeline prediction: 2025 - Greg Epstein cites Sam Altman’s AGI timing belief Timeline prediction: 2029 - Greg Epstein cites Ray Kurzweil’s AGI timing belief

Pivotal Quotes: "Don't worry about the price of the LLM to get started." — Andrew Ng: Advice on prioritizing building a working product before optimizing inference cost "There isn't like one best LLM. There's the right LLM to use for the task at hand." — Ed Donner: Explaining how AI engineers should choose models based on business requirements "We are building the single best driver." — Waymo co-CEO (as quoted by Sadie St. Lawrence): Describing Waymo’s approach to autonomous driving as one scalable intelligence

Implications: For builders, the message is to start with working systems and measurable baselines, then optimize. For industry, security, governance, and model choice will get more complex as agentic AI grows. For society, the episode urges skepticism toward hype and a more human-centered approach to AI progress.