862: In Case You Missed It in January 2025

Episode Summary

Executive Summary: This In Case You Missed It episode highlights three practical themes from January: thinking exponentially about technology shifts, where Azeem Azhar urges first-principles planning and feedback-loop-aware forecasts; real-world hybrid quantum computing use cases in scheduling and logistics; and how to choose and evaluate foundation models and AI agents using criteria like cost, latency, compliance, and custom metrics rather than generic accuracy alone.

Main Topics: Exponential thinking and first principles: Azeem Azhar explains why humans and businesses struggle to intuit exponential growth, why linear forecasts are misleading, and why planners should use first-principles thinking and feedback loops when modeling technology adoption and infrastructure needs. Quantum computing for practical optimization: Dr. Florian Neukart describes hybrid quantum computing as useful for hard optimization problems such as scheduling, logistics, vehicle routing, satellite constellations, and financial optimization, where classical methods rely on heuristics and approximations. How to choose a foundation model: Kirill Aramenko and Atlena Pontev outline 12 selection criteria for foundation models, including cost, modality, customization, inference, latency, benchmarks, language support, scale, compliance, licensing, and environmental impact. Fine-tuning and dataset fit: Atlena Pontev emphasizes that the right model often depends on dataset format and the specific fine-tuning or RAG use case, with practical choices driven by how well the model aligns with the structure of the data. Evaluating AI agents with layered metrics: Brooke Hopkins explains that agent performance should be assessed with multiple custom, often reference-free metrics, because conversational success is a spectrum and standard accuracy alone misses real user outcomes. Production monitoring and human review: The episode stresses that automated metrics should be used to identify the most interesting or likely-failing cases, while human review remains important for understanding edge cases and improving agent behavior.

Key Arguments: Exponential processes are hard for people to reason about, so businesses should avoid linear assumptions and instead model growth with compounding percentages and feedback loops. Many common technologies already demonstrate exponential change, such as smartphones and storage capacity growth, so listeners have direct evidence of how quickly systems can scale. Quantum computing is valuable when scheduling, logistics, and optimization problems become too complex for classical exact methods and require better-than-heuristic solutions. Foundation model selection is not one-size-fits-all; the right choice depends on practical constraints like data format, latency, compliance, and environmental cost. For AI agents, a single accuracy score is insufficient because many different conversation paths can be acceptable or unacceptable; layered metrics provide a fuller picture. Reference-free evaluation is especially useful for non-deterministic LLM systems because it can infer correctness from context instead of requiring fixed gold answers. Human review should focus on the most informative failures, not every example, making automated metrics a triage tool rather than a replacement for judgment.

Data Points: Foundation model selection criteria: 12 - Kirill Aramenko lists 12 factors businesses should consider when choosing a foundation model. Episode number: 862 - John Crohn introduces this as the January In Case You Missed It episode. Podcast episode referenced for quantum computing: 851 - John Crohn refers to his conversation with Dr. Florian Neukart. Podcast episode referenced for model selection: 853 - John Crohn refers to the discussion with Kirill Aramenko and Atlena Pontev. Podcast episode referenced for AI agent metrics: 857 - John Crohn references the conversation with Brooke Hopkins. Example scale in storage growth: 1 GB to 100 GB to 10 TB to a petabyte and beyond - John Crohn uses this as an example of exponential-scale data growth witnessed by practitioners. Example of planning horizon: 10 bucks a month - Azeem Azhar uses this as an example of early, compounding pension saving. Scheduling example test classes: 250-something test classes - Florian Neukart describes vehicle production testing as a complex scheduling problem. AI expansion timeline: Two and a half years ago there was no such thing; two years ago there was no such thing as starting foundation models - Kirill Aramenko highlights how quickly the foundation model landscape emerged.

Pivotal Quotes: "our intuition about the future is linear with the reality of IT" — Azeem Azhar: Used to explain why people misjudge exponential technological change. "Nothing grows that way. Everything follows a phase of a logistic S-curve where you have an exponential phase that tails off." — Azeem Azhar: Explaining why linear business forecasts are usually unrealistic. "it’s really hard to find the line between this is objectively a good conversation and this is objectively a failing conversation, but rather it’s a spectrum" — Brooke Hopkins: Describing why AI agent evaluation requires layered metrics rather than a single correctness score.

Implications: Listeners should model AI and tech adoption as exponential, choose models by operational fit, and evaluate agents with layered metrics. For industry, the message is clear: better forecasting, optimization, and monitoring are now competitive necessities.