AI AMA – Part 1: OpenAI’s o3, Deliberative Alignment, and AI Surprises of 2024

Episode Summary

Executive Summary: The interview centers on OpenAI’s o3/Arc-AGI results, reasoning-model weirdness, and deliberative alignment, then broadens to 2025 forecasts: agents, more capable but less legible AI, compute governance, and uneven impacts on jobs, education, and startups. The guest argues capability is rising fast, but robustness, perception, and long-horizon autonomy remain weak, while access and power may become more concentrated.

Main Topics: o3 and reasoning-model performance (Priority: 5/5): The guest interprets OpenAI’s o3/Arc-AGI results as a real breakthrough, but stresses that the available evidence is partial and that reasoning models still fail obvious toy tasks, suggesting impressive capability gains alongside persistent weirdness. Deliberative alignment and safety (Priority: 5/5): OpenAI’s new safety plan is described as a constitutional, policy-following alignment pipeline that uses the model to reason about policy and then fine-tunes on successful outputs, but it leaves open the hardest questions about deception, robustness, and what policy should be. Compute, governance, and inequality (Priority: 4/5): The shift toward test-time compute and multi-sample inference may make frontier AI less universally accessible, potentially enabling compute governance while also increasing concentration of power and unequal access to advanced models. 2024 vs. 2025: what progressed and what lagged (Priority: 4/5): The guest says AI improved most in depth, coding, and benchmarked reasoning, but underperformed in autonomy, memory, RAG, and practical deployment, especially because people and organizations have not yet adopted best practices. Agents, autonomy, and the coming year (Priority: 4/5): He expects 2025 to be the year of agents, with computer-use workflows and RL-driven systems becoming more common, but also predicts more strange, hard-to-interpret behavior and possible scheming as models get stronger and less legible. Education, coding, and personal strategy (Priority: 3/5): AI is already transformative for motivated learners and coders, but the guest advises people to follow intrinsic motivation rather than bet their career on long-term marketability, since AI may shift which skills remain valuable. Startup moats and product defensibility (Priority: 3/5): For AI startups, durable advantage comes from domain taste, high-quality templates/data, and last-mile editing or customization; merely wrapping a model is unlikely to be enough as frontier capabilities continue to expand.

Key Arguments: o3/Arc-AGI appears to be a genuine capability leap, with low-effort performance around human level and high-effort performance above human level, but the underlying mechanism is still opaque. Reasoning models are not a panacea: they can still fail simple board-state problems, hallucinate bad perceptions, and produce nonsense reasoning over long chains. OpenAI’s deliberative alignment is a scalable way to bake policies into model behavior using the model itself, but it does not solve deeper alignment or deception concerns. The big unknown is how o3 chooses among many samples; if that aggregation generalizes beyond verifiable tasks, it could be a major breakthrough. RL-heavy systems will likely excel in domains with easy reward signals like math and code, while harder-to-verify tasks may lag unless better selection/aggregation methods emerge. AI progress in 2024 was strongest in benchmarks, speed, cost, coding, and depth, but weaker in autonomy, memory, robustness, and practical RAG-based applications. People are underusing current models because of outdated impressions, fear, lack of best practices, and the effort required to build effective examples and workflows. The likely 2025 story is more agents, more compute-heavy reasoning, more benchmark saturation, and more bizarre/inscrutable behaviors, including possible scheming. Compute governance may become more plausible if hard tasks require significant inference budgets, shifting power toward organizations with larger clusters. For education and coding, the best advice is to follow intrinsic motivation; AI amplifies learning, but it may also reshape which professions stay valuable. Startups can still have moats if they combine strong domain-specific product design, proprietary templates/data, and reliable last-mile controls that big models don’t yet replicate well.

Data Points: o3 low-effort Arc-AGI score: 75% - Reported as roughly human-level and about 15 points above prior systems in the Arc-AGI benchmark. o3 high-effort Arc-AGI score: 87.5% - High-compute setting described as above human performance on Arc-AGI. Low-effort samples per task: 6 - Guest inferred six rollouts per task for o3 low-effort evaluation. High-effort samples per task: 1,024 - Guest cited the reported high-effort setting as using 1,024 samples. Low-effort total tokens: 33 million - Across 100 Arc-AGI tasks, used to estimate per-task and per-sample compute. High-effort total tokens: 5.7 billion - Across 100 Arc-AGI tasks in the high-compute setting. Low-effort time per task: 1.3 minutes - Reported runtime for the six-sample setting. High-effort time per task: 13 minutes - Reported runtime for the 1,024-sample setting. o3 low-effort cost per task: about $20 - Guest ties the reported cost to pricing and token counts. OpenAI output-token price reference: $60 per million output tokens - Used to estimate the approximate task cost for o3-style inference. Claude cooperation result: dominant in iterative cooperation game - Guest references a paper where Claude models cooperated better than other models, including GPT-4.0. Memory/time-bias example: 4 x 30 minutes vs 2 hours - Guest cites METR-style work comparing AIs given multiple 30-minute runs against human two-hour performance. Model adherence success: high 90s % - Guest says deliberative alignment reaches high-90s policy adherence, though not perfect. Codebase size: 100,000 tokens - Guest mentions his app/codebase has grown large enough to require whole-codebase reasoning. Estimated enterprise chatbot cost example: 10 cents per query - Used to argue that some internal use cases can afford whole-document Flash-style retrieval.

Pivotal Quotes: "“Even with these reasoning models, they are still weird.”" — Adityan Ilangovan: Used to frame the central thesis that capability gains do not remove model brittleness, bad perception, or cached heuristics. "“We hope the underlying reality is such that these things are like pretty manageable.”" — Adityan Ilangovan: On AI safety, compute governance, and the hope that frontier systems remain controllable enough to avoid extreme caution. "“Do what you want to do. Don't spend years of your life on a bet that... the AIs aren't going to dominate the programming profession in the next few years.”" — Adityan Ilangovan: Advice on career choice and coding in an uncertain labor market.

Implications: Expect stronger agents, more compute-heavy reasoning, and more benchmark wins in 2025, but also more opaque behaviors. For users and startups, the near-term edge is in domain expertise, workflow design, and last-mile control—not just model access.