What Comes After ChatGPT? The Mother of ImageNet Predicts The Future

Episode Summary

Executive Summary: In this podcast episode, Fei-Fei Li and Justin Johnson discuss their new company World Labs and its product 'Marble,' a generative model that creates explorable 3D worlds from text or images. They explore spatial intelligence as the next frontier beyond language models, the evolution from their pioneering work in image captioning, the importance of scaling compute and data, and the role of academia in AI research. They also reflect on the differences between human and machine understanding, the need for physics and dynamics in world models, and the potential applications in gaming, VFX, simulation, and design.

Main Topics: Marble and World Labs Product (Priority: 5/5): Marble, a generative model that creates explorable 3D worlds from text or images, with interactive editing features such as color changes and object replacement. It uses Gaussian splats for real-time rendering on mobile and VR devices. Spatial Intelligence Definition and Importance (Priority: 5/5): The concept of spatial intelligence as the ability to reason, understand, move, and interact in space, contrasted with language intelligence. This is presented as the next big frontier in AI. Scaling Compute and Data for World Models (Priority: 4/5): Discussion on the history of deep learning driven by compute scaling, from AlexNet to modern models using thousands of GPUs. The need for more compute and data to train world models. Role of Academia vs. Industry in AI Research (Priority: 4/5): Reflections on the role of academia in AI—from open datasets like ImageNet to the current imbalance where industry has more resources. Advocating for better public resourcing and continued open science. Physics Understanding and Causal Models (Priority: 4/5): The limitation of current AI in understanding physics causally; the difference between pattern fitting and true understanding. The need for physics and dynamics in 3D world models.

Key Arguments: Spatial intelligence is complementary to language intelligence and represents the next frontier in AI. Scaling compute and data is essential for advancing world models; the entire history of deep learning is about scaling compute. Transformers are natively models of sets, not sequences, which is a key insight for building world models. Open science and academia still have a crucial role but need better resourcing to keep up with industry. Current AI models lack true understanding; they fit patterns but do not have causal models of physics or space. Gaussian splats are a useful atomic unit for real-time 3D rendering, but other representations (e.g., tokens, frames) may emerge. Language and spatial models will likely work together multimodally, not replace each other. Physics and dynamics must be integrated into world models for real-world applications, either through simulation or learned approaches.

Pivotal Quotes: "I think the whole history of deep learning is in some sense the history of scaling up compute." — Justin Johnson: Justin Johnson discussing the history of deep learning and the scaling of compute. "Spatial intelligence is the capability that allows you to reason, understand, move, and interact in space." — Fei-Fei Li: Fei-Fei Li describing the vision of spatial intelligence. "Transformers are actually not a model of sequences. A transformer is natively a model of sets, and that's very powerful." — Justin Johnson: Justin Johnson making a key architectural point about Transformers.

Implications: World Labs' Marble represents a significant step toward spatially intelligent AI with potential to transform creative industries, simulation, and robotics. The discussion highlights the need for new architectures beyond sequence modeling and the complementary nature of spatial and linguistic intelligence, challenging current LLM-centric approaches.