Why Can't AI Make Its Own Discoveries? — With Yann LeCun

Episode Summary

Executive Summary: Jan LeCun argues that today’s LLMs are powerful retrieval tools, not true reasoners or discoverers: they lack world models, persistent memory, planning, and the ability to ask novel questions. He says progress from scaling text-only models is slowing, and the next breakthrough will come from non-generative architectures like JEPA that learn abstract representations from video and sensor data to support common sense, prediction, and planning.

Main Topics: Why LLMs don’t produce scientific discovery (Priority: 5/5): LeCun says current generative AI excels at regurgitating and recombining text but cannot invent new solutions or ask original research questions, which are central to discovery. The limits of chain-of-thought and reasoning models (Priority: 5/5): He argues that adding more tokens, post-hoc reasoning traces, or reranking outputs does not create true reasoning, which in humans involves search over internal mental representations rather than text. Diminishing returns from scaling LLMs (Priority: 5/5): LeCun believes the text-data frontier is largely exhausted and that more compute, synthetic data, and fine-tuning will bring only slow, expensive gains rather than human-level AI. A new paradigm: world models and JEPA (Priority: 5/5): He proposes Joint Embedding Predictive Architectures as a non-generative alternative that learns abstract latent representations from images/video, enabling prediction, planning, and common sense. Physical-world understanding from video and sensors (Priority: 4/5): LeCun says understanding reality requires learning from video and other sensory data, not just text; future systems should predict abstract outcomes like ‘the object will fall’ rather than pixel-level detail. Commercial deployment, timelines, and AI winter risk (Priority: 4/5): The interview addresses whether massive LLM investments will pay off before a new paradigm arrives, and whether delays or overpromises could trigger backlash or an AI winter. Open source vs proprietary AI (Priority: 4/5): LeCun says open source is accelerating innovation faster than proprietary labs, citing DeepSeek and Llama, and argues global collaboration beats any single company’s monopoly on ideas.

Key Arguments: LLMs are trained to reproduce text statistics and retrieve existing knowledge, not to invent new hypotheses or solutions. Discovery requires asking the right questions, framing problems correctly, and searching solution spaces—skills current LLMs lack. Chain-of-thought can improve apparent reasoning but mostly just extends token generation; it is not the same as internal reasoning over mental models. Text-only training is insufficient because humans and animals learn common sense from rich sensory input, especially vision and interaction. Scaling up LLMs further will yield diminishing returns because high-quality natural text data is largely exhausted and synthetic data is costly and limited. The future of AI lies in architectures that learn abstract world representations and can plan sequences of actions toward goals. JEPA-style systems avoid reconstructing pixels and instead predict latent representations, which better support understanding and planning. Open source spreads ideas faster, attracts diverse talent, and lowers deployment costs, making it strategically stronger in many cases than closed systems.

Data Points: Public web text tokens used for training: ~10^13 to 10^14 tokens - LeCun says large LLMs have already consumed roughly the whole public internet and more, indicating data saturation. Text equivalent read time: ~400,000 years - He estimates it would take one person about this long to read 10^14 tokens at 12 hours per day. Child visual data in 4 years: ~10^14 bytes - He compares a four-year-old’s visual experience to the scale of LLM training data. Optic nerve bandwidth: ~1 MB/sec per optic nerve - Used to estimate how much visual information reaches a child’s brain. Baby’s world-knowledge learning window: ~9 months - He notes babies learn intuitive physics and object permanence very early with relatively little data. Meta AI users: 600 million - LeCun says Meta AI already has hundreds of millions of users, though not all are highly active. OpenAI funding mentioned: $6.6 billion - Cited by the host as a recent raise tied to ChatGPT’s commercial success. Anthropic funding mentioned: $3.5 billion and $4 billion last year - Used to illustrate the scale of capital flowing into LLM-first companies. Enterprise proof-of-concept conversion: ~10% to 20% - Host cites that only a minority of enterprise AI pilots make it into production due to reliability/cost issues. Timeline for new paradigm: 3 to 5 years - LeCun estimates that practical architectures for world modeling and planning may emerge in this window.

Pivotal Quotes: "There is absolutely no way in hell to the idea that we're going to have a country of genius in a data center, that's complete BS." — Jan LeCun: He rejects claims that scaling LLMs alone will soon produce human-level or PhD-level AI. "The question is not just solving the problem. It's also asking the right questions." — Jan LeCun: He explains why scientific discovery requires more than retrieval or step-by-step answer generation. "What we're missing, okay, with AI systems, is understanding the physical world, having persistent memory, and being able to reason and plan." — Jan LeCun: He summarizes the core capabilities needed for more general intelligence.

Implications: The AI industry may need to shift from text-only scale bets to world-model research. Expect strong near-term utility from retrieval and assistants, but true discovery, planning, and reliable autonomy likely require new architectures, more data from sensors, and broader open collaboration.