Episode Summary
Executive Summary: Jürgen Schmidhuber traces the path from recurrent neural networks to LSTM as a solution to long-range dependency learning, arguing that AGI requires general-purpose recurrent architectures, self-improving learning, and reinforcement learning. He frames intelligence as fundamentally simple, highlights historical roots of backpropagation and deep learning, and describes how LSTMs became commercially useful once compute became cheap enough.
Main Topics: AGI as the guiding research goal (Priority: 5/5): Schmidhuber says his work has long focused on building a general-purpose learning machine that can learn new professions, improve its own learning algorithm, and eventually become a self-improving system that solves broadly defined problems. RNNs versus feed-forward networks (Priority: 5/5): He argues recurrent neural networks are the neural equivalent of general-purpose computers, while feed-forward networks are more like calculators. RNNs can represent sequential computation and arbitrary programs, making them foundational for AGI. The vanishing gradient problem (Priority: 5/5): Early recurrent networks failed on long time lags because backpropagated gradients either vanished or exploded exponentially. This prevented them from learning dependencies across many time steps and blocked practical sequence learning. LSTM architecture and why it worked (Priority: 5/5): LSTM solves vanishing gradients by using a simple linear memory cell plus gates that control reading, writing, and forgetting. This preserved error flow over long time spans and enabled practical sequence learning. Historical development of backpropagation and deep learning (Priority: 4/5): Schmidhuber places modern neural network training in a longer history, citing early deep feed-forward work in the 1960s and automatic differentiation/backpropagation ideas from the 1970s before their later neural-network adoption in the 1980s. Commercial impact of LSTMs (Priority: 4/5): Once compute became sufficiently cheap, LSTMs became widely useful in speech recognition, translation, language modeling, image segmentation, chatbots, and music composition, with major products such as Google Translate and Android speech systems. Simple foundations of intelligence and self-improving systems (Priority: 4/5): He argues intelligence may ultimately be simple in principle, pointing to asymptotically optimal problem solvers and the Gödel machine as evidence that general problem solving and self-modification can be formally described.
Key Arguments: AGI requires a general-purpose computational architecture, which in neural networks means recurrent networks with feedback connections. Traditional feed-forward networks cannot naturally represent sequential programs or long-horizon dependencies the way RNNs can. The core technical barrier for early RNNs was the vanishing/exploding gradient problem over long time lags. LSTM fixes this by creating a near-constant memory path through a linear cell and gates that regulate memory flow. LSTMs were not only a scientific breakthrough but became commercially practical only when hardware got fast and cheap enough. Reinforcement learning in partially observable environments is the missing module beyond pattern recognition for true AGI. Intelligence may be simpler than commonly assumed; the field may eventually reduce to a small set of core principles and algorithms. There already exists an asymptotically optimal general problem solver, but its constant overhead makes it impractical for small real-world tasks. Self-modifying systems like the Gödel machine are a step toward practical general problem solvers that can rewrite themselves when proofs justify improvement.
Data Points: Start of Schmidhuber's AGI-oriented publishing: 1987 - He says he has been working since his first publications on building a general-purpose learning machine. Early deep feed-forward networks: 1965 - He cites Ivakhnenko and Lapa’s deep multilayer work as a precursor to modern deep learning. Layer depth in early deep networks: 7–8 layers - Ivakhnенко-era systems had several layers, which was considered deep even by early-2000s standards. Backpropagation formulation: 1970 - He credits Seppo Linnainmaa’s automatic differentiation work as the modern backpropagation foundation. First RNN work: Late 1980s - Schmidhuber began working on recurrent networks during this period. First student and vanishing-gradient diagnosis: 1991 - Sepp Hochreiter’s diploma thesis identified the vanishing gradient problem in RNNs. LSTM improvement with forget gate: Around 2000 - Felix Gers introduced the forget gate, improving memory and forgetting behavior. Evolutionary LSTM topology work: 2009 - Justin Bayer’s work showed that evolved LSTM-like topologies could outperform fixed ones in some tasks. Commercial speech recognition usage: 2 billion Android phones - He cites LSTM-based speech recognition as widely deployed on Android devices. Google Translate switch to LSTMs: November 2016 - He says Google Translate moved away from the old system to an LSTM-based core around this time. Early computer speed: ~1 operation/second - He references Zuse’s early program-controlled computer as an example of primitive compute. Computing cost improvement: 10x cheaper every 5 years - He describes the long-term trend in compute cost that eventually made LSTMs practical. Relative compute speed by 2010 vs early computers: ~10 billion times faster - He argues this scale of improvement helped unlock practical LSTM deployment.
Pivotal Quotes: "I have to build something that learns to become smarter than myself, such that I can retire and such that this smarter thing is going to further self-improve and solve all the problems that I cannot solve." — Jürgen Schmidhuber: His origin story and motivation for pursuing AGI. "The difference between a recurrent network and a feed-forward network is a bit like the difference between a general-purpose computer and a mere calculator." — Jürgen Schmidhuber: His central analogy explaining why RNNs matter for AGI. "The whole AI thing is going to be really simple." — Jürgen Schmidhuber: His broader thesis that intelligence and general problem solving may have elegant, compact foundations.
Implications: For listeners and the industry, the episode frames RNNs/LSTMs as foundational steps toward AGI, not just sequence models. It suggests progress depends on combining memory, learning, and self-improvement with enough compute, and that future AI may come from simpler underlying principles than expected.