Titans: Neural Long-Term Memory for LLMs, with author Ali Behrouz

Episode Summary

Executive Summary: The episode explores Titans, a new long-term memory architecture for LLMs that treats memory as a trainable neural network updated at test time. Ali Beyrouz argues this bridges the “missing middle” between fixed model weights and short context windows, enabling more coherent, long-running, and enterprise-aware AI systems. The discussion covers human-memory inspiration, associative memory training, momentum/surprise updates, hybrid integration with attention, efficiency tradeoffs, and future applications beyond language.

Main Topics: The “missing middle” in LLM memory (Priority: 5/5): The host frames current LLMs as having world knowledge in weights and working memory in context windows, but lacking durable, evolving long-term memory needed for continuity, identity, and enterprise context. Human-memory inspiration and hybrid architecture (Priority: 5/5): Beyrouz explains that Titans is motivated by human memory: short-term/long-term separation, selective forgetting, and the idea that memory should be dynamic rather than a fixed vector or matrix. Titans as neural memory updated at test time (Priority: 5/5): The core novelty is using an MLP-like neural network as the memory module, updated by gradient descent during inference, rather than explicit retrieval stores or fixed recurrent state. Associative memory, queries/keys/values, and runtime learning (Priority: 5/5): The memory is trained to approximate attention by predicting values from keys, so that future queries can retrieve relevant payloads from learned associations. Surprise, momentum, and memory management (Priority: 4/5): The paper’s update rule uses token-level surprise plus momentum to decide what to store, decay, or emphasize, mirroring how salient episodes can shape memory over time. Hybrid integration strategies: memory as context, gate, or layer (Priority: 4/5): The conversation compares three ways to combine long-term memory with attention. Beyrouz argues context and gate are more principled than layer interleaving, and often perform better. Scaling, efficiency, and future applications (Priority: 4/5): The discussion covers how Titans can be reformulated efficiently, how well it extrapolates to long contexts, and potential uses in enterprise systems, agents, decision-making, and reinforcement learning.

Key Arguments: LLMs need a persistent, evolvable memory layer to support long-running tasks, enterprise adaptation, and agent coherence beyond the context window. Titans is conceptually distinct because memory itself is a neural network, not just a vector or matrix state. Attention can be understood as a non-parametric associative memory; Titans approximates that behavior with a trainable parametric memory module. Surprise-based updates are a principled way to decide what deserves memorization, and momentum helps preserve salient episodes over multiple tokens. Memory should be input-dependent: decay rates, update strengths, and momentum terms are conditioned on the current token rather than fixed globally. Hybrid architectures are likely the right direction because attention and recurrent-style memory solve different problems and can complement one another. Architecture and memory design matter as much as data scale; better memory structures may be key to long-context generalization. Titans’ best results appear in long-context settings, with the MAC-style hybrid being especially strong. Future progress will likely require combining long-term memory with other techniques and perhaps continued pretraining, while addressing catastrophic forgetting.

Data Points: Context length improvement: Up to 2 million tokens - The host says Titans can scale to very long contexts and references results showing the model handling contexts in the millions of tokens. Long-context accuracy: ~70% - The host cites an example that Titans can go to 10 million tokens with about 70% accuracy. Custom model pricing anchor: Millions to tens of millions of dollars - The host compares enterprise-scale training costs to OpenAI’s custom model offerings. Small business cost estimate: Tens of thousands to low hundreds of thousands of dollars - The host estimates custom memory-enabled models for smaller firms based on fine-tuning costs. OCI compute savings: Up to 50% less - Sponsor segment mentioning Oracle Cloud Infrastructure compute cost claims. OCI storage savings: Up to 70% less - Sponsor segment mentioning Oracle Cloud Infrastructure storage cost claims. OCI networking savings: Up to 80% less - Sponsor segment mentioning Oracle Cloud Infrastructure networking cost claims. Shopify e-commerce share: 10% of all e-commerce in the United States - Sponsor segment describing Shopify’s market presence. NetSuite customer count: Over 42,000 businesses - Sponsor segment describing NetSuite adoption. Memory architecture scale comparison: Half the heads / half the layers - Beyrouz suggests a practical heuristic: allocate roughly half of attention heads or layers to memory and the rest to attention.

Pivotal Quotes: "Context is the that which is scarce." — Host: Used to frame the central bottleneck in AI systems: not world knowledge, but context assembly and persistence. "I really believe that before that, I mean, at this time at least, we need to get inspired from how our brain works." — Ali Beyrouz: Explains the neuroscience-inspired motivation behind Titans and the broader hybrid-memory design philosophy. "One thing that is important here is there are some like equivalent versions that we can use to understand how we can set these parameters for our design." — Ali Beyrouz: Describes the paper’s emphasis on connected formulations and multiple architectural perspectives rather than a single final answer.

Implications: Titans suggests a plausible path to AI systems with durable, adaptive memory for enterprises and agents. If scaled well, it could improve long-context reasoning, personalization, and continual operation, but catastrophic forgetting and efficient implementation remain major hurdles.