Episode Summary
Executive Summary: Jacob Buckman argues that context length is a crucial but under-measured scaling axis in AI, and that the real challenge is balanced model design across weights, state, and computation. He explains Manifest AI’s “power retention” architecture, which unifies attention and recurrence through chunked computation to enable long-context training/inference efficiently, and describes their CUDA tooling and a 3B coding model as practical proof points.
Main Topics: Context as a scaling axis (Priority: 5/5): Buckman frames long context as a fundamental bottleneck in AI, comparable to parameter and data scaling, and emphasizes that utility depends on whether the model can actually use the context effectively. Power retention and attention-recurrence unification (Priority: 5/5): He explains retention as a linear-attention family that can be computed either recurrently or with attention, and says chunking gives the best of both worlds: parallelism plus linear scaling. State size as the key lens (Priority: 5/5): The discussion reframes transformer inefficiency as a state-size problem: KV cache is treated as state, and various long-context methods are described as reductions along time, heads, feature, or layers axes. Balanced compute-optimal architectures (Priority: 5/5): Buckman argues that models should be balanced across weight flops and state flops; architectures that are too state-heavy or too weight-heavy are suboptimal compared with a Goldilocks ratio. Hardware and CUDA co-design (Priority: 4/5): Manifest AI built a custom CUDA framework, Vidrial, to maximize performance beyond Triton by systematically searching kernel layouts, enabling faster flash attention and power retention kernels. PowerCoder as proof of concept (Priority: 4/5): The team converted StarCoder 2 into PowerCoder using power retention, showing that existing pretrained weights can be adapted with limited retraining and then extended to longer contexts. Evaluation and limits of long-context metrics (Priority: 4/5): Buckman criticizes naive metrics like needle-in-a-haystack tests and argues for in-context learning curves and pretraining loss as better ways to assess true context utility.
Key Arguments: Context length is not useful by itself; it is useful only if the model can exploit the extra tokens to improve prediction and task performance. Long-context capability should be treated as another scaling axis, with its own scaling laws and compute-optimal balance, not as a binary feature. Transformers are constrained by growing KV-cache state; many long-context methods are just different ways of shrinking or managing that state. Retention mathematically bridges attention and recurrence, allowing linear-cost processing while retaining GPU-friendly chunked parallelism. The best architecture balances weight flops and state flops; doubling the smaller side can be nearly free in runtime while improving performance. Existing pretrained models can be adapted to power retention with relatively modest retraining rather than training from scratch. Custom kernel engineering matters: low-level implementation choices can yield meaningful speedups over standard Triton/flash-attention implementations.
Data Points: Deep learning experience: ~10 years - Buckman says he has worked on deep learning since around 2016. PowerCoder model size: 3 billion parameters - The published coding model is described as a 3B parameter adaptation of StarCoder 2. Human Eval performance: ~30% accuracy - Buckman cites StarCoder 2/PowerCoder as having around 30% accuracy on HumanEval. Metamorphosis retraining time: ~2 hours - Recovering performance from StarCoder to PowerCoder on 128 H100s took around two hours. Cluster size: 128 H100s - Used for the short retraining run that realigned the model to power retention. Speedup over Triton power-retention kernels: 2x to 4x - Vidrial CUDA kernels outperform the Triton implementation on power retention. Speedup over Tree Attention flash attention: ~20% - The framework reportedly beats Tree Attention’s flash attention on many problem shapes. Gain on standard flash-attention shapes: ~1% to 2% - On common shapes expected by flash-attention authors, gains are modest. Inference speedup at 64K context: ~100x - Buckman claims large inference speedups for long-context training/inference when switching to power retention. Training speedup at 64K context: ~10x - He claims substantial training-time savings for long-context workloads. Context lengths mentioned: 4K, 16K, 32K, 64K, 100K, 1M tokens - Used to illustrate scaling, retraining, and the gap between advertised max context and actual utility. State-size gap: ~2,000x larger - A transformer with 1,000 tokens of context is framed as having roughly 2,000x the state of an equivalent LSTM. State-size gap versus LSTM: ~5 orders of magnitude - Buckman argues transformers at very long contexts have vastly larger state than LSTMs/RNNs.
Pivotal Quotes: "What you really want is an architecture that's balanced, right?" — Jacob Buckman: He is explaining his compute-optimal design philosophy and why no single axis, including state, should dominate. "All contexts are not created equal here, right?" — Jacob Buckman: He is distinguishing raw context length from actual utility and model ability to use the context effectively. "The reason why the quadratic scaling of a transformer sort of is a thing is because the state of a transformer is growing laterally." — Jacob Buckman: He reframes transformer inefficiency as a state-size problem, motivating windowing, GQA, latent attention, and retention.
Implications: The conversation suggests long-context AI will be won by balanced architectures and hardware-aware implementations, not just bigger context windows. For builders, the takeaway is to optimize state/weight tradeoffs and evaluate real context use, not headline token counts.