The Utility of Interpretability — Emmanuel Amiesen

Episode Summary

Executive Summary: The conversation centers on Anthropic’s circuit tracing and mechanistic interpretability release: open-source tools, notebooks, and visualizations that let researchers inspect how small open models compute specific outputs. Emmanuel explains how sparse autoencoders, attribution graphs, and interventions reveal intermediate representations, planning, shared cross-lingual features, and limitations like reconstruction error and unmodeled attention. The discussion also covers the field’s history, open questions, safety relevance, and why more people should experiment with these tools.

Main Topics: Circuit tracing release and open-source tooling (Priority: 5/5): Emmanuel introduces the newly released code, notebooks, and UI for tracing model computations on open models like Gemma 2B and Llama 1B, emphasizing that users can inspect and intervene on model internals without huge compute. How mechanistic interpretability works (Priority: 5/5): The speakers explain superposition, sparse autoencoders, features as directions, and attribution graphs as the core machinery for extracting and connecting internal concepts inside transformer models. Concrete behaviors discovered in models (Priority: 5/5): Examples include multi-hop reasoning, planning in poetry, multilingual feature sharing, medical diagnosis reasoning, and chain-of-thought faithfulness failures, illustrating that models do more than next-token matching. What the tools can and cannot explain (Priority: 4/5): Emmanuel is careful about limits: the method does not fully explain attention, some graph nodes are explicit errors, and reconstruction gaps mean the system exposes only a subset of model computation. Historical development and field accessibility (Priority: 4/5): The discussion traces mech interp from Chris Olah’s blog and vision-model feature visualization to modern LLM work, arguing the field is young, open, and accessible to researchers without PhDs or large clusters. Safety, applications, and future research directions (Priority: 4/5): The group discusses why understanding model internals matters for safety, debugging, jailbreak analysis, model design, and future work on making models easier to interpret from the start. Behind-the-scenes production of the release (Priority: 3/5): They describe the amount of manual effort behind polished blog posts and diagrams, the role of Fellows and data-viz tooling, and why the extra communication work was worth it.

Key Arguments: Circuit tracing lets researchers explain a model’s computation for a given token, not just identify surface-level outputs. Smaller open models often use the same reasoning circuits as larger models on some tasks, suggesting shared mechanisms across scales. Sparse autoencoders help unpack superposed representations by learning a sparse dictionary of features in the residual stream or MLP activations. Attribution graphs are a practical way to trace how intermediate features influence one another and the final output. The method is validated by interventions: suppressing or swapping an internal feature should change the model’s prediction if the hypothesis is correct. Models can plan ahead in generation; in poetry, a rhyme decision can shape earlier word choices before the next line is produced. Chain-of-thought text is not always faithful; a model can work backward from a hinted answer while presenting a plausible reasoning path. Interpretability is valuable for safety because deployed models are increasingly embedded in critical workflows, and we need our understanding to scale with capability. Open-source release lowers barriers so external researchers can test hypotheses, extend methods, and study failures or novel behaviors. Attention remains a major gap: current circuits focus mostly on MLPs and do not fully decompose attention-based computation.

Data Points: Open model examples: Gemma 2B and Llama 1B - Emmanuel says the release lets anyone trace behavior in open-source models, with Gemma 2B highlighted as an easy starting point. Method availability: Free Colab-compatible notebooks - He notes the notebooks can run on Google Colab and do not require expensive GPUs on the free tier. Feature scale: 34 million features - A later reference mentions that in scaling SAEs, about 60% of the 34 million features did not activate. Inactive features: 60% - This is mentioned in a discussion of sparse autoencoders and the difficulty of labeling and interpreting features. Human effort on production: Months of work - The visual release is described as requiring substantial manual effort, with experiments, validation, and diagram polishing over a long period. Rapid prototype time: Minutes - The UI allows a user to enter a prompt and generate a circuit graph in minutes. Planning example sampling: 10 poems - Emmanuel mentions sampling ten poems for each plan to show that the internal plan affects the whole output distribution. Historical training scale: 100 GPUs - He contrasts current interpretability work with large-scale training, noting many interp projects can be done without clusters of around 100 GPUs. Model count: 2 papers - The release is framed as a pair of papers: one methods paper on circuit tracing and one biology/behavior paper. Behavior categories: 2 to 3 categories - When describing what users can do with the release, Emmanuel organizes contributions from low-effort exploration to method extension.

Pivotal Quotes: "We recently released a pair of papers about sort of like the work that we've been doing over the last months." — Emmanuel: Opening self-introduction about the new Anthropic mechanistic interpretability release. "Explain any token that the model samples... blow up the internal state of the model and show all of the sort of intermediate things that the model was thinking about before it got to the final token." — Emmanuel: High-level description of what circuit tracing does. "We don't explain everything... if you hit any errors, hit nodes that make sense, and build up a reasonable hypothesis, and then, when you test it with interventions, it works, you've at least understood some... of the computation." — Emmanuel: Clarifying the limits and validation standard for interpretability explanations.

Implications: The release makes mech interp more accessible and testable, enabling outside researchers to probe model internals, spot limitations, and improve safety-oriented understanding. It also shows the field is moving from toy examples to real open models, but key gaps like attention and full faithfulness remain.