Mechanistic Interpretability

Mechanistic interpretability is a subfield of explainable artificial intelligence that aims to understand the internal workings of neural networks by reverse-engineering their concrete computations into human-understandable parts. Rather than treating a model as a black box and only studying its inputs and outputs, mechanistic interpretability seeks to identify the specific structures, representations and algorithms a network uses. Its central objects of study are features, which are interpretable properties encoded in a model's internal activations, and circuits, which are the connected computations that combine features to produce behaviour.

Motivation

Modern large language models are trained, not programmed, so their internal decision procedures are not written down anywhere and must be discovered. This opacity is a concern for AI safety. If researchers cannot see why a model produces a given output, they cannot easily verify that it is reasoning honestly, detect deceptive or misaligned behaviour, or predict how it will act in novel situations. Mechanistic interpretability responds by attempting to build, in effect, a microscope for neural networks, so that their computations can be inspected in the way a biologist inspects cells.

Features

A feature is a direction in a model's internal activation space that corresponds to a human-interpretable concept, such as a reference to a particular city, a programming syntax pattern, or an abstract idea like deception. A key obstacle is superposition, the phenomenon in which a network packs far more features into its neurons than it has neurons, so that individual neurons respond to many unrelated concepts at once. To untangle this, researchers use sparse autoencoders, which decompose the dense activations into a much larger set of sparse, individually meaningful features. Using this method, interpretability teams have extracted tens of millions of features from production language models, including features that activate on specific concepts across languages and modalities.

Circuits

Once features are identified, the next step is to trace how they interact through the network's layers to carry out a computation. A well-known early example is the induction circuit found in small attention-only transformers. Two attention heads in consecutive layers cooperate: the first head looks backward to find where the current token previously appeared, and the second head copies the token that followed it, allowing the model to continue a repeated pattern. This kind of finding demonstrates that meaningful algorithms can be isolated inside a network and described precisely.

Progress and tools

Anthropic has been a leading contributor to the field, publishing extensively through its Transformer Circuits work and releasing circuit-tracing tools as open source in 2025. The research community has applied these tools to open-weight models such as Gemma and small Llama variants, extending mechanistic analysis beyond a single lab. Related efforts include representation engineering, which studies and steers high-level concept directions, and activation steering, which nudges model behaviour by adding feature directions during inference.

The field remains young and faces real limits. Interpreting a full frontier model end to end is far beyond current capability, findings from small models do not always transfer to large ones, and verifying that an identified circuit is the true cause of a behaviour rather than a correlate is methodologically difficult. Nonetheless, mechanistic interpretability is widely regarded as one of the more promising routes toward trustworthy and verifiable AI systems.

References

1. Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, Anthropic.
2. Templeton, A. et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. Anthropic.
3. Wikipedia contributors. (2025). Mechanistic interpretability. en.wikipedia.org.