- Type
- AI research subfield
- Parent field
- Explainable AI, AI safety
- Core objects
- Features and circuits
- Key contributor
- Anthropic and academic labs
- Goal
- Reverse-engineer model internals
- Related
- Explainable AI, Sparse autoencoder
- Type
- AI research subfield
- Parent field
- Explainable AI, AI safety
- Core objects
- Features and circuits
- Key contributor
- Anthropic and academic labs
- Goal
- Reverse-engineer model internals
- Related
- Explainable AI, Sparse autoencoder
Mechanistic interpretability is a subfield of explainable artificial intelligence that aims to understand the internal workings of neural networks by reverse-engineering their concrete computations into human-understandable parts. Rather than treating a model as a black box and only studying its inputs and outputs, mechanistic interpretability seeks to identify the specific structures, representations and algorithms a network uses. Its central objects of study are features, which are interpretable properties encoded in a model's internal activations, and circuits, which are the connected computations that combine features to produce behaviour.
Motivation
Modern large language models are trained, not programmed, so their internal decision procedures are not written down anywhere and must be discovered. This opacity is a concern for AI safety. If researchers cannot see why a model produces a given output, they cannot easily verify that it is reasoning honestly, detect deceptive or misaligned behaviour, or predict how it will act in novel situations. Mechanistic interpretability responds by attempting to build, in effect, a microscope for neural networks, so that their computations can be inspected in the way a biologist inspects cells.
Features
A feature is a direction in a model's internal activation space that corresponds to a human-interpretable concept, such as a reference to a particular city, a programming syntax pattern, or an abstract idea like deception. A key obstacle is superposition, the phenomenon in which a network packs far more features into its neurons than it has neurons, so that individual neurons respond to many unrelated concepts at once. To untangle this, researchers use sparse autoencoders, which decompose the dense activations into a much larger set of sparse, individually meaningful features. Using this method, interpretability teams have extracted tens of millions of features from production language models, including features that activate on specific concepts across languages and modalities.
Circuits
Once features are identified, the next step is to trace how they interact through the network's layers to carry out a computation. A well-known early example is the induction circuit found in small attention-only transformers. Two attention heads in consecutive layers cooperate: the first head looks backward to find where the current token previously appeared, and the second head copies the token that followed it, allowing the model to continue a repeated pattern. This kind of finding demonstrates that meaningful algorithms can be isolated inside a network and described precisely.
Progress and tools
Anthropic has been a leading contributor to the field, publishing extensively through its Transformer Circuits work and releasing circuit-tracing tools as open source in 2025. The research community has applied these tools to open-weight models such as Gemma and small Llama variants, extending mechanistic analysis beyond a single lab. Related efforts include representation engineering, which studies and steers high-level concept directions, and activation steering, which nudges model behaviour by adding feature directions during inference.
The field remains young and faces real limits. Interpreting a full frontier model end to end is far beyond current capability, findings from small models do not always transfer to large ones, and verifying that an identified circuit is the true cause of a behaviour rather than a correlate is methodologically difficult. Nonetheless, mechanistic interpretability is widely regarded as one of the more promising routes toward trustworthy and verifiable AI systems.
Mechanistic interpretability aligns closely with Malaysia's emphasis on responsible and trustworthy AI. The Malaysia AI Governance and Ethics guidelines, coordinated through the Ministry of Science, Technology and Innovation and the National AI Office, stress transparency and accountability in AI systems. Interpretability research provides the technical foundation for such principles, offering methods to inspect why a model behaves as it does rather than relying on outputs alone.
For regulated sectors, this matters concretely. Bank Negara Malaysia expects financial institutions deploying AI in credit scoring or fraud detection to be able to explain and justify automated decisions. While mechanistic interpretability is still a research frontier rather than a compliance tool, it points toward the deeper forms of model understanding that regulators such as the Securities Commission and Bank Negara increasingly ask for.
Malaysian universities and research institutes, including MIMOS and university machine-learning groups, can contribute to and draw on this globally open field, since interpretability tools released by labs such as Anthropic are freely available. As Malaysia develops sovereign models like ILMU, the ability to audit their internal behaviour supports both public trust and the national goal of safe AI adoption across government and industry.