Mechanistic interpretability

What it is, and what it is not

A trained network is a large numerical artefact whose behaviour nobody designed. Mechanistic interpretability treats it as a system to be reverse engineered: find the components, work out what each represents, and describe how they compose into an algorithm.

It is distinct from the broader interpretability tradition of feature attribution — saliency maps, SHAP values, 'which input mattered'. Those describe input–output relationships. Mech interp asks about the internal computation, which is a stronger claim and correspondingly harder to establish.

The vocabulary you need

Superposition, and why neurons are messy

A model has far more useful features to represent than it has dimensions. The resolution is superposition: features are stored as near-orthogonal directions in a space too small to hold them all orthogonally, tolerating a little interference in exchange for capacity.

The consequence is that individual neurons are usually polysemantic — a single unit fires for several unrelated things, because it participates in several superposed directions. This is why 'find the neuron for X' mostly fails, and why sparse autoencoders, which try to recover a larger set of sparse, more monosemantic directions from the dense activations, became the field's main tool.

How mech interp connects to safety

Four concrete uses. Detecting dormant capability: behavioural testing cannot show that something absent from your test set is absent from the model, but finding the mechanism can show it is present. Verifying fixes: after safety training, does the unsafe behaviour's mechanism still exist and merely go unused? Auditing: is the model representing something it should not, such as the user's inferred demographics? Understanding failure: a mechanistic account of why a jailbreak works suggests a targeted fix rather than another round of adversarial examples.

None of these are solved. But each is a question behavioural evaluation cannot answer even in principle, which is the honest case for the field.

Theoretical and empirical styles

Theoretical work reasons about what architectures can represent and what training dynamics should produce — the mathematical framework for transformer circuits, toy models of superposition, analyses of what attention can and cannot compute. It produces predictions and clean concepts, at the risk of describing an idealisation rather than a real model.

Empirical work measures actual trained models — probing, ablating, patching, clustering, training autoencoders on activations. It produces findings about systems people deploy, at the risk of accumulating correlations without an organising account.

The productive pattern is alternation: a theoretical prediction that a component should exist, an empirical search that finds it, an intervention that confirms its role. Induction heads are the canonical example of that loop closing.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()

# 1. OBSERVE — capture an activation with a forward hook.
acts = {}
def save(name):
    def hook(module, inputs, output):
        acts[name] = output.detach()
    return hook

h = model.transformer.h[6].mlp.register_forward_hook(save("mlp6"))
with torch.no_grad():
    model(**tok("The Eiffel Tower is in Paris", return_tensors="pt"))
h.remove()
print(acts["mlp6"].shape)      # [1, T, 768]

# 2. INTERVENE — replace the same activation and see whether output changes.
def zero_mlp(module, inputs, output):
    return torch.zeros_like(output)

h = model.transformer.h[6].mlp.register_forward_hook(zero_mlp)
with torch.no_grad():
    ablated = model(**tok("The Eiffel Tower is in", return_tensors="pt")).logits
h.remove()

# 3. COMPARE — the change in the target token's logit is the effect size.
# Repeat across layers to get a profile, and across seeds and inputs before
# believing any of it. That loop is the whole method.

Note: Always remove hooks. A forgotten hook stays attached and silently corrupts every subsequent forward pass — the single most common bug in this kind of work.

The criticisms, taken seriously

Scale. Careful circuit analysis has been done on small models and narrow tasks; frontier models are three orders of magnitude larger, and it is genuinely unclear whether the methods extend or whether the object of study is different at that size.

Faithfulness. An explanation that predicts the outputs may not describe the computation. Ablation results are confounded because removing a component takes the model off-distribution, and it may fail for that reason rather than because the component did the thing you claimed.

Cherry-picking. A clean circuit for a narrow behaviour on a small model is publishable; the many attempts that produced nothing interpretable usually are not. Without those, the base rate is unknown.

Practical impact. The field's safety case is largely prospective. Concrete interventions that demonstrably improved a deployed system's safety through mechanistic understanding remain scarce.