Okay, quick thought experiment. Two ways to figure out a magic trick.
One, you watch it a thousand times, log what works, and build a model of the behaviour. Two, you get the magician to show you their hands, frame by frame, and see the mechanism.
Most AI research is watching the trick. Mechanistic interpretability is grabbing the magician’s wrist. Completely different game, and we’re about to play it.
What “mechanistic” actually means
“Interpretability” in AI is a big tent. It covers:
- Saliency maps. “Which pixels made the model look at this image?”
- LIME / SHAP. “Which features statistically mattered for this prediction?”
- Probing. “Does this layer seem to know what country a city is in?”
All of these are real, useful, shipped-in-production tools. They tell you what the model seems to do. They just don’t tell you how it does it. Both questions matter, they’re just different questions.
Mechanistic interpretability asks a totally different question:
Goal: reverse engineering. Not “poke it and see what falls out.” Open it up and read the code.
We can see which words mattered. We don't know why, or what "bank" even meant to the model.
bankbank is polysemous, two candidate senses.We can see the model choosing between two meanings based on surrounding context. That's mechanistic.
How this demo works
The behavioral panel highlights tokens using hard-coded saliency weights, mimicking what a tool like LIME or SHAP outputs. The mechanistic panel runs a scripted feature competition between two pre-built "senses" of bank; the winner is determined by which context words appear (geographic vs financial lexicon). Real circuits in GPT-2 behave remarkably like this, see the IOI paper (Wang et al. 2022) for the actual version.
Three things MI wants to find
Features. What concepts does the model represent?
Every neural network learns to represent stuff. “Dog”. “Curved line”. “Toxic language”. “The year 1990”. These are features. MI wants to find them, label them, and figure out how the model uses them.
Okay so here’s the idea that broke my brain the first time: a feature is a direction in activation space. Not a neuron. A direction.
If you’ve got 768 neurons in a layer, one feature might be “cat” = (0.3, -0.8, 0.1, …). A specific combination of activations across those 768 dimensions. When cat-like inputs show up, the activations shift in the “cat direction”. Read that again, it’s weird and it’s important.
Features aren’t neurons. They’re directions made of neurons.
Vision models have well-catalogued features now:
- Low-level: edge, colour, texture detectors
- Mid-level: curves, corners, eyes
- High-level: faces, cars, specific dog breeds, even “pose”
Language models have them too. “This is a URL”. “Subject of the current clause”. “We’re inside quotation marks”. “This code needs a closing bracket”. Recent sparse-autoencoder work has surfaced millions of these.
Circuits. How do features connect?
A circuit is a small sub-network that computes a specific function. Like: “this combination of 4 attention heads, working together, detects whether someone’s referring back to a noun from two sentences ago.” MI wants to find these and understand them completely.
Transformers don't read a sentence left-to-right. Every token looks at every other token, through dozens of attention heads. Each head learned a different job. Click a head.
How this demo works
Five heads are curated based on real patterns discovered in GPT-2. Each head has a hand-specified attention matrix: rows are query tokens (doing the looking), columns are key tokens (being looked at). Arc width = attention weight. Induction heads were the first fully characterized circuit in a transformer, see Olsson et al. 2022, "In-context Learning and Induction Heads."
The example above isn’t hypothetical. Those patterns are the kind of thing researchers have actually identified in GPT-2. An induction head is a specific two-attention-head circuit that was the first full circuit ever characterised in a transformer. It’s how models do in-context learning: see a pattern, complete the pattern.
A real circuit description reads like: “Layer 7 Head 4 writes information about the previous token into the residual stream; Layer 9 Head 2 reads that information and moves it forward whenever the attention pattern matches.” Dry, mechanical, reproducible. That’s the vibe. I know, so romantic.
Universality. Do the same features show up in different models?
If GPT and Gemini and a vision model all develop the same “curve detector” circuit, that’s a big deal. Not arbitrary. More like natural categories that emerge from learning.
Some universality has already been found. InceptionV1 and CLIP (two very different vision models) both develop curve detectors with similar structure. Multiple transformer families grow induction heads at roughly the same training stage. The space of useful features, given natural data, seems constrained. There are “correct” things to learn, and good models find them.
Big if true. Means MI discoveries should generalise: understand how one model does X, other models probably do X similarly.
Why this is genuinely hard
A modern LLM has:
- ~70 billion parameters (for medium-sized)
- thousands of neurons per layer
- dozens of layers
- no labels on anything
The model didn’t ship with documentation. Its internal structure isn’t organised in human-friendly ways. Two specific problems make life difficult:
In a real transformer, many neurons fire for wildly different concepts. This is a real finding, neurons that light up for cats and legal documents and the digit 7. Click any input to see which neurons wake up.
How this demo works
Twelve neurons are shown. Each has a feature dictionary, a list of inputs it fires for, with a strength value. When you click an input on the left, every neuron lights up proportionally. You'll see most neurons fire for more than one thing (that's polysemanticity). Anthropic's Towards Monosemanticity paper (2023) showed you can decompose such neurons into cleaner features using sparse autoencoders.
Superposition is the technical reason for polysemanticity. 512 neurons, 5000 concepts, you have to share. The clever (and spooky) thing: the model stores concepts at angles to each other in high-dim space. A careful reader can disentangle them approximately. Not cleanly. Not perfectly. Just well enough.
Which is why naive interpretation (“what does neuron 734 do?”) gives confused answers. The right question is “what are the feature directions in this 512-dim space?” And that’s what sparse autoencoders finally cracked.
The analogy that captures it
Think about a compiled program. You have the binary. Millions of 1s and 0s. You can run it. You can test it. You can’t read the source directly.
A reverse engineer’s job: take the binary, reconstruct the algorithms. “Ah, this block is doing sorting. That block is encryption. Here’s where the password check is.”
That’s mechanistic interpretability. Except the binary is a neural network, and the “source code” we’re looking for is a description of what algorithms it learned.
The exciting part: unlike compiled software, the algorithms the network learned weren’t designed by a human. We don’t know what we’ll find. Some of what has been found:
- A two-head circuit that copies information across context (induction heads)
- A multi-head circuit that handles “John gave Mary the book. Mary gave it back to ___” (the IOI circuit)
- A mod-arithmetic algorithm that emerges suddenly during training (grokking)
- A feature that fires for “the concept of the assistant persona, including its restrictions and imprisonment metaphors” (Claude 3 Sonnet)
Every one of those was a surprise. We’re doing discovery on artefacts we built but didn’t design.
Where behavioural methods fit
They aren’t wrong. They’re complementary. A good interpretability investigation usually looks like:
- Behavioural tools find what. SHAP values, saliency maps, probing. Cheap and fast. They give you a hypothesis: “the model seems to care about token X here.”
- Mechanistic tools explain why. Activation patching, circuit extraction, ablation. Slow and expensive. They give you a causal account.
- Mechanistic validation. Disable the alleged circuit, see if the behaviour breaks. Insert it elsewhere, see if it reproduces.
The mechanistic part is what turns a suggestive correlation into a rigorous claim. That’s the specific thing MI adds.
What success looks like
Finish line. We’ve succeeded when:
- We can identify every feature a model represents.
- We can trace every computation: for a given input, here’s the exact path of information through the network.
- We can predict failures before they happen: “this model will fail on inputs with property X because its circuit for handling X is weak.”
- We can verify: “this model genuinely doesn’t have a deceptive capability hidden inside, because we’ve checked the circuits.”
Not there yet. Maybe 5% of the way, if I’m being generous. But the progress in the last three years has been kind of wild.
Research referenced in this post
- Mechanistic Interpretability, Variables, and the Importance of Interpretable Bases
- Zoom In: An Introduction to Circuits
- A Mathematical Framework for Transformer Circuits
- In-context Learning and Induction Heads
- Toy Models of Superposition
- Towards Monosemanticity
- 200 Concrete Open Problems in Mechanistic Interpretability
- The Building Blocks of Interpretability