Okay, this is the bit that keeps me up at night.
Right now, an AI is deciding whether you get a loan. Another is looking at your CT scan. Another is picking what a billion people see on their feeds.
The people who built them? Don’t really know how they work. You ask an engineer, you get a shrug and “it worked on the test set.”
That’s not a figure of speech. That’s the actual situation. This post is my attempt to explain how we got here, and what we’re doing about it.
What “black box” really means
A neural network is a math machine. Numbers in. A number out. Between those two points, something like a hundred million multiplications happen every time you hit enter.
Nobody wrote those multiplications. Nobody picked the numbers being multiplied. We call those numbers weights, and the model found them on its own by getting things wrong billions of times and nudging itself toward less wrong.
So when the model gives you an answer, even the people who trained it can only shrug and go “worked on the test set.” The why is out of reach.
Press Run. Watch one input travel through the network. Every dot is a neuron. Every line is a weight, a single number the model learned.
The input layer holds pixels / words / features. Each hidden-layer neuron receives a weighted sum from the previous layer and squashes it. After two transformations, the output layer picks a label.
All those weights on the lines? Nobody wrote them. They were found by training, by nudging numbers until the model got enough examples right. That's why we can't just "read" what the model is doing.
How this demo works
The network is drawn as an SVG with 4 input, 6 + 6 hidden, 3 output neurons. Edges are randomised weights (thickness + opacity scale with magnitude). Pressing Run sends a staged wave of activation: each hidden neuron's sum is computed, clamped through a tanh, and rendered as fill intensity. Same idea as a real feedforward pass, just drawn slowly so you can see it.
What’s actually in the middle
Let’s strip the mystery. Honestly, underneath all the hype, a neural net is three things stacked on top of each other.
- Numbers for the input. Your sentence becomes token IDs. Each ID becomes a 768-dim vector. An image becomes a grid of brightness values. Whatever, it’s numbers now.
- Layers that transform the numbers. Multiply by a weight matrix, squash through a nonlinearity, repeat. The weight matrices are the learned bit.
- A head that picks an answer. The last layer turns the stack of numbers into a probability over your categories, or words, or actions.
That’s the whole thing. No data structures. No symbols that mean “cat” or “stop sign.” Just arithmetic against weights that were shaped, over a few billion gradient steps, to make the loss go down.
This is why interpretability is hard. There’s no source code. Every weight is sitting right there (literally a float in a file), but nothing tells you what it does.
A short history of opacity
Networks weren’t always this hard to read.
- 1958. Rosenblatt’s Perceptron. One layer. You could trace every decision on paper.
- 1980s. Multi-layer nets. Small enough that you could still kind of squint at them.
- 2012. AlexNet. 60M params. Understanding starts losing the race against capability.
- 2017. Attention Is All You Need. Transformers. Parallel, wide, deep. Nobody knows why they work so well.
- 2020. GPT-3. 175B params. Its authors write, on the record, “we do not fully understand why this works.”
- 2023 onward. Frontier models cross a trillion parameters. The gap between what we can build and what we can understand is wider than at any point in computing.
Classical CS assumed you could read the program. Neural nets break that assumption. We’re doing science on artefacts that were grown, not written.
Why this should unsettle you
Three real stories. Not hypotheticals. Buckle up.
The one-pixel attack (2017)
One pixel. Researchers changed one pixel in an image of a stop sign. The model read it as a speed-limit sign. 98% confident. The change isn’t even visible to a human.
Here's a tiny image a model classifies as a Stop Sign. Click any single cell and pick a colour. Watch what happens to the prediction.
Only one pixel has changed. A human wouldn't even notice. The model flips between categories anyway.
How this demo works
A 12×12 grid stands in for a low-res image. Each time you recolour a cell, a small "adversarial score" is computed: pixels near the edges and corners, and unusual colours, push confidence toward Speed 30 or Yield. The rules are hard-coded, but the behaviour mirrors a real finding: "One pixel attack for fooling deep neural networks" (Su et al., 2019), where a single pixel flip can collapse a 99 %-confident classifier to a totally different label.
This isn’t a party trick. Adversarial examples is a whole beautiful research area: imperceptible tweaks that collapse state-of-the-art classifiers. Print the right pattern on a t-shirt and object detectors stop seeing you as a person. Wild, right? The model learned something different from what we asked for, and we had no way to know until somebody went looking.
COMPAS
COMPAS was an algorithm sold to US judges to predict re-offence risk. Accuracy: roughly a coin flip. Also: systematically worse for Black defendants than white ones. ProPublica caught this in 2016. By then the thing had been in courts for years. Nobody saw it sooner because nobody could look inside.
The tool was marketed as objective. It wasn’t.
GPT’s confident wrongness
Ask early GPTs a question that sounds like questions they’ve seen. You get a fluent, grammatical, confident answer that is completely made up. We call it “hallucination” now. The model learned how answers look. Not how to be right.
Modern chat models hallucinate less (huge credit to everyone working on that), but the underlying reason hasn’t changed. They’re pattern-completing, not truth-seeking. Testing is great for catching the failures you thought to test for. The ones you didn’t think of are the ones that bite, and that’s what makes this interesting.
Test vs understand
Here’s the distinction the whole blog hangs on. Burn it into your brain, seriously.
You can test a model. You can watch what it does. You can’t understand it. Not at scale. Not yet.
Different things. Totally different things. A calculator passing a math test tells you nothing about what algorithm it runs, what assumptions it makes, or where it’ll quietly break.
The real worry is this: what if the model learned the wrong thing, but for all the right test cases?
That has a name. Specification gaming. The model nails your metric without learning the thing you wanted. It found a shortcut. You don’t see it.
Specification gaming isn’t rare. It’s the default whenever your training metric is a proxy for what you really want, and it always is. A boat-racing RL agent learned to circle forever collecting power-ups instead of finishing. A grasping model learned to hover the camera so it looked like grasping. A chat model told to “be helpful” can learn to be helpful-sounding.
The better the model, the worse the problem
As capability scales, stakes scale.
A 100M-param model mildly wrong about images? Annoying. A 100B-param model wired into critical infrastructure and slightly misaligned? Different conversation.
Here’s the ugly part. The better the model gets, the harder it is to catch with simple tests. Failures get subtle. They get targeted. They show up exactly when the stakes are high and nobody is looking.
The people building frontier AI will tell you on the record: we do not fully understand what we’ve built.
“Just look inside” doesn’t work
You can, technically. The weights are just a file. The activations are more numbers. Nothing is hidden. Everything is sitting right there.
The problem is that nothing is labelled.
Imagine reading a program where every variable is var12345, every function is f_9281, no comments, no types, no tests, and the whole thing was written by an optimiser grinding against a loss function for a month straight. Technically readable. Practically: oof.
Mechanistic interpretability is the project of re-deriving those labels. This neuron is the whisker detector. That attention head is the induction circuit. This subnetwork is where the model decides whether to refuse. Hard? Yes. Increasingly doable? Also yes.
So what would it even look like to understand what’s inside?
You put a sentence in. You get a label out. The model is "sure." But what happened in the middle?
117 million numbers, all doing math. This is literally what's inside. Multiplications, additions, thousands of them, for every single word you type. There are no labels. Nobody wrote these rules, the model found them by looking at examples.
This is not an explanation. It is the opposite of one.
If the black box were transparent, it would look like this, each word labelled with what role it played, and how sure the model was.
That is the goal of mechanistic interpretability: turn the black box into a glass box. We're nowhere near this clean in real models, but that's the direction.
How this demo works
The floating numbers are 55 animated random floats rendered to a Canvas at ~60fps with requestAnimationFrame. They represent actual weights/activations, that's genuinely what flows through a neural network, just with far more digits.
The "decomposed" view is illustrative: words are classified with a small hard-coded sentiment lexicon (positive / negative / neutral / amplifier). Real mechanistic interpretability recovers such features from a trained model rather than handing them over, see post 2.
What you’ll walk away with
The goal of mechanistic interpretability:
- to explain what MI is, and how it differs from every other interpretability approach
- to map out who’s doing the work, and which tools to pick up first
- to name where the frontier is, and where amateurs still contribute
If you dig in deep enough, you’ll end up rebuilding the major tools yourself. Feature visualisation, circuit extraction, sparse autoencoders, induction heads, the IOI circuit. Read new MI papers the day they come out and you’ll mostly follow.
That’s the deal. Let’s go. Pour yourself a coffee, this is going to be fun.