MLPs: The Other Half of Every Block

The MLP (multi-layer perceptron, or feed-forward network) is the second sublayer in every transformer block. It contains ~⅔ of the model’s parameters and operates position-wise: each token’s residual stream vector is processed independently.

This post defines the MLP, derives the key-value memory interpretation (Geva et al., 2021) that underlies most modern MLP interpretability, covers neuron archetypes and superposition, and connects the framework to factual editing (ROME/MEMIT) and sparse autoencoders.

Demo: neuron activations

Cycle through neurons. Some are monosemantic (Python keywords, capital letters, France-related contexts). Some are polysemantic, firing on multiple unrelated concepts. The polysemantic case is explained by superposition (below).

Definition

For input $x \in \mathbb{R}^{d_\text{model}}$ at one position, a transformer MLP computes:

where:

• $W_\text{in} \in \mathbb{R}^{d_\text{ffn} \times d_\text{model}}$
• $W_\text{out} \in \mathbb{R}^{d_\text{model} \times d_\text{ffn}}$
• $\sigma$ is a non-linearity (GeLU, ReLU, or in modern models SwiGLU)
• $d_\text{ffn} = 4 \cdot d_\text{model}$ is the standard ratio

Sizes:

Three properties:

1. Position-wise. No mixing across token positions. Operates in parallel on each token’s residual vector.
2. Up-projection then down-projection. Hidden width $4\times$ wider than the residual stream. Storage capacity scales with $d_\text{ffn}$.
3. Non-linearity is essential. Without $\sigma$, two stacked linear layers collapse to one and the MLP cannot represent any nonlinear pattern.

Key-value memory interpretation

Decompose $W_\text{in}$ row-wise and $W_\text{out}$ column-wise:

• Row $n$ of $W_\text{in}$, written $k_n^\top$, is a vector in $\mathbb{R}^{d_\text{model}}$: the key of neuron $n$.
• Column $n$ of $W_\text{out}$, written $v_n$, is a vector in $\mathbb{R}^{d_\text{model}}$: the value of neuron $n$.

Then:

The MLP is a sum of $d_\text{ffn}$ scaled value-vectors, where each scaling coefficient is a non-linearly gated dot product of $x$ with the corresponding key.

Equivalently:

• The key $k_n$ tests whether $x$ matches a specific pattern (large $k_n^\top x$ ⇒ match).
• The non-linearity gates: only neurons whose match exceeds threshold contribute.
• Each contributing neuron writes its value $v_n$ to the residual stream, scaled by activation.

This is a soft, sparse key-value lookup over a learned database of $d_\text{ffn}$ entries per block. (Geva et al., 2021)

Neuron archetypes

Empirically, MLP neurons fall into recurring categories:

Surface-feature neurons (early layers)

Fire on lexical patterns: capital letters, punctuation, specific morphemes, code-syntax tokens. Their values write tags downstream blocks consume.

Syntactic neurons (mid layers)

Fire after grammatical patterns: possessives, definite articles, sentence-initial positions. Values bias the next-token distribution toward syntactically valid continuations.

Factual-recall neurons (mid-to-late layers)

Encode specific facts. Meng et al. (2022, ROME) demonstrated that “the Eiffel Tower is in Paris” can be located to a small set of neurons in mid layers and surgically edited (so the model claims the Eiffel Tower is in Rome) by modifying $W_\text{out}$ columns at those positions.

Abstract / semantic neurons (late layers)

Fire on higher-level patterns: sentiment, sarcasm, discourse markers. Harder to characterize from top-activating examples alone.

Uninterpretable from top examples

A non-trivial fraction of neurons have no clean concept-level description. Often these are polysemantic.

Polysemanticity and superposition

Most real neurons are polysemantic: top-activating contexts span multiple unrelated concepts.

Elhage et al. (2022, “Toy Models of Superposition”) explain why. When features are sparse (most off most of the time), a $d$-dimensional space can represent ~$d / \log d$ features by overlapping them at non-orthogonal angles. The non-linearity in the MLP allows partial recovery: only one feature in a superposed pair is typically active in any given input, so interference is bounded.

Consequences:

1. The “real” interpretable features are directions (linear combinations of neurons), not single neurons.
2. Reading individual neuron activations gives a tangled, polysemantic picture.
3. To recover monosemantic features, train an overcomplete dictionary on cached activations: a sparse autoencoder (SAE).

Bricken et al. (2023, “Towards Monosemanticity”) and Templeton et al. (2024, “Scaling Monosemanticity”) trained SAEs on Claude 3 Sonnet and recovered millions of monosemantic features ranging from “the Golden Gate Bridge” to “code with security vulnerabilities.”

Direct logit attribution for MLPs

Because each neuron’s contribution to the residual stream is $\sigma(k_n^\top x) v_n$, its contribution to the final logit of token $w$ is:

# in TransformerLens
mlp_act = cache["post", layer, "mlp"]      # [seq, d_ffn]
W_out = model.W_out[layer]                 # [d_ffn, d_model]
W_U = model.W_U[:, answer_id]              # [d_model]
neuron_dla = mlp_act[-1] * (W_out @ W_U)   # [d_ffn]
top_neurons = neuron_dla.argsort(descending=True)[:10]

Why MLPs hold the knowledge

Three lines of evidence support the claim that factual knowledge lives in MLPs:

1. Parameter share. MLPs are ~⅔ of total parameters. Most learned content is statistically there.
2. Editing. ROME and MEMIT edit specific facts by modifying MLP weights at specific layers (typically mid-layers, around layer 5–8 in GPT-2 medium). Editing attention weights does not produce the same effect.
3. Causal tracing. Meng et al. (2022) corrupt subject tokens, then restore individual layers’ activations one at a time and measure which restoration recovers the correct prediction. The signal localizes to mid-layer MLPs.

A clean operational summary: attention moves information; MLPs add new information. Both contribute additively to the residual stream. Their roles are complementary.

Activation functions in modern models

SwiGLU adds a gating branch:

This requires three matrices instead of two, but the key-value interpretation extends: each neuron’s “key” is now a (gate, input) pair, and the value is still the corresponding $W_\text{out}$ column. Most MLP interpretability tooling generalizes with minor modification.

What we have so far

All four components communicate exclusively via the residual stream. Every interpretability tool in this series operates on that interface.

The next post runs a full forward pass through GPT-2 small, end-to-end, with concrete numbers at every stage.

Resources

Foundational papers

• Transformer Feed-Forward Layers Are Key-Value Memories
  Geva et al., 2021 · the key-value framing
• Locating and Editing Factual Associations in GPT (ROME)
  Meng et al., 2022 · causal tracing + rank-one MLP edits
• Mass-Editing Memory in a Transformer (MEMIT)
  Meng et al., 2023 · scaling ROME to thousands of edits
• Toy Models of Superposition
  Elhage et al., Anthropic 2022 · why polysemanticity is rational
• Towards Monosemanticity
  Bricken et al., Anthropic 2023 · SAEs on a 1-layer transformer
• Scaling Monosemanticity
  Templeton et al., Anthropic 2024 · SAEs on Claude 3 Sonnet, millions of features
• GLU Variants Improve Transformer
  Shazeer, 2020 · why SwiGLU replaced GeLU in modern LLMs

Tools and code

• Neuronpedia
  browse top-activating contexts for MLP neurons and SAE features across many models
• SAELens
  train and analyze sparse autoencoders on any HF transformer
• ROME · code & demo
  Bau Lab · reproduce factual editing in GPT-2 / GPT-J
• TransformerLens · MLP analysis
  cache MLP activations, decompose neuron contributions to logits
• Zoom In: An Introduction to Circuits
  Olah et al., Distill 2020 · the original feature-and-circuit framing (vision)