Writing
Notes on world models, interpretability, and the papers I keep returning to. In roughly the order they were written.
2026
One Lens, Many Worlds
World-model research is fragmenting across at least three architectural families, and interpretability tooling doesn't compose across them. We proposed a small type system that fixes the composition problem without asking...
The Full Forward Pass: Putting Every Piece on the Belt
End-to-end execution: tokens → embeddings → six attention+MLP blocks → unembedding. Tensor shapes, layer-by-layer logit-lens trajectory, and the surface area available for mechanistic analysis.
MLPs: The Other Half of Every Block
MLPs hold ~⅔ of a transformer's parameters and act as key-value memories: each neuron is a learned (key, value) pair that adds to the residual stream when the key matches....
Attention: How Every Position Decides Who to Listen To
Attention is a dot-product-based routing mechanism. Each head decomposes into a QK circuit (where to attend) and an OV circuit (what to write back), enabling head-level interpretability.
The Residual Stream: The Belt That Runs the Whole Factory
The residual stream is a per-position running sum that every block reads from and writes to. Because it is additive and lives in a single coordinate frame, it admits direct...
Tokens: The Strange Alphabet Models Actually See
A token is a subword unit drawn from a fixed vocabulary of 30K, 200K entries. Tokenization shapes capability: arithmetic, multilingual coverage, character-level reasoning all depend on it.
The Transformer, Demystified: A Factory Floor That Runs on Language
A decoder-only transformer is a stack of N identical blocks operating on a residual stream of token vectors. Six stations, one conveyor belt, one probability distribution per step.
How Training Works: The Ball Rolling Downhill
Every weight started life as a random number. All the grammar, all the facts, learned by being wrong billions of times. This has a name, and it's the closest thing...
Layers: What Each Floor of the Building Does
Layer 1 sees edges. Layer 5 sees this specific person. Same pixels, different lens at every level. That's why deep learning is deep.
Weights & Connections: Where Knowledge Actually Lives
Every fact, every grammar rule, every pattern a model ever learned is compressed into a pile of numbers. Nobody designed a single one of them.
Neurons & Activations: The On/Off Switch That Isn't
Your brain has 86 billion neurons. GPT-4 has about 1.8 trillion. Same word, not remotely the same thing. And the AI version is stunningly simple.
Who's Doing This, and Why Now?
A quick tour of the labs, the people, and the 8-year sprint that took mechanistic interpretability from a weird hobby to a central safety bet.
What Is Mechanistic Interpretability?
Two ways to understand a magic trick. Most AI research does one. Mechanistic interpretability does the other, and it's a completely different game.
The Black Box Problem
An AI decides whether you get a loan. Another helps diagnose your cancer. The people who built them don't fully know how they work either. That's not a metaphor.
2025
Never Lost in the Middle Again: The U-Shape Is a Training Artifact
Long-context LLMs read the beginning of a document. They read the end. They mostly hallucinate the middle. Turns out the middle isn't hard — the training data just told the...
APEX: Load Balance Was Never the Right Loss
Every MoE model has a load-balancing loss. Every one is doing the same wrong thing at the token level. This post is the ten-minute version of why, and what the...
2024
Temporal Intelligence: Thirty-Five Years of State, in One Long Look
The Elman network from 1990 had already committed to the whole idea. Keep some state. Evolve it as new inputs arrive. Predict from it. Every generation since is engineering —...
Large-Scale PDF Retrieval: The Boring Bits Are Where the Accuracy Lives
Every RAG tutorial hands you 50 lines and a happy demo. At the scale the demo doesn't matter and FAISS starts to earn its name, the accuracy is decided by...
Genetic Algorithms + Deep Learning: Where the Hype Actually Cashes Out
Nine out of ten papers that combine GA with deep learning are hype dressed as synergy. The tenth is legitimately using GA for the exact thing gradient descent can't touch:...
Financial Advisory LLM for India: When 'Speak English' Isn't Enough
A generic assistant will happily answer an Indian user's tax question — in dollars, in US brackets, referencing a 401(k). The whole paper is about closing that gap without pretending...
CAPTCHA With CNN + LSTM: A Small Model, Meanly Trained
Big models overfit. Small models generalize. The whole paper is basically that one sentence, applied to distorted-text CAPTCHAs, backed up by one aggressive augmentation and a loss function nobody uses...
Multi-Lead ECG: The Diagnosis Lives in the Disagreement
A cardiologist doesn't stare at one squiggle. They read twelve of them side by side, and the arrhythmia usually gives itself away in the way the leads disagree. Most ML...
Neural Seq2Seq With Attention: The Bottleneck, and How to Kill It
The old encoder-decoder tried to cram an entire paragraph into a single vector. It went about as well as you'd expect. Attention is the small idea that fixed it, and...