Temporal Intelligence: Thirty-Five Years of State, in One Long Look

Dec 19, 2024 10 min read

Companion note to Temporal Intelligence: Foundation to State-of-the-art Advancements of Sequential Learning Units and Models.

The paper is a survey. Which — full disclosure — I have complicated feelings about, because a lot of surveys read like Wikipedia articles with a bibliography. So this post is the opinionated version. The story I’d tell over coffee, not the neutral one I had to write for print.

The story, in one sentence: the Elman network from 1990 had already committed to the whole idea, and everything since is engineering.

The one idea that hasn’t moved

Take a stream of inputs. Keep some state. Evolve the state as new inputs arrive. Predict from the state. That’s every sequential learner ever built.

Vanilla RNNs, LSTMs, GRUs, transformers, state-space models — they all subscribe to that commitment. What they disagree on is:

What form does the state take? A vector? A cache? A structured continuous flow?
How does the state evolve? Written by a gate? Grown by concatenation? Integrated by a transition matrix?
What compute regime is that evolution cheap in? Sequential? Parallel? Both?

Scrub the year slider and watch the frontier move. The x-axis is calendar year; the y-axis is practical effective context length in log scale.

Interactive · Thirty-five years of sequential learners, one axis

Scrub the year slider. Watch the practical effective-context length climb from tens of steps to hundreds of thousands. Each era's dominant model gets its state visualized on the right.

Year 1990

How this demo works

Effective context length is the practical, "you can actually train and use it" number, not a theoretical bound. The curve is a piecewise fit to what each era actually delivered on published benchmarks — Elman/vanilla RNN (~10-20 steps), LSTM (~100s), Transformer (~2-8k, later 32-128k), SSM/Mamba (100k+). State visualizations are stylized: a small hidden vector for RNN/LSTM, a growing KV cache for transformers, a structured continuous state for SSMs.

There are four visible knees. Each one is a specific idea that unlocked the next era. Let’s go through them.

Knee 1: 1997 — LSTM’s gate

The vanilla RNN is correct in theory: state is passed forward, gradient flows backward, learning happens. In practice, the gradient either vanishes or explodes past about ten steps. You can prove it in math. You feel it after five minutes of training.

The LSTM replaced multiplicative gradient flow with additive flow along a “cell state” that a gate can choose to read from or write to. Gradients now travel through addition, which is much better behaved. Suddenly hundreds of steps became tractable.

Everything else stayed the same. Sequential compute. One step at a time. Slow to train, slow to deploy. But it worked, which none of the vanilla RNNs really did.

Knee 2: 2014-2016 — attention

The seq2seq bottleneck. I wrote about this one earlier and won’t repeat myself. The short version: the encoder compressed everything into one vector, the decoder read from that vector, and information got lost. Attention let the decoder look back at the whole encoder state, per step.

This one wasn’t really a change to the state itself. It was a change to what you’re allowed to read from. Recurrence stayed. But the idea that “the state can include everything you’ve ever seen, and you can query it dynamically” was going to be huge.

Knee 3: 2017 — attention is the whole model

The transformer paper (Vaswani et al.) does one radical thing: remove recurrence entirely. No hidden state that gets passed forward. Instead, at every step, attend over every past step directly.

The compute trade is different: attention is quadratic in sequence length, and the KV cache grows linearly. LSTMs were linear in compute, constant in state size. Transformers pay more per step and remember more per step. On the hardware of 2017, that was a great trade — GPUs love parallelism, and transformer training is embarrassingly parallel across positions in a way LSTM training just isn’t.

That parallelism is why transformers ate everything for the next five years. The idea that state should be a growing cache rather than a fixed vector turned out to be the right call for the hardware of the era.

Knee 4: 2022+ — state-space models

Then context windows started getting really long, and the quadratic-in-length attention cost stopped being cute.

State-space models (S4, Mamba, and family) bring recurrence back — but with the good parts of the transformer era intact. Structured continuous-time dynamics for the state. Parallel training via clever math (associative scan, convolutional view). Linear-time inference because they’re recurrent underneath.

The current frontier is a mix. Hybrids that alternate SSM blocks with attention blocks. Pure-attention scaled to million-token contexts. Pure-SSM competing at the language-model frontier for the first time. The bet isn’t settled.

What the story is actually about

Two axes, running underneath the whole thing:

What compute is cheap. Sequential CPUs → recurrent networks. Parallel GPUs → transformers. Long-context inference → SSMs. Every era’s dominant architecture is downstream of what the hardware makes free.
How you encode time-dependence. Explicitly gated (LSTM), implicitly via a growing cache (transformer), or through structured dynamics (SSM). Each makes different things easy and different things hard.

If you strip out the marketing, “which architecture is best?” is really “which pairing of state-form × compute-regime × task-mix fits my constraints?” There is no universal winner. There will not be a universal winner.

Why the story isn’t over

The current sequence models are stunningly good at predict the next token in a stream where recent context matters most. They’re still not great at:

Very long-range dependencies where the relevant signal is a specific event a million tokens ago. Attention can theoretically reach it. Attention often doesn’t.
Hierarchical time. Events on wildly different timescales, in the same sequence. Human language has this — a novel and a tweet are the same “language” but need different temporal reasoning. Current models don’t differentiate.
World-model rollouts. State that’s a good enough summary that you can plan against it, not just predict against it. This is the thing I care most about right now.

That’s the segue to the papers I’m writing now — Never Lost in the Middle and One Lens, Many Worlds — both of which are asking questions about state that the survey couldn’t quite reach.

Full paper on Google Scholar.