Recurrent Memory: Write, Erase, Decay, Read

Part 4 of 4 on how a transformer remembers. KV cache · Linear attention · Kernelization · → Recurrent memory

Linear attention compresses the KV cache into one fixed-size board, and the price is that everything written lands on top of everything else. From 2020 to 2025, the entire architecture line has been doing one thing: teaching that board how to forget. Four operations, four knobs — drag them yourself and DeltaNet, Mamba-2 and KDA turn out to be the same equation on different settings.

Everything fits on one board

Attention keeps every token. That’s the whole trick and the whole bill: the KV cache grows with sequence length, and so does the cost of reading it back.

Linear attention makes a different trade. Instead of a growing list, keep one fixed $d\times d$ matrix $S$. Write into it, read out of it. Cost per token is constant, memory is constant, and the sequence can be as long as you like.

The bill just arrives somewhere else. A fixed-size board means every write lands on top of every other write. Nothing gets its own slot; everything is superposed. So the interesting question stops being how do I write and becomes how do I forget.

That’s the whole post. Four things you can do to one board — write, erase, decay, read — and by the end, one equation that covers five years of architectures.

01 One full round trip

The minimal unit of recurrent memory has two moves. Write: slam the outer product of a $(k,v)$ pair onto the board. Read: multiply the board by a query.

\[\underbrace{S \leftarrow S + k^{\top}v}_{\text{write}}\qquad\qquad \underbrace{o = qS}_{\text{read}}\]

$k, v, q$ are row vectors of length $d$, so $k^{\top}v$ is $d\times d$ and $qS$ is $1\times d$. Does it actually work? Query with the exact key you wrote with:

\[kS = k\,(k^{\top}v) = (k k^{\top})\,v = \|k\|^2\,v\]

The direction comes back perfect. The magnitude is off by a factor of $|k|^2$. Drag the slider and watch that factor:

02 If all you can do is add

The board has a plus. It has no minus. That alone causes two problems, and both of them are fatal.

Problem one: you can’t fix a mistake. Write twice with the same key — a variable that got reassigned, say — and a pure-accumulation board ends up holding both values stacked on top of each other. The read gives you their sum, not the latest one. Flip the switch to see the difference:

Problem two: it fills up, and there’s no getting off. The board is $d\times d$, and in $d$-dimensional space there are at most $d$ mutually orthogonal directions. The $(d+1)$-th key necessarily has nonzero inner product with something already stored. Crosstalk isn’t bad luck, it’s arithmetic. Here’s a $d=6$ board with more and more facts packed into it — watch mean recall collapse. And flip the switch: it separates the two diseases, and the second one is where erasing actually earns its keep.

03 Delta Rule: read before you write

The fix is embarrassingly plain. It’s how you’d update a hash table: look at what’s stored under this key, then write only the difference. Three lines.

\[\begin{aligned} v_{\text{old}} &= k_t S_{t-1} &&\text{// ① read}\\ u_t &= \beta_t\,(v_t - v_{\text{old}}) &&\text{// ② the delta}\\ S_t &= S_{t-1} + k_t^{\top}u_t &&\text{// ③ write back} \end{aligned}\]

Expand that and a nicer form falls out (with $|k|=1$):

\[S_t = \big(I - \beta_t\,k_t^{\top}k_t\big)\,S_{t-1} \;+\; \beta_t\,k_t^{\top}v_t\]

That $(I-\beta k^{\top}k)$ on the left is a generalized Householder matrix. At $\beta=1$ it is exactly the orthogonal projection that annihilates the $k$ direction — and leaves every memory orthogonal to $k$ completely untouched, not one hair moved. At $\beta<1$ it’s a partial version: it scales the $k$ component down instead of deleting it. (At $\beta=2$ you’d get the true Householder reflection; learned gates stay in $[0,1]$.)

The linearity is also why this idea survived. The erase step is an honest linear map, not an elementwise if-else, so it can be reassociated into chunkwise-parallel form and actually run fast on a GPU. That’s what Yang et al. did to DeltaNet in 2024 — and it’s why a 2021 idea became a 2025 architecture.

04 Decay: let old memories fade on their own

The Delta Rule has a blind spot: it can only forget things that have a replacement coming in. Topic switched, section finished, you just want the room back — it can’t help you, because there’s no new $v$ to write.

Mamba-2’s answer is blunt. Every step, multiply the whole board by something less than 1.

\[S_t = \alpha_t\,S_{t-1} + k_t^{\top}v_t\]

A token written at time $x$ — how much of it is left at $x+t$? Multiply the $\alpha$’s along the way:

\[\prod_{i=x}^{x+t}\alpha_i \;=\; \exp\!\Big(\sum_{i=x}^{x+t}\log \alpha_i\Big)\]

The right-hand form matters more than it looks. A running product becomes a prefix sum of logs, and prefix sums parallel-scan. That is the entire reason gating fits inside a chunkwise kernel at all.

05 The mixing board

One equation covers the whole line from 2020 to 2025:

\[S_t \;=\; \underbrace{\mathrm{Diag}(\boldsymbol{\alpha}_t)}_{\text{decay}}\;\underbrace{\big(I - \beta_t k_t^{\top}k_t\big)}_{\text{erase}}\;S_{t-1} \;+\; \underbrace{\beta_t\,k_t^{\top}v_t}_{\text{write}}\]

Turn erase off and pin $\alpha$ to 1 — that’s 2020 linear attention. Turn erase on — DeltaNet. Leave erase off but pull $\alpha$ down — gated linear attention, the Mamba-2 shape. Both at once — Gated DeltaNet. Swap the scalar $\alpha$ for a per-channel vector — KDA, the layer Kimi Linear is built out of.

Go dial it yourself. The panel runs a 10-step write script — including one overwrite with a repeated key, and one write past capacity — then reads every fact back and scores it.

So what did five years actually buy

Look at the equation again and the history reads as one sentence: linear attention was born able to write, and everything since has been about learning to forget.

2020 gave you addition. 2021 added a subtraction that is surgical but reactive — it can only erase where something new is arriving. 2024 added a decay that is bulk but blind — it can clear room, but it fades the function signature you still need along with the temp variable you don’t. Multiplying them together fixed neither weakness individually; it just meant the model no longer had to pick one. And 2025’s move — per-channel $\alpha$ — is the smallest-looking change with the biggest consequence: the model stops choosing only how much to forget and starts choosing which dimensions to forget in.

But dial the demo past capacity and you’ll see the thing none of this fixes. Nine facts into a six-dimensional board, and every rule is losing information. The decaying rules even score worse on “all facts” — because they’ve deliberately written off the oldest entries to keep the newest ones at near-perfect recall. That’s not a defect. That is the correct trade, and it’s why the last knob matters at all.

Which is also why nobody ships a pure linear stack. Kimi Linear interleaves three KDA layers for every one full-attention layer; Qwen3-Next builds the same shape out of Gated DeltaNet. The recurrent layers handle the near, the frequent, the constant-cost majority; the occasional full-attention layer goes back and fishes out whatever the board decided to sacrifice. Linear attention was never a replacement for attention. It’s the cheap 75% of the stack — and getting that 75% right turned out to be a five-year argument about forgetting.

---

A $d\times d$ board, a plus, a minus, and a fade. Everything else is scheduling.

References

• Linear attention — Katharopoulos et al., Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (arXiv:2006.16236)
• Delta rule / fast weights — Schlag, Irie, Schmidhuber, Linear Transformers Are Secretly Fast Weight Programmers (arXiv:2102.11174)
• Making DeltaNet fast — Yang et al., Parallelizing Linear Transformers with the Delta Rule over Sequence Length (arXiv:2406.06484)
• Gating — Yang et al., Gated Linear Attention Transformers with Hardware-Efficient Training (arXiv:2312.06635) · Dao & Gu, Transformers are SSMs (Mamba-2, arXiv:2405.21060)
• Both at once — Yang, Kautz, Hatamizadeh, Gated Delta Networks: Improving Mamba2 with Delta Rule (arXiv:2412.06464)
• Per-channel gating — Moonshot AI, Kimi Linear: An Expressive, Efficient Attention Architecture (arXiv:2510.26692)

Every panel above is live, interactive and bilingual — drag the knobs, switch the language, and the demos follow.