From 2K to a Million Tokens: How Context Windows Actually Grew

A sequel to the four posts on how a transformer remembers — KV cache · Linear attention · Kernelization · Recurrent memory. You don’t need them, but a few sections below lean on them.

GPT-3 shipped with 2,048 tokens of context in 2020. Five years later, 1M is a normal number on a spec sheet, and a couple of models claim 4M or 10M. That’s a 500× jump in five years, and it’s tempting to read it as “GPUs got bigger.” They did, but that’s maybe a fifth of the story.

Most write-ups of this list the methods — RoPE scaling, FlashAttention, ring attention, GQA, sparse attention, hybrids — and stop there. A list tells you what people did. It doesn’t tell you why each one works, or why you need all of them at once. That’s what this post is for.

Here’s the frame I’ll use. A context window is held in place by five things: positions (the model only trusts distances it trained on), compute (attention is $L^2$), memory (activations in training, the KV cache at inference), data (you need text where something far back matters), and evaluation (does the model actually use the window, or just accept it). Every method below removes exactly one of those. Get rid of one, and the next one becomes the wall.

Every panel is live — press play, drag things, switch the language and the demos follow.

0. The window is just what the model practiced on

Start with the most boring fact, because everything else hangs off it. Pretraining doesn’t feed documents one at a time. The pipeline tokenizes everything, glues documents together with an end-of-document token, and chops the stream into rows of exactly $L$ tokens. That’s called packing.

So the model sees position 0, position 1, …, position $L-1$, billions of times each. It sees position $L$ exactly zero times. There’s no rule in the architecture that says “stop at $L$” — the weights would happily run on a longer input. The window is simply the set of positions the model ever got a gradient on. And anything longer than $L$ got cut in half, so the model also never learned to connect the two halves.

Two things to take from that panel. First, the window is a training fact, not an architecture fact — which is good news, because it means you can move it. Second, the triangle. Inside each row, token $i$ scores itself against every earlier token, $L(L+1)/2$ pairs. Double $L$ and you quadruple the work. That’s why nobody just pretrains at 1M from step one.

1. Positions: a set of dials, and the slow ones run out

Almost every current LLM encodes position with RoPE. Split each query and key vector into pairs of dimensions. Pair $i$ of the token at position $m$ gets rotated by the angle $m\,\theta_i$, where

\[\theta_i = \text{base}^{-2i/d}\]

When a query at position $m$ meets a key at position $n$, the rotations partially cancel and only $(m-n)\,\theta_i$ survives in the dot product. So attention sees relative distance. Nice.

The intuition I like: each pair is a dial. With $d=128$ and base 10,000, the fastest dial turns 1 radian (57°) per token — it wraps around every ~6 tokens and is great at telling neighbors apart. The slowest dial turns about $10^{-4}$ radians per token; one full turn takes ~54,400 tokens.

Now train at $L = 4096$. The fast dials go round and round and see every angle thousands of times. The slow dials only ever sweep a sliver of their circle. Then you hand the model a 16K input, and the slow dials point at angles the network has literally never seen. The attention scores that depend on them go weird, and perplexity explodes. This happens before memory is even a concern. Drag the position past the training line and watch the hands go red.

Every fix is one question: which dials do you slow down?

Once you see it as dials, all the position tricks collapse into the same question.

Position Interpolation (Chen et al., 2023) is the blunt answer: slow every dial down by the extension factor $s$, i.e. use position $m/s$ instead of $m$. Now no dial ever leaves its trained range. The cost: the fast dials also slow down by $s$, so two neighboring tokens now look $s$ times closer than they used to. Local word order gets blurry. A short fine-tune (~1,000 steps) mostly repairs it, and this is how early LLaMA got to 32K.

NTK-aware scaling (bloc97, 2023) noticed the fast dials were never the problem — they already wrap many times inside $L$. So instead of dividing positions, raise the base: $\text{base}’ = \text{base}\cdot s^{d/(d-2)}$. The fast dials barely change, the slowest one gets slowed by about $s$, and everything in between is smoothly in between.

Bigger base (ABF) is the same idea done during training rather than at inference: crank the base way up (10,000 → 500,000 in Llama 2 Long and Llama 3; 1,000,000 in Code Llama and Qwen3), then keep training on long data so the model gets used to its new slow dials.

YaRN (Peng et al., 2023) makes the cut explicit. For each pair, ask how many full turns it makes inside the training window, $r_i = L/\lambda_i$. If it wraps more than ~32 times, leave it alone — it’s seen everything. If it doesn’t even finish one turn, interpolate it fully. In between, ramp linearly. Plus one more piece I’ll get to in a second (temperature). It needs ~0.1% of pretraining data to fine-tune, and DeepSeek-V3 and the Qwen family both use it.

LongRoPE (Microsoft, 2024) says: why guess the curve? Search for a separate rescale factor per dimension with an evolutionary search, leave the first few positions uninterpolated, and extend in stages. That’s how Phi-3 got its 128K variants, and the paper reports going past 2M.

In the panel below the bars are dials, fast on the left, slow on the right. Flip between methods and watch which ones get stretched.

2. Even with perfect positions, the softmax gets diluted

Say you’ve fixed positions completely. There’s a second, quieter problem: more keys means more competitors.

One query, one key that actually matters, and $n-1$ distractors with roughly random logits. The softmax gives the right key a weight of about

\[\frac{e^{z^*}}{e^{z^*} + (n-1)\cdot \mathbb{E}[e^{z}]}\]

which goes like $1/n$. At 4K the right key wins. At 1M it’s still the single highest score, but it’s holding a sliver of the total weight, and the output is mostly an average of noise. The entropy of the attention distribution climbs like $\log n$.

The fix is embarrassingly small: sharpen the logits as the context grows. Multiply them by something that rises with $\log n$. YaRN’s version is to scale $q$ and $k$ each by $0.1\ln s + 1$. Llama 4 applies a temperature scaling at inference for long inputs. Same idea everywhere: keep the peak a peak.

3. Or don’t stretch at all: remap the distances

All of section 1 changes the dials. There’s a sneakier option that doesn’t touch them: make sure the model never sees a distance it didn’t train on.

Dual Chunk Attention (An et al., 2024) cuts the sequence into chunks smaller than the training length. Inside a chunk, relative positions are the normal ones, so nearby tokens keep their exact order. Across chunks, the query gets a fixed large position index, so the distance gets capped near the edge of the trained range — the model sees “far, as far as I know” rather than “1.3 million tokens away”. It loses the exact distance between far-apart tokens, which it never knew how to use anyway. A third rule handles the neighboring chunk so locality doesn’t break at the boundary. Every distance the model computes lands inside its trained range. No fine-tuning needed.

This is not a toy. Qwen2.5-1M trains up to 256K and then uses DCA at inference to go 4× further, to 1M.

Llama 4 took the most radical version: in its iRoPE design some layers have no positional encoding at all. Those layers can only match by content, so there’s no dial to run out. The remaining RoPE layers attend locally, where distances stay small. That’s part of how Scout claims 10M.

4. The bill, part one: never write down the $L \times L$ matrix

Positions handled. Now the arithmetic. One head at 128K tokens has a score matrix of $131{,}072^2$ entries — 32 GiB in bf16. Per head, per layer. You can’t store that, and you don’t need to.

FlashAttention (Dao et al., 2022) computes the exact same result in tiles. Load a block of queries into the GPU’s tiny, fast on-chip SRAM, stream key/value tiles past it, and never write the full score matrix to the big, slow HBM. The catch is that softmax needs the max and the sum over the whole row, and you only have one tile at a time. The trick is the online softmax: keep a running max $m$ and running sum $\ell$, and when a new tile brings a bigger max, rescale what you’ve accumulated so far:

\[\begin{aligned} m' &= \max(m,\ \max_j s_j) \\ \ell' &= e^{m-m'}\ell + \textstyle\sum_j e^{s_j - m'} \\ O' &= e^{m-m'}O + \textstyle\sum_j e^{s_j-m'} v_j \end{aligned}\]

Divide by $\ell$ at the end and you get the exact softmax. Watch the numbers below — the running output gets corrected every time a bigger score shows up, and the final answer matches the brute-force one to floating-point precision.

Memory goes from $L^2$ to $L$. Compute is still $L^2$ — FlashAttention makes long context possible, not cheap. Keep that in mind; the rest of the post is about the compute.

5. The bill, part two: one sequence, many GPUs

FlashAttention gets the score matrix out of memory, but activations still grow with $L$, and at some length one GPU just can’t hold the sequence. So split it.

Ring attention (Liu, Zaharia, Abbeel, 2023) puts the GPUs in a ring. Each one keeps its own chunk of queries and holds one chunk of keys/values. Every step, each GPU computes its queries against the KV it currently has, then passes that KV to its neighbor — and the send overlaps with the compute, so the network is mostly free. After $N$ steps, every query chunk has met every KV chunk. No GPU ever held the whole thing. Max length scales with the number of GPUs.

There’s a wrinkle, and it’s a fun one. With a causal mask, the GPU holding the first chunk has almost nothing to do (its queries can’t look at anything later), and the GPU holding the last chunk does all the work. Everyone waits for the slowest one. The fix is like dealing cards from both ends of the deck: split into $2N$ chunks and give GPU $i$ chunks $i$ and $2N-1-i$. Now every GPU gets one cheap chunk and one expensive chunk. Llama 3 does exactly this split.

The other family, DeepSpeed-Ulysses, does an all-to-all right before attention: swap from “each GPU has a slice of the sequence for all heads” to “each GPU has the whole sequence for a few heads”, run attention locally, swap back. Simple, but you can’t split wider than your number of heads.

6. Store less per token: GQA and MLA

Now inference. As the KV cache post showed, every token leaves a key and a value behind in every layer, and that pile grows linearly. Do the arithmetic for Llama 3 70B: 80 layers × 8 KV heads × 128 dims × 2 (K and V) × 2 bytes = 320 KiB per token. At 128K that’s 40 GiB for one sequence. At 1M ($2^{20}$ tokens), 320 GiB. The weights don’t run out first. The cache does.

So the architecture question becomes: how little can each token leave behind?

GQA (and its extreme, MQA) lets several query heads share one key/value head. Llama 3 70B has 64 query heads but only 8 KV heads, so the cache is 8× smaller than full multi-head attention would be. Quality barely notices, because the heads were storing a lot of redundant stuff anyway.

MLA (DeepSeek-V2/V3) goes further: instead of storing keys and values at all, store one compressed latent vector per token — 512 numbers — and expand it back into per-head keys and values when you need them. Better: the up-projection can be folded into the query projection, so attention runs directly in the latent space. One snag: RoPE rotates by position, and a position-dependent rotation can’t be folded into a fixed matrix. So MLA carries a separate tiny 64-dim rotary key alongside the latent. 576 numbers per token per layer, ~69 KiB per token for DeepSeek-V3. Compare to 320 KiB above.

7. Most layers only need to look nearby

GQA and MLA shrink each token’s footprint, but every layer still keeps every token. Here’s the observation that breaks that: most of what a layer needs is close by. Syntax, the current sentence, the last few lines of code.

So give most layers a sliding window: they only attend to the last $W$ tokens, and their cache stops growing at $W$. Stack them and information still travels — layer 1 sees $W$ back, layer 2 sees $2W$ back through layer 1, and so on. Mistral 7B used a 4K window everywhere.

But “information can travel $kW$ tokens” is not the same as “the model can exactly recall a phone number from 500K tokens ago”. Squeezing it through a chain of windows blurs it. So the modern recipe interleaves: Gemma 2 alternates local and global layers 1:1; Gemma 3 goes 5 local (window 1024) to 1 global, which makes the cache roughly a sixth of what it would be; gpt-oss alternates dense and 128-token banded layers. The global layers do the long-distance recall, the local ones do everything else for almost nothing.

The sink toggle in the panel is a side story worth knowing. StreamingLLM found that models dump a lot of attention onto the first few tokens — not because they matter, but because softmax has to put its weight somewhere. Evict those and a sliding-window model falls apart. Keep them and it’s fine. gpt-oss bakes a learned sink into each head for exactly this reason.

8. Read only the blocks that matter

Sliding windows decide where to look by a fixed rule: nearby. What if the model decided for itself?

That’s learned sparse attention, and the shape of it is always the same: a cheap scout and an expensive reader. The scout looks at everything, but with a tiny budget. The reader does full attention, but only where the scout points.

NSA (DeepSeek, 2025) runs three branches per query. A compressed branch squashes each block of keys into one summary token and attends to those — a blurry overview of the whole context. Those coarse scores pick the top few blocks, and a selected branch reads those blocks at full resolution. A sliding window branch always covers the recent tokens. Learned gates mix the three.

DSA in DeepSeek-V3.2 is leaner. A “lightning indexer” — a handful of tiny heads in FP8 — scores the query against every previous token, and the main attention reads only the top 2,048. The indexer is still technically $O(L)$ per query, but it’s so small that the expensive part, the real attention, is now a constant per query no matter how long the context is. Moonshot’s MoBA is in the same family: gate over block summaries, read the top-$k$ blocks.

Notice in the panel that the scout can miss. With a small budget, a relevant block sometimes doesn’t make the cut. That’s the real trade-off of sparse attention, and why these methods are trained end-to-end rather than bolted on.

9. A fixed-size notebook for most layers

The last architectural move is the most aggressive: for most layers, don’t keep a cache at all. Keep a fixed-size state instead — the $d\times d$ board from the linear attention and recurrent memory posts. Each token writes into it, the board gets edited and decayed, and it never grows. A linear layer costs the same per token at 1M as at 1K.

The catch, which those posts spent a lot of time on: a fixed board has finite capacity. Push enough facts through it and old ones get blurred or overwritten. Exact recall of one specific line from 800K tokens ago is precisely what a fixed state is bad at.

So nobody ships pure linear. They ship hybrids: Jamba puts one attention layer per eight; MiniMax-01 puts a softmax layer after every seven lightning-attention layers, trains to 1M and runs inference to 4M; Qwen3-Next and Kimi Linear go 3:1. Memory grows with the slope of the attention fraction — a 7:1 hybrid’s cache grows at an eighth of the rate. The linear layers do the cheap bulk work. The few attention layers keep the archive.

10. The recipe: train short, extend at the end

Now put the training run together. Attention cost per token grows with $L$, so you want to spend as few tokens as possible at long length. The standard recipe: do almost all of pretraining short, then stretch the window in a brief final phase.

Llama 3 405B pretrained at 8K, then grew to 128K in six stages over ~800B tokens — about 5% of its 15.6T. It only moved to the next stage once short-context scores had recovered and needle tests passed at the new length. DeepSeek-V3 went 4K → 32K → 128K with YaRN, two phases of 1,000 steps each. Qwen2.5-1M went 4K → 32K → 65K → 131K → 262K while raising the RoPE base to 10M, then used DCA at inference for the last 4× to 1M.

Two things the recipe quietly depends on. Data: most web documents are short, so long documents (books, whole code repos, long transcripts) get upsampled, and labs synthesize tasks where the answer depends on something far away. The surprising finding (Fu et al., 2024) is that a few billion tokens of well-mixed long data is enough — the skill is mostly latent and just needs to be unlocked. Short skills: if you only feed long data, short-context ability decays, so you keep mixing in high-quality short data.

Hit play and watch the cost meter. The long phase is a sliver of the tokens and a big chunk of the attention bill.

11. Accepting 1M tokens is not the same as using them

Here’s the uncomfortable part. Everything so far makes a model accept a long input without breaking. None of it guarantees the model uses it.

The classic test is needle in a haystack: hide one sentence somewhere in a long document, ask for it back. Almost every model now aces it, and that’s the problem — finding a string that literally matches your question is the easiest thing attention does. RULER (NVIDIA, 2024) added multi-needle, multi-hop variable tracking, and aggregation tasks, and found that many models fell apart well before their advertised window. Lost in the Middle (Liu et al., 2023) showed accuracy is U-shaped over position: great at the start, great at the end, sagging in the middle. NoLiMa (2025) removed the word overlap between the question and the needle, and scores dropped hard with length.

So there are two numbers for every model, the advertised one and the effective one, and the gap between them is where the actual research is right now. (The heatmaps below are an illustrative pattern, not real scores — the shape is what matters.)

12. Putting it together: how real models got to a million

Now the punchline. No single trick gets you to 1M. Each one removes one constraint, and the next one becomes binding. Look at how the actual million-token models stacked them:

• Qwen2.5-1M: progressive training to 256K with a huge RoPE base (positions), GQA (memory), sparse attention to speed up prefill (compute), and DCA plus YaRN’s attention scaling at inference for the last 4× (positions and dilution — without retraining).
• MiniMax-01: seven lightning-attention layers per softmax layer (memory and compute), trained to 1M, inference to 4M.
• Llama 4 Scout: iRoPE — NoPE global layers plus chunked local RoPE layers — and inference-time temperature scaling (positions + dilution). Trained at 256K, advertises 10M.
• DeepSeek-V3.2: MLA (memory) plus DSA sparse attention (compute). Stays at 128K, but makes each token there much cheaper.
• Gemini 1.5 shipped 1M (and 10M in research) in early 2024 and was the first to make it routine. The architecture isn’t public, so I won’t guess.

Build your own below. Slide the length, flip the components, and watch which gauge goes red first. It’s almost always the one you haven’t fixed yet.

So what actually happened between 2K and 1M

Squint at the whole thing and it’s five fixes for five walls.

Positions stopped being a wall once people saw RoPE as dials and asked which ones were actually out of range — interpolate the slow ones, leave the fast ones, sharpen the softmax, or remap distances so the model never sees a new one. Memory stopped being a wall in training when FlashAttention refused to write down the $L\times L$ matrix and ring attention refused to put the sequence on one GPU, and at inference when GQA, MLA, local layers and hybrids each cut what a token leaves behind. Compute is still quadratic for any layer that does full attention; the escape is to have fewer such layers and make them read less. Data turned out to be cheaper than expected — a few billion well-mixed long tokens at the end of the run. And evaluation is the one that’s still open: we got very good at making models accept a million tokens, and we’re still working on making them use all of it.

---

A context window is the set of distances a model practiced on, times the budget to pay for them. Everything else is engineering.

References

• Packing & recipe — Llama Team, The Llama 3 Herd of Models (arXiv:2407.21783) · DeepSeek-AI, DeepSeek-V3 Technical Report (arXiv:2412.19437) · Qwen Team, Qwen2.5-1M Technical Report (arXiv:2501.15383) · Qwen3 Technical Report (arXiv:2505.09388)
• RoPE — Su et al., RoFormer (arXiv:2104.09864)
• Position Interpolation — Chen et al. (arXiv:2306.15595) · ABF — Xiong et al., Effective Long-Context Scaling of Foundation Models (arXiv:2309.16039) · YaRN — Peng et al. (arXiv:2309.00071) · LongRoPE — Ding et al. (arXiv:2402.13753)
• Dual Chunk Attention — An et al., Training-Free Long-Context Scaling of Large Language Models (arXiv:2402.17463)
• FlashAttention — Dao et al. (arXiv:2205.14135) · Ring Attention — Liu, Zaharia, Abbeel (arXiv:2310.01889) · Striped Attention — Brandon et al. (arXiv:2311.09431) · DeepSpeed-Ulysses — Jacobs et al. (arXiv:2309.14509)
• MQA — Shazeer (arXiv:1911.02150) · GQA — Ainslie et al. (arXiv:2305.13245) · MLA — DeepSeek-AI, DeepSeek-V2 (arXiv:2405.04434)
• Sliding window & sinks — Jiang et al., Mistral 7B (arXiv:2310.06825) · Gemma Team, Gemma 3 (arXiv:2503.19786) · Xiao et al., Efficient Streaming Language Models with Attention Sinks (arXiv:2309.17453)
• Sparse attention — Yuan et al., Native Sparse Attention (arXiv:2502.11089) · Lu et al., MoBA (arXiv:2502.13189) · DeepSeek-AI, DeepSeek-V3.2-Exp (DSA, 2025)
• Hybrids — Lieber et al., Jamba (arXiv:2403.19887) · MiniMax, MiniMax-01 (arXiv:2501.08313) · Moonshot AI, Kimi Linear (arXiv:2510.26692)
• Data & evaluation — Fu et al., Data Engineering for Scaling Language Models to 128K Context (arXiv:2402.10171) · Gao et al., How to Train Long-Context Language Models (Effectively) (arXiv:2410.02660) · Hsieh et al., RULER (arXiv:2404.06654) · Liu et al., Lost in the Middle (arXiv:2307.03172) · Modarressi et al., NoLiMa (arXiv:2502.05167)

Every panel above is live, interactive and bilingual — drag the knobs, switch the language, and the demos follow.