Shrinking what each token leaves behind
MHA
GQA
MQA
MLA
▶ play
one token · one layer · 8 query heads, head dim 128 (toy)
The bill · KV cache for one sequence
context
128K
▶ sweep