Dual Chunk Attention
play
naive (plain RoPE)
Dual Chunk Attention
chunk size c
6
rows = queries i, columns = keys j (causal). Each cell: the relative distance the model sees. Trained on 8 positions, so distances 0–7 only.