Recipe: train short, extend at the end

Train short, extend at the end

training sequence length (log scale) vs tokens seen

per-token compute at the current length

training compute spent so far

this recipe

all at final L

cost model (illustrative, same for every preset): a 70B-shaped dense model, 80 layers, d_attn = 8192 · per token ≈ 2N + 2·layers·L·d_attn (causal), ×3 for fwd+bwd