A quantitative reconstruction of FlashAttention-3 and FlashAttention-4: Hopper TMA/WGMMA pipelines, Blackwell TMEM and 2-CTA MMA, softmax bottlenecks, FP8 accuracy, backward scheduling, and benchmark evidence.
A derivation- and implementation-oriented reconstruction of vocabulary projection, logits and loss gradients, weight tying, score processors, temperature, top-k/top-p, beam search, and local token-selection policy.
A code- and shape-oriented study note on scaled dot-product attention: Q/K/V, score tensors, finite bias, visibility masks, softmax over keys, dropout, and weighted value mixing.
A tensor- and code-oriented reconstruction of FlashAttention-1: online softmax, exact tiled attention, HBM traffic, backward recomputation, masks, and backend evidence.