A quantitative bridge from Transformer tensor algebra to GPU execution: kernels, grids, blocks, warps, SM residency, registers, shared memory, coalescing, tiling, fusion, roofline limits, and profiler evidence.
A quantitative reconstruction of FlashAttention-3 and FlashAttention-4: Hopper TMA/WGMMA pipelines, Blackwell TMEM and 2-CTA MMA, softmax bottlenecks, FP8 accuracy, backward scheduling, and benchmark evidence.
A kernel-oriented reconstruction of FlashAttention-2 work partitioning, parallelism across sequence rows, prefill/decode shape changes, and the boundary where decode requires a different schedule.
A derivation- and implementation-oriented study of one-token decode, Flash-Decoding, exact partial-softmax reduction, FlashMask, GQA-aware cache access, SDPA dispatch, and the FA2-to-FA3 kernel boundary.