M4.6 - FlashAttention-3/4 Hardware Pipelines: Hopper, Blackwell, and the Moving Bottleneck
A quantitative reconstruction of FlashAttention-3 and FlashAttention-4: Hopper TMA/WGMMA pipelines, Blackwell TMEM and 2-CTA MMA, softmax bottlenecks, FP8 accuracy, backward scheduling, and benchmark evidence.