#flashattention-3 • Tags • Transformer System

2 posts · Transformer System

Tag: #flashattention-3

M4.6 - FlashAttention-3/4 Hardware Pipelines: Hopper, Blackwell, and the Moving Bottleneck

🗓 2026-07-15 • Transformer System • ⏱ 132 min read

A quantitative reconstruction of FlashAttention-3 and FlashAttention-4: Hopper TMA/WGMMA pipelines, Blackwell TMEM and 2-CTA MMA, softmax bottlenecks, FP8 accuracy, backward scheduling, and benchmark evidence.

#numinens #transformer #flashattention-3 #flashattention-4 #hopper #blackwell

Read →

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries

🗓 2026-07-07 • Transformer System • ⏱ 102 min read

A derivation- and implementation-oriented study of one-token decode, Flash-Decoding, exact partial-softmax reduction, FlashMask, GQA-aware cache access, SDPA dispatch, and the FA2-to-FA3 kernel boundary.

#numinens #transformer #flash-decoding #flashmask #sdpa #kernel

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #flashattention-3

Tag: #flashattention-3

M4.6 - FlashAttention-3/4 Hardware Pipelines: Hopper, Blackwell, and the Moving Bottleneck

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries