#sdpa • Tags • Transformer System

2 posts · Transformer System

Tag: #sdpa

M4.4 - FlashAttention-2 Work Partitioning and the Decode Boundary

🗓 2026-07-07 • Transformer System • ⏱ 105 min read

A kernel-oriented reconstruction of FlashAttention-2 work partitioning, parallelism across sequence rows, prefill/decode shape changes, and the boundary where decode requires a different schedule.

#numinens #transformer #flashattention-2 #flash-decoding #sdpa #kernel

Read →

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries

🗓 2026-07-07 • Transformer System • ⏱ 102 min read

A derivation- and implementation-oriented study of one-token decode, Flash-Decoding, exact partial-softmax reduction, FlashMask, GQA-aware cache access, SDPA dispatch, and the FA2-to-FA3 kernel boundary.

#numinens #transformer #flash-decoding #flashmask #sdpa #kernel

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #sdpa

Tag: #sdpa

M4.4 - FlashAttention-2 Work Partitioning and the Decode Boundary

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries