#gqa • Tags • Transformer System

2 posts · Transformer System

Tag: #gqa

M5.7 - MQA and GQA: KV-Head Sharing, Cache Bytes, and Grouped Decode Kernels

🗓 2026-07-15 • Transformer System • ⏱ 57 min read

A quantitative reconstruction of multi-query and grouped-query attention: KV-head ownership, checkpoint conversion, cache bytes, logical broadcast, grouped decode reuse, tensor parallelism, and failure diagnosis.

#numinens #transformer #kv-cache #mqa #gqa #attention

Read →

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries

🗓 2026-07-07 • Transformer System • ⏱ 102 min read

A derivation- and implementation-oriented study of one-token decode, Flash-Decoding, exact partial-softmax reduction, FlashMask, GQA-aware cache access, SDPA dispatch, and the FA2-to-FA3 kernel boundary.

#numinens #transformer #flash-decoding #flashmask #sdpa #kernel

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #gqa

Tag: #gqa

M5.7 - MQA and GQA: KV-Head Sharing, Cache Bytes, and Grouped Decode Kernels

M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries