A code-, shape-, and systems-oriented study of packed multi-head attention, target-by-source cross-attention, head composition, conditional head routing, concatenation, and output projection.
A code- and shape-oriented study note on scaled dot-product attention: Q/K/V, score tensors, finite bias, visibility masks, softmax over keys, dropout, and weighted value mixing.
A complete derivation and implementation trace for rotary position embedding: relative-offset identity, pair layouts, frequency bases, partial rotary, numerical precision, cache integration, and paper/runtime variants.