A tensor-, algebra-, and kernel-level reconstruction of Multi-Head Latent Attention: latent KV cache state, decoupled RoPE, weight absorption, DeepSeek reference paths, FlashMLA scheduling, FP8 cache layout, sparse variants, and parity diagnosis.
A derivation- and implementation-oriented treatment of request-dependent quantization: dynamic activation scales, integer GEMM epilogues, attention scale placement, persistent KV-cache rate and metadata, streaming append state, KIVI, KVQuant, PrefixQuant, QServe W4A8KV4, production FP8 cache formats, and serving correctness.
A derivation-, training-, runtime-, and evaluation-oriented reconstruction of context extension: positional extrapolation, interpolation, recurrence, learned segment summaries, frequency scaling, adaptation, and effective-context tests.
A complete derivation and implementation trace for rotary position embedding: relative-offset identity, pair layouts, frequency bases, partial rotary, numerical precision, cache integration, and paper/runtime variants.