Preface - Transformer System
A technical study of foundation models, from learning objectives and Transformer architecture to training, execution, and deployment, followed by autonomous-agent infrastructure around a fixed trained model.
Tag: #transformer
M3.11 — Representation Readout and Task-Head Inference
A derivation- and implementation-oriented study of how contextual Transformer rows become pooled vectors, sequence and token predictions, extractive spans, and retrieval scores without an autoregressive generation loop.
M4.8 - Extrapolation, Alternative Scaling Laws, and Data Constraints
A source-audited study of extrapolation: scale-ordered validation, alternative scaling-law families, predictive model selection, repeated data, dependence, and effective-data constraints.
M4.9 - Over-Training, Downstream Metrics, and Lifecycle Allocation
A source-audited study of over-training, downstream metric behavior, lifecycle cost, measured serving demand, uncertainty-aware artifact selection, pilot gates, and a training authorization that hands release and rollback requirements to the serving and deployment modules.
M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation
A source-audited study of neural scaling experiments: measurement contracts, power laws, joint model-data loss surfaces, compute-optimal allocation, paper-contract comparison, fitting, and uncertainty.
M6.10 - Teacher-Student Distillation and Model Transfer
A mechanism-level reconstruction of knowledge distillation: student architecture, teacher supervision, matched targets, prefix distributions, teacher access, and online or cached training execution.
M6.11 - Pruning, Structural Compression, and Executable Sparsity
A mechanism-level reconstruction of Transformer pruning: removal units, masks, dependency closure, sparse and smaller-dense artifacts, recovery, and the runtime proof required before zeros become acceleration.
M5.5 - KV Retrieval, Offload, and Hierarchical Memory
A derivation- and implementation-oriented study of exact KV offload, compute-near-data attention, hierarchical residency, query-dependent retrieval, recallable memory, consistency, fallback, and serving evaluation.
M4.2 - GPU Execution Model for Transformer Kernels: Launch Geometry, Memory Hierarchy, Tiling, and Profiling
A quantitative bridge from Transformer tensor algebra to GPU execution: kernels, grids, blocks, warps, SM residency, registers, shared memory, coalescing, tiling, fusion, roofline limits, and profiler evidence.
M4.6 - FlashAttention-3/4 Hardware Pipelines: Hopper, Blackwell, and the Moving Bottleneck
A quantitative reconstruction of FlashAttention-3 and FlashAttention-4: Hopper TMA/WGMMA pipelines, Blackwell TMEM and 2-CTA MMA, softmax bottlenecks, FP8 accuracy, backward scheduling, and benchmark evidence.
M5.7 - MQA and GQA: KV-Head Sharing, Cache Bytes, and Grouped Decode Kernels
A quantitative reconstruction of multi-query and grouped-query attention: KV-head ownership, checkpoint conversion, cache bytes, logical broadcast, grouped decode reuse, tensor parallelism, and failure diagnosis.
M5.8 - Multi-Head Latent Attention: Latent KV State, Weight Absorption, and FlashMLA
A tensor-, algebra-, and kernel-level reconstruction of Multi-Head Latent Attention: latent KV cache state, decoupled RoPE, weight absorption, DeepSeek reference paths, FlashMLA scheduling, FP8 cache layout, sparse variants, and parity diagnosis.
M6.6 - MoE Runtime Systems: Placement, Dispatch, Kernels, and Topology
A tensor-to-cluster study of MoE execution: logical and physical experts, variable-size dispatch, all-to-all communication, grouped and block-sparse kernels, prefill/decode modes, expert replication, topology, overlap, precision, observability, and failure recovery.
M6.7 - Low-Rank Theory: Intrinsic Dimension, Subspaces, and Dynamics
A rigorous study of why low-rank adaptation can work and where it fails: intrinsic-dimension measurement, matrix manifolds, SVD reconstruction, factor redundancy, initialization gradients, NTK preservation, expressive-power bounds, feature learning, and forgetting.
M6.8 - Conditional Compute Beyond FFN Experts: Attention, Depth, and Modalities
A tensor-level study of routing outside FFN experts: Mixture of Attention Heads, SwitchHead, Mixture-of-Depths, causal route prediction, per-layer KV consequences, MoD+MoE composition, and staged multimodal sparse training.
M6.9 - Routed, Shared, and Continual Adapters
A mechanism-level study of mixtures of LoRA adapters: routing granularity, joint versus post-hoc training, HydraLoRA factor sharing, router collapse, static and dynamic composition, continual low-rank state, and multi-adapter runtime integration.
M7.7 - Draft Sources, Dependent Heads, and Hardware-Aware Trees
A derivation-, tensor-, training-, and runtime-level comparison of draft models, retrieval and prompt lookup, self-speculation, dependent draft heads, multi-candidate exact sampling, token-tree construction, and hardware-aware tree optimization.
M7.8 - Speculative Serving: Scheduling, Evaluation, Release, and Rollback
A derivation-, state-, scheduler-, and operations-level treatment of continuous-batch speculative decoding, adaptive draft budgets, goodput control, target and draft cache synchronization, evaluation, canary release, and rollback.
M8.10 - Deployment Evaluation, Release, and Rollback
A statistical, systems, and release-engineering treatment of quantized deployment decisions: immutable candidate tuples, paired non-inferiority tests, workload-shaped latency and goodput curves, memory and energy ledgers, Pareto selection, shadow and canary release, production observability, and cache-safe rollback.
M8.3 - Outlier Geometry and Equivalent Transforms
A derivation- and implementation-oriented treatment of LLM quantization outliers: geometric diagnostics, mixed-precision decomposition, SmoothQuant scaling, affine shifts, fixed and learned rotations, structured learned transforms, graph fusion, calibration, and runtime cost.
M8.4 - Scalar Weight-Only PTQ: RTN, GPTQ, and AWQ
A derivation- and implementation-oriented treatment of scalar weight-only post-training quantization: round-to-nearest, layer reconstruction geometry, OBD/OBS/OBQ, GPTQ compensation and lazy block updates, AWQ activation-aware scaling, calibration state, packed artifacts, and runtime kernels.
M8.5 - Extreme Weight Compression and Representations
A derivation- and implementation-oriented treatment of extreme weight-only compression: sensitivity-weighted nonuniform codebooks, dense-and-sparse exceptions, SpQR's hierarchical representation, AQLM additive codebooks, QuIP# incoherence and E8 lattice codes, VPTQ second-order vector quantization, artifact bit ledgers, and runtime decoder contracts.
M8.6 - Live-Data Quantization: Activations and KV Cache
A derivation- and implementation-oriented treatment of request-dependent quantization: dynamic activation scales, integer GEMM epilogues, attention scale placement, persistent KV-cache rate and metadata, streaming append state, KIVI, KVQuant, PrefixQuant, QServe W4A8KV4, production FP8 cache formats, and serving correctness.
M8.7 - Quantization-Aware Training and Low-Bit Optimization
A derivation- and implementation-oriented treatment of quantization-aware training: fake-quant arithmetic, straight-through gradients, learned step sizes and clipping, observer schedules, LLM-QAT data-free distillation, weight/activation/KV insertion sites, efficient QAT variants, distributed training state, and export parity.
M8.8 - Native Binary and Ternary Transformers
A derivation- and implementation-oriented treatment of native low-bit Transformer design: BitLinear, binary and ternary weight codes, latent-weight optimization, variance control, BitNet b1.58, OneBit SVID and distillation, activation-lowering variants, distributed scale ownership, and the boundary between a code alphabet and an executable runtime.
M8.9 - Quantized Artifacts and Runtime Integration
A byte-, tensor-, graph-, and kernel-level treatment of quantized deployment artifacts: safetensors, GGUF, ONNX QDQ/QOperator, packing and alignment, sharding, converter and loader state machines, topology-aware placement, kernel admission, parity ladders, startup memory, security, and runtime integration across vLLM, TorchAO, TensorRT-LLM, and llama.cpp.
M3.6 — Dense and Gated MLPs: Channel Mixing, Training, and Runtime
A derivation- and implementation-oriented study of Transformer MLPs: position-wise channel mixing, activation and GLU variants, SwiGLU width matching, backward flow, fused execution, knowledge interpretations, and memory alternatives.
M3.7 — Residual Stream, Normalization, and Block Assembly
A derivation- and implementation-oriented reconstruction of residual learning, LayerNorm and RMSNorm geometry, backward flow, Pre-LN/Post-LN ordering, depth scaling, dropout, and Transformer block variants.
M3.8 — Output Head and Decoding Policy
A derivation- and implementation-oriented reconstruction of vocabulary projection, logits and loss gradients, weight tying, score processors, temperature, top-k/top-p, beam search, and local token-selection policy.
M6.3 - LoRA Foundations: Low-Rank Updates, Gradients, and Target Modules
A derivation- and implementation-oriented study of LoRA: low-rank update geometry, tensor shapes, forward and backward passes, initialization, scaling, optimizer state, target-module selection, and the limits of the low-rank assumption.
M6.4 - The Low-Rank Adaptation Design Space: QLoRA, AdaLoRA, rsLoRA, LoRA+, LoRA-FA, PiSSA, LoftQ, and DoRA
A mechanism-level comparison of major LoRA variants: quantized-base training, adaptive rank allocation, rank-stable scaling, factor-specific optimization, frozen-factor activation savings, SVD initialization, quantization-aware initialization, and magnitude/direction decomposition.
M6.5 - Multi-Adapter Serving: Kernels, Residency, Batching, and KV Compatibility
A systems study of serving many LoRA adapters over one base model: heterogeneous batching, segmented kernels, paging, residency, merge and hotswap semantics, request/KV compatibility, scheduling, observability, and failure recovery.
M7.5 - EAGLE Feature Drafting: Uncertainty, Dynamic Trees, HASS, and EAGLE-3
A tensor- and runtime-level study of EAGLE feature autoregression, shifted-token conditioning, draft training, dynamic trees, HASS context alignment, and the EAGLE-3 design transition.
M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime
A formula-, tensor-, code-, and runtime-level study of DeepSeek-V3 multi-token prediction modules, shared embeddings and heads, teacher forcing, released weight structure, KV ownership, and speculative serving.
M8.1 - Quantized Numbers: Scales, Zero Points, Error, and the Kernel Contract
A derivation- and implementation-oriented foundation for model quantization: affine maps, error, scale shapes, quantized matrix multiplication, accumulator bounds, packed storage, kernel execution, and effective memory cost.
M8.2 - Calibration, Sensitivity, and Precision Maps
A derivation- and implementation-oriented treatment of quantization calibration: observer state, clipping objectives, representative corpora, output-aware error, sensitivity scans, Hessian alternatives, mixed-precision optimization, and runtime-feasible precision maps.
M3.2 — Multi-Head and Cross-Attention: Projection, Layout, and Variants
A code-, shape-, and systems-oriented study of packed multi-head attention, target-by-source cross-attention, head composition, conditional head routing, concatenation, and output projection.
M3.1 — Scaled Dot-Product Attention and Masks: The Score Path Inside One Attention Call
A code- and shape-oriented study note on scaled dot-product attention: Q/K/V, score tensors, finite bias, visibility masks, softmax over keys, dropout, and weighted value mixing.
M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch
A cost- and shape-oriented study note on Mixture-of-Experts layers, dense FFN baselines, top-k routing, token dispatch, expert capacity, overflow, and total-vs-active parameter ledgers.
M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing
A systems-oriented study note on scaled Mixture-of-Experts routing, load balancing, expert parallelism, all-to-all communication, shared experts, fine-grained experts, device-limited routing, and DeepSeek-style MoE.
M7.1 - Speculative Decoding: Draft, Verify, Accept, Reject
A distribution- and serving-state-oriented study note on speculative decoding, draft models, target verification, exact speculative sampling, correction distributions, acceptance rate, speedup ledgers, and KV rollback.
M7.2 - Medusa: Multi-Head Candidates, Tree Attention, and Verification
A mechanism-oriented study note on Medusa decoding heads, same-model candidate generation, candidate trees, tree attention masks, typical acceptance, Medusa-1, Medusa-2, and branch KV state.
M7.3 - Lookahead Decoding: Jacobi Iterations, N-Gram Pools, and Parallel Verification
A mechanism-oriented study note on Lookahead decoding, Jacobi iterations, 2D windows, n-gram pools, lookahead and verification branches, exactness, masks, and KV state.
M7.4 - Multi-Token Prediction Foundations: Objectives, Gradients, and Prediction Heads
A derivation- and implementation-oriented study of multi-token prediction objectives, shifted labels, shared-trunk gradients, prediction-head designs, training memory, and the boundary between denser supervision and faster inference.
M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure
A serving-oriented study note on KV cache lifecycle, prefill, decode, TTFT, TPOT, memory admission, and why autoregressive inference turns attention state into a scheduler problem.
M5.2 - Continuous Batching and Scheduler Pressure
A serving-scheduler study note on static batching failure, iteration-level scheduling, continuous batching, chunked prefill, TTFT, TPOT, goodput, and KV-aware admission control.
M5.3 - KV Memory Layout and Exact Reuse: PagedAttention, vAttention, and Prefix Caches
A derivation- and implementation-oriented study of physical KV layout, PagedAttention, vAttention, exact prefix identity, RadixAttention, ownership, and execution reuse.
M5.4 - Long-Context KV Policies: Retention, Compression, and Budget Allocation
A systems-oriented study of long-context KV pressure, sliding windows, attention sinks, H2O, SnapKV, PyramidKV, PyramidInfer, layer/head budgets, and the accuracy contracts behind lossy cache policies.
M5.6 - Prefill/Decode Disaggregation and KV Placement
A systems-oriented study note on prefill/decode interference, chunked prefill, disaggregated serving, KV transfer, placement, routing, goodput, and exact cache handoff contracts.
M4.1 - Transformer Cost Model: Parameters, Activations, FLOPs, and Memory Traffic
A code- and shape-oriented study note on Transformer cost: parameters, activations, FLOPs, memory traffic, prefill, decode, and why FLOPs alone do not predict runtime.
M4.3 - IO-Aware Attention and FlashAttention-1: Online Softmax, Tiling, and Exactness
A tensor- and code-oriented reconstruction of FlashAttention-1: online softmax, exact tiled attention, HBM traffic, backward recomputation, masks, and backend evidence.
M4.4 - FlashAttention-2 Work Partitioning and the Decode Boundary
A kernel-oriented reconstruction of FlashAttention-2 work partitioning, parallelism across sequence rows, prefill/decode shape changes, and the boundary where decode requires a different schedule.
M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries
A derivation- and implementation-oriented study of one-token decode, Flash-Decoding, exact partial-softmax reduction, FlashMask, GQA-aware cache access, SDPA dispatch, and the FA2-to-FA3 kernel boundary.
M3.5 — Context Extension
A derivation-, training-, runtime-, and evaluation-oriented reconstruction of context extension: positional extrapolation, interpolation, recurrence, learned segment summaries, frequency scaling, adaptation, and effective-context tests.
M3.4 — Rotary Position Embedding
A complete derivation and implementation trace for rotary position embedding: relative-offset identity, pair layouts, frequency bases, partial rotary, numerical precision, cache integration, and paper/runtime variants.
M3.3 — Positional Geometry and Attention Bias
A derivation- and implementation-oriented reconstruction of absolute coordinates, sinusoidal bases, learned positions, relative representations, attention bias, cache offsets, and positional design choices.
M2.2 — Embeddings and Initial Model State
A derivation- and implementation-oriented reconstruction of input embedding lookup, initialization, gradient flow, factorization, weight tying, and the handoff from initial rows to contextual computation.
M2.1 — Tokenization and Sequence Contracts
How text becomes model input: learning and applying tokenizers, preserving IDs and offsets, formatting conversations, batching sequences, and checking compatibility.
M3.10 — Single-Request Inference Execution
A derivation- and implementation-oriented study of one autoregressive request: prompt state, prefill/decode, last-row logits, token selection, KV append timing, positions, stopping, beam reorder, streaming, and cache equivalence.
M3.9 — Training Execution: Teacher Forcing, Masks, Loss, and Parallelism
A derivation- and code-path reconstruction of Transformer training: shifted supervision, causal/padding/packed/loss masks, cross-entropy and smoothing, accumulation, Adam, Noam scheduling, mixed precision, distributed parallelism, checkpoints, and reproducibility.
M0.0 - Roadmap
A reading guide to Transformer System: what each module explains, the questions behind its articles, and where to find their main sources.
M1.1 — Why Transformers, Which Architecture, and What Executes
Sequence modeling from information dependencies to executable Transformer architectures: recurrence, convolution, attention, training, and inference.