https://transformer-system.numinens.com/ https://transformer-system.numinens.com/archive https://transformer-system.numinens.com/categories/transformer-system https://transformer-system.numinens.com/posts/preface https://transformer-system.numinens.com/posts/m0-0-roadmap https://transformer-system.numinens.com/posts/m1-1-road-to-attention https://transformer-system.numinens.com/posts/m2-1-text-tokenization https://transformer-system.numinens.com/posts/m2-2-embeddings-and-projectors https://transformer-system.numinens.com/posts/m3-1-scaled-dot-product-attention-score-path https://transformer-system.numinens.com/posts/m3-2-multi-head-attention-head-layout https://transformer-system.numinens.com/posts/m3-3-position-encoding-and-context-geometry https://transformer-system.numinens.com/posts/m3-4-rotary-position-embedding-rope https://transformer-system.numinens.com/posts/m3-5-length-extrapolation-context-extension https://transformer-system.numinens.com/posts/m3-6-ffn-mlp-channel-mixing-gated-capacity https://transformer-system.numinens.com/posts/m3-7-residual-normalization-block-assembly https://transformer-system.numinens.com/posts/m3-8-output-head-logit-interface https://transformer-system.numinens.com/posts/m3-9-training-execution-teacher-forcing-masks-parallelism https://transformer-system.numinens.com/posts/m3-10-inference-execution https://transformer-system.numinens.com/posts/m3-11-representation-readout-task-heads https://transformer-system.numinens.com/posts/m4-1-transformer-cost-model https://transformer-system.numinens.com/posts/m4-2-gpu-execution-model-transformer-kernels https://transformer-system.numinens.com/posts/m4-3-io-aware-attention https://transformer-system.numinens.com/posts/m4-4-flashattention-v2-decoding-kernel-boundary https://transformer-system.numinens.com/posts/m4-5-decode-kernels-backend-boundaries https://transformer-system.numinens.com/posts/m4-6-flashattention-hardware-pipelines https://transformer-system.numinens.com/posts/m4-7-empirical-training-scaling-laws-compute-optimal-allocation https://transformer-system.numinens.com/posts/m4-8-extrapolation-functional-forms-data-constraints https://transformer-system.numinens.com/posts/m4-9-overtraining-downstream-metrics-lifecycle-allocation https://transformer-system.numinens.com/posts/m5-1-autoregressive-serving-state https://transformer-system.numinens.com/posts/m5-2-kv-memory-management https://transformer-system.numinens.com/posts/m5-3-kv-layout-exact-reuse https://transformer-system.numinens.com/posts/m5-4-long-context-kv-policies https://transformer-system.numinens.com/posts/m5-5-kv-retrieval-offload-hierarchical-memory https://transformer-system.numinens.com/posts/m5-6-prefill-decode-disaggregation-kv-placement https://transformer-system.numinens.com/posts/m5-7-mqa-gqa-kv-head-sharing https://transformer-system.numinens.com/posts/m5-8-mla-latent-cache-absorbed-kernels https://transformer-system.numinens.com/posts/m6-1-routed-capacity-moe-experts-dispatch https://transformer-system.numinens.com/posts/m6-2-deepseek-moe-routing-load-balancing https://transformer-system.numinens.com/posts/m6-3-lora-qlora-adapter-serving https://transformer-system.numinens.com/posts/m6-4-low-rank-adaptation-design-space https://transformer-system.numinens.com/posts/m6-5-multi-adapter-serving-runtime https://transformer-system.numinens.com/posts/m6-6-moe-runtime-placement-dispatch-kernels https://transformer-system.numinens.com/posts/m6-7-low-rank-theory-subspaces-dynamics https://transformer-system.numinens.com/posts/m6-8-conditional-compute-attention-depth-modalities https://transformer-system.numinens.com/posts/m6-9-routed-shared-continual-adapters https://transformer-system.numinens.com/posts/m6-10-teacher-student-distillation-model-transfer https://transformer-system.numinens.com/posts/m6-11-pruning-structural-compression-executable-sparsity https://transformer-system.numinens.com/posts/m7-1-speculative-decoding-draft-verify https://transformer-system.numinens.com/posts/m7-2-medusa-tree-verification https://transformer-system.numinens.com/posts/m7-3-lookahead-decoding-jacobi-verification https://transformer-system.numinens.com/posts/m7-4-multi-token-prediction-deepseek-mtp https://transformer-system.numinens.com/posts/m7-5-eagle-feature-drafting-dynamic-trees https://transformer-system.numinens.com/posts/m7-6-deepseek-mtp-module-chain https://transformer-system.numinens.com/posts/m7-7-draft-sources-dependent-heads-hardware-aware-trees https://transformer-system.numinens.com/posts/m7-8-speculative-serving-evaluation-release-rollback https://transformer-system.numinens.com/posts/m8-1-quantized-numbers-scales-zero-points-error https://transformer-system.numinens.com/posts/m8-2-calibration-sensitivity-precision-maps https://transformer-system.numinens.com/posts/m8-3-outlier-geometry-equivalent-transforms https://transformer-system.numinens.com/posts/m8-4-scalar-weight-only-ptq https://transformer-system.numinens.com/posts/m8-5-extreme-weight-compression-representations https://transformer-system.numinens.com/posts/m8-6-live-data-quantization-activations-kv-cache https://transformer-system.numinens.com/posts/m8-7-quantization-aware-training-low-bit-optimization https://transformer-system.numinens.com/posts/m8-8-native-binary-ternary-transformers https://transformer-system.numinens.com/posts/m8-9-quantized-artifacts-runtime-integration https://transformer-system.numinens.com/posts/m8-10-deployment-evaluation-release-rollback https://transformer-system.numinens.com/posts/x1-1-unitization-geometry-metadata https://transformer-system.numinens.com/posts/x1-2-encoders-projectors-fusion https://transformer-system.numinens.com/posts/x1-3-masks-training-runtime-state https://transformer-system.numinens.com/posts/afterword-authoring-research-loops