Archive
Jump via the sidebar, or scroll.
2026-09 (2)
2026-08 (1)
2026-07 (54)
M4.8 - Extrapolation, Alternative Scaling Laws, and Data Constraints M4.9 - Over-Training, Downstream Metrics, and Lifecycle Allocation M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation M6.10 - Teacher-Student Distillation and Model Transfer M6.11 - Pruning, Structural Compression, and Executable Sparsity X1.1 - Unitization and Geometry Metadata Across Modalities X1.2 - Multimodal Encoders, Projectors, Resamplers, and Fusion X1.3 - Multimodal Masks, Training, Runtime, and State M5.5 - KV Retrieval, Offload, and Hierarchical Memory M4.2 - GPU Execution Model for Transformer Kernels: Launch Geometry, Memory Hierarchy, Tiling, and Profiling M4.6 - FlashAttention-3/4 Hardware Pipelines: Hopper, Blackwell, and the Moving Bottleneck M5.7 - MQA and GQA: KV-Head Sharing, Cache Bytes, and Grouped Decode Kernels M5.8 - Multi-Head Latent Attention: Latent KV State, Weight Absorption, and FlashMLA M6.6 - MoE Runtime Systems: Placement, Dispatch, Kernels, and Topology M6.7 - Low-Rank Theory: Intrinsic Dimension, Subspaces, and Dynamics M6.8 - Conditional Compute Beyond FFN Experts: Attention, Depth, and Modalities M6.9 - Routed, Shared, and Continual Adapters M7.7 - Draft Sources, Dependent Heads, and Hardware-Aware Trees M7.8 - Speculative Serving: Scheduling, Evaluation, Release, and Rollback M8.10 - Deployment Evaluation, Release, and Rollback M8.3 - Outlier Geometry and Equivalent Transforms M8.4 - Scalar Weight-Only PTQ: RTN, GPTQ, and AWQ M8.5 - Extreme Weight Compression and Representations M8.6 - Live-Data Quantization: Activations and KV Cache M8.7 - Quantization-Aware Training and Low-Bit Optimization M8.8 - Native Binary and Ternary Transformers M8.9 - Quantized Artifacts and Runtime Integration M3.6 — Dense and Gated MLPs: Channel Mixing, Training, and Runtime M3.7 — Residual Stream, Normalization, and Block Assembly M3.8 — Output Head and Decoding Policy M6.3 - LoRA Foundations: Low-Rank Updates, Gradients, and Target Modules M6.4 - The Low-Rank Adaptation Design Space: QLoRA, AdaLoRA, rsLoRA, LoRA+, LoRA-FA, PiSSA, LoftQ, and DoRA M6.5 - Multi-Adapter Serving: Kernels, Residency, Batching, and KV Compatibility M7.5 - EAGLE Feature Drafting: Uncertainty, Dynamic Trees, HASS, and EAGLE-3 M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime M8.1 - Quantized Numbers: Scales, Zero Points, Error, and the Kernel Contract M8.2 - Calibration, Sensitivity, and Precision Maps M3.2 — Multi-Head and Cross-Attention: Projection, Layout, and Variants M3.1 — Scaled Dot-Product Attention and Masks: The Score Path Inside One Attention Call M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing M7.1 - Speculative Decoding: Draft, Verify, Accept, Reject M7.2 - Medusa: Multi-Head Candidates, Tree Attention, and Verification M7.3 - Lookahead Decoding: Jacobi Iterations, N-Gram Pools, and Parallel Verification M7.4 - Multi-Token Prediction Foundations: Objectives, Gradients, and Prediction Heads M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure M5.2 - Continuous Batching and Scheduler Pressure M5.3 - KV Memory Layout and Exact Reuse: PagedAttention, vAttention, and Prefix Caches M5.4 - Long-Context KV Policies: Retention, Compression, and Budget Allocation M5.6 - Prefill/Decode Disaggregation and KV Placement M4.1 - Transformer Cost Model: Parameters, Activations, FLOPs, and Memory Traffic M4.3 - IO-Aware Attention and FlashAttention-1: Online Softmax, Tiling, and Exactness M4.4 - FlashAttention-2 Work Partitioning and the Decode Boundary M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries
2026-06 (8)
M3.5 — Context Extension M3.4 — Rotary Position Embedding M3.3 — Positional Geometry and Attention Bias M2.2 — Embeddings and Initial Model State M2.1 — Tokenization and Sequence Contracts M3.10 — Single-Request Inference Execution M3.9 — Training Execution: Teacher Forcing, Masks, Loss, and Parallelism M0.0 - Roadmap