M1.1 — Why use a Transformer, and what runs?

Compare step-by-step memory, local windows and attention through translation examples. Then assemble the main Transformer designs and trace training and generation, so later articles can examine each part without losing the whole picture.

Main sources: A1; A2; A4; A5; Attention Is All You Need; The Annotated Transformer.

M2.1 — How does text become the IDs a model expects?

Follow the creation and use of text-splitting rules, then chat formatting and grouped inputs. Vocabulary changes and streaming show how different processing choices can turn the same visible text into different model inputs.

Main sources: A3; A6; Subword BPE (Sennrich et al.); SentencePiece; Released tokenizer implementations.

M2.2 — How do token IDs become the model’s starting vectors?

Build a lookup table, trace how its rows learn, then examine smaller tables, shared input/output weights and saved-model changes. The order connects a simple lookup to training and practical compatibility problems.

Main sources: A2; A7; PyTorch embedding documentation; ALBERT; Press and Wolf on weight tying.

M3.1 — How does attention choose what to read?

Trace one calculation from matching scores through allowed connections to weighted information. Worked examples connect the formula to code, training and common mistakes; M3.2 then combines several of these calculations.

Main sources: A10; A11; The Annotated Transformer; PyTorch attention documentation.

M3.2 — Why use several attention heads?

Build several attention reads and combine their results, then follow a decoder reading an encoded source sentence. Keeping this construction explicit makes the later comparisons—sharing heads, mixing them, or removing them—easier to understand.

Main sources: A4; A10; A12; The Annotated Transformer; Are Sixteen Heads Really Better than One?; Talking-Heads Attention.

M3.3 — How is word order represented?

Compare fixed and learned position vectors with ways of expressing relative distance inside attention. Small examples then show how position mistakes arise when inputs are combined or new text is generated.

Main sources: A8; A9; Attention Is All You Need; Shaw relative positions; T5; ALiBi.

M3.4 — How can rotation represent the distance between tokens?

Start with a two-number example of rotary position embedding (RoPE), then trace its code, precision and saved attention state. This establishes the basic mechanism before M3.5 asks what longer inputs require.

Main sources: A17; RoFormer; LLaMA implementations.

M3.5 — Why can’t we simply allow longer inputs?

Compare changing how positions are represented with carrying earlier computed vectors or learned summaries between text segments. Then examine the training, memory costs and tests needed to demonstrate useful access to distant evidence without harming shorter-input performance.

Main sources: A8; A9; A17; A23; Transformer-XL; AutoCompressors; YaRN; LongRoPE.

M3.6 — What happens inside each token’s vector?

Build the feed-forward network that transforms each token’s vector, then compare versions whose two branches multiply their results. Training and cost examples ground the later discussion of memory-like behavior and alternative designs.

Main sources: A13; GLU Variants; FFNs as Key-Value Memories; LLaMA implementations.

M3.7 — How do the parts form a trainable deep model?

Keep a running state and add each layer’s update, then compare ways to center or rescale its numbers. Complete block examples show why the order of these steps matters for learning and execution.

Main sources: A2; A4; A12; A13; A14; ResNet; LayerNorm; RMSNorm.

M3.8 — How is the next token chosen?

Turn the final vector into token scores, explain how training uses them, then compare highest-score selection, weighted random choices and searches across continuations. Separating learned scores from selection rules explains why one model can produce different answers.

Main sources: A2; A15; The Annotated Transformer; The Curious Case of Neural Text Degeneration; Best-First Beam Search.

M3.9 — Why can training process many positions together?

Follow a loop that uses known next-token targets without letting a position see future tokens, then calculates errors and updates weights. Extend it to combined examples, multiple GPUs, saved runs and different learning objectives.

Main sources: A3; A5; A11; A15; The Annotated Transformer; PyTorch distributed training documentation.

M3.10 — How does one prompt become generated text?

Follow repeated token choices, then add reuse of earlier computation, position tracking, stopping and streaming. The timing matters: choosing a token does not mean the model has processed it for the next step.

Main sources: A4; A5; A15; The Annotated Transformer; Hugging Face Transformers; Meta Llama and llama.cpp implementations.

M3.11 — What can a Transformer do besides generate text?

Select and combine token vectors that reflect surrounding text to predict labels, extract answers or support search. Then compare how training and processing queries and documents together or separately change the result.

Main sources: A2; A4; A5; BERT; Sentence-BERT; ColBERT; Sentence Transformers implementations.

After M4.1, follow M4.2–M4.6 for GPU execution, or M4.7–M4.9 for experiment-based decisions.

Revisit M4.9 using the measurement methods in M5 and M8.10 to replace assumed costs with measured ones.

M4.1 — What does a Transformer cost to run?

Count weights, working memory, calculations and data movement separately, then compare training, reading a prompt and generating one token. Worked examples show why model size alone cannot tell you whether it fits in memory or runs quickly.

Main sources: A16; Mixed Precision Training; ZeRO; Megatron; Roofline.

M4.2 — Why do GPU programs for the same calculation run at different speeds?

Follow an operation from software to groups of GPU workers and their memory. Small examples explain reusing data, waiting and wasted work, then show how measurements test whether a proposed improvement actually helps.

Main sources: A18; CUDA programming guides; Roofline; Nsight Compute.

M4.3 — Can attention avoid storing its full score table?

Derive a small running summary that combines score blocks correctly, then trace FlashAttention’s forward calculation and its training gradients. The proof comes first to show how storage changes without changing the attention formula.

Main sources: A18; Online Normalizer; Memory-efficient attention (Rabe and Staats); FlashAttention.

M4.4 — After attention saves memory, how can it keep the GPU busy?

FlashAttention-2 gives workers separate rows and avoids repeatedly normalizing their results. Follow the ordinary calculation and the backward steps used for learning, then see why one-token generation needs a different division of work.

Main sources: A19; FlashAttention-2; FlashAttention v2.0.0 source; CUDA and Nsight documentation.

M4.5 — How can many GPU workers help generate just one token?

Divide attention over saved history among workers and combine their results correctly. Then trace where saved attention data lives, which connections are allowed and which software actually runs: the right formula alone does not establish those details.

Main sources: A19; Flash-Decoding; FlashInfer; FlashMask; PyTorch attention documentation.

M4.6 — How does new GPU hardware change attention design?

Follow FlashAttention-3 and -4 as they overlap loading data, multiplying matrices and calculating attention weights. Examples for running and training the model, including calculations using fewer bits, expose new limits in memory, waiting and accuracy.

Main sources: A19; FlashAttention-3; FlashAttention-4; NVIDIA Hopper and Blackwell documentation.

M4.7 — Should a fixed training budget buy a larger model or more tokens?

Build the answer from comparable small runs, explain the fitted curves, then derive the trade-off. Comparisons of Kaplan, Chinchilla and later studies show why measurement choices and uncertainty matter more than memorizing one published ratio.

Main sources: Hestness learning curves; Kaplan scaling laws; Chinchilla; Porian scaling-law analysis.

M4.8 — Can small experiments predict much larger runs?

Test predictions on larger runs kept aside, compare curves that bend differently, then ask what changes when training reuses limited data. Concrete examples explain why more processed tokens need not mean more fresh information.

Main sources: Revisiting Neural Scaling Laws; Broken Neural Scaling Laws; Hernandez on repeated data; Scaling Data-Constrained Language Models.

M4.9 — When is training a smaller model longer worth it?

Compare extra training’s effects on prediction error and task scores, including jumps caused by scoring rules. Then weigh training and later-use costs under uncertain demand, and run a small trial before approving full training.

Main sources: Gadre on over-training; Beyond Chinchilla-Optimal; Wei on emergence; Schaeffer on metric sensitivity.

M5.1 — What must a server remember while each answer is still being generated?

Follow saved attention data from creation through growth, branching and release. Token-by-token examples explain when updates are safe to use, why finishing need not free memory immediately, and what later scheduling decisions need to know.

Main sources: A20; A24; Transformers cache documentation; vLLM and SGLang runtime sources; OpenTelemetry.

M5.2 — Which requests should the server work on next?

Start with requests arriving and finishing at different times, then process new prompts alongside ongoing answers. Small timelines compare breaking prompts into chunks, reserving memory, meeting deadlines, sharing capacity fairly, and pausing and resuming work when needed.

Main sources: A20; A24; A26; ORCA; SARATHI/Sarathi-Serve; FastServe; VTC.

M5.3 — How can requests share saved attention data and earlier work safely?

Trace token positions to storage, then follow shared memory blocks as requests diverge. Shared-prompt examples distinguish storing fewer copies, skipping repeated prompt processing and reducing repeated reads—three benefits with different requirements.

Main sources: A24; A25; PagedAttention; vAttention; RadixAttention; Hydragen; ChunkAttention.

M5.4 — Which parts of a long history should be kept?

Compare recent-history windows, importance-based selection and replacements for stored vectors. Small examples show what each preserves, which methods need training changes, and why lower memory use does not automatically mean good answers or faster service.

Main sources: A24; A25; StreamingLLM; H2O; SnapKV; PyramidKV; Dynamic Memory Compression.

M5.5 — How can attention use history that no longer fits on the GPU?

Compare reading the full allowed history—by staged transfers or computation near storage—with reading only selected parts. Timelines and interrupted requests show where transfer delays, missing information and recovery work enter.

Main sources: A25; FlexGen; InfiniGen; RetrievalAttention; InfLLM; LMCache.

M5.6 — Should prompt processing and answer generation run on different serving workers?

First see why prompt work can interrupt ongoing answers, then trace the saved attention data passed between workers. Examples compare separate pools, shared caches and predicted answer lengths, including when transfer delays outweigh the benefit.

Main sources: A20; A24; A26; DistServe; Splitwise; MemServe; Mooncake; TetriInfer.

M5.7 — Can attention heads share saved information?

Compare separate, fully shared and grouped key/value vectors, then examine how a model is converted, trained further and run across GPUs. Examples separate storage savings from model quality and speed, and compare head sharing with sharing between layers.

Main sources: A27; Multi-Query Attention; Grouped-Query Attention; FlashInfer; Cross-Layer Attention; YOCO.

M5.8 — Can attention use a compact learned history directly?

First separate content from position, then rearrange the calculations so full historical keys and values need not be rebuilt. Concrete storage layouts and GPU examples connect that idea to execution, while keeping model conversion, rounding error and selective attention separate.

Main sources: A28; DeepSeek-V2 and -V3; FlashMLA; TransMLA; MHA2MLA.

Expert route: M6.1 → M6.2 → M6.6 → M6.8. Adapter route: M6.3 → M6.4 → M6.5, with M6.7 for theory and M6.9 for combinations and new tasks over time. M5 helps with the serving chapters. Finish with M6.10 → M6.11; teacher supervision is one possible recovery method after pruning, not a requirement.

When a distilled or pruned model also uses low-bit representations, continue into M8 for those additional choices and checks.

M6.1 — How can a model have many networks but use only a few per token?

Start with one feed-forward network, then add alternatives called experts. Trace selection, crowded experts, token rearrangement and combined outputs to distinguish avoided calculations from added routing, memory and training work.

Main sources: A21; Sparsely-Gated Mixture-of-Experts; Switch Transformer; Mixtral; Expert Choice; Soft MoE.

M6.2 — How do we keep some experts from being overloaded while others barely learn?

Follow DeepSeek’s successive responses: smaller experts, shared work, limits on which expert groups can be chosen and feedback that redirects future tokens. Worked examples separate learning useful differences, balancing work and limiting communication instead of treating them as one problem.

Main sources: A21; A29; DeepSeekMoE; DeepSeek-V2; DeepSeek-V3; Auxiliary-Loss-Free Load Balancing.

M6.3 — How can a model learn a new task without updating all its weights?

With a LoRA adapter, keep the original weights fixed and train an added update built from two smaller matrices. Follow gradients, saved files and merging, then compare other adaptation methods to see what still costs memory and computation.

Main sources: A22; LoRA; PEFT implementations; Prompt Tuning; Prefix-Tuning; Houlsby adapters.

M6.4 — Why are there so many versions of LoRA?

Compare methods that reduce the original weights’ storage, allocate added weights among model parts, or change how learning starts and proceeds. Then examine which changes can be combined and what must survive saving and loading. These are responses to different problems, not a ranking.

Main sources: A22; QLoRA; AdaLoRA; rsLoRA; LoRA+ and LoRA-FA; PiSSA, LoftQ and DoRA.

M6.5 — How can one model serve many task-specific adapters at once?

Follow requests through shared calculations, adapter loading, memory competition and safe replacement. Examples show why each request must keep the right adapter and compatible saved history, then compare what real serving systems provide and where their limits lie.

Main sources: A22; PetS; Punica; S-LoRA; Compress then Serve; PEFT and vLLM implementations.

M6.6 — How do chosen experts actually run across several GPUs?

Follow one small batch from expert placement through data transfers, calculations and combined results. Then compare ways to handle uneven loads, one-token generation and changes in placement.

Main sources: A21; A29; DeepSpeed-MoE; MegaBlocks; Megatron Core; DeepEP; EPLB and SGLang.

M6.7 — Why can restricting a weight update still work?

Separate meanings of “low rank,” then use mathematical examples to connect allowed weight changes with training behavior. The final sections explain how to test new learning and retained skills before choosing an adapter size.

Main sources: A22; Intrinsic-dimension studies (Li; Aghajanyan); Malladi’s kernel view of fine-tuning; Zeng and Lee on expressive power; Biderman on learning and forgetting.

M6.8 — What else can a model choose to compute?

Compare choosing attention components, skipping a block for some tokens, and training image-and-text models that use experts. Follow each through learning and saved history to see why similar-looking shortcuts behave differently.

Main sources: A29; Mixture of Attention Heads; SwitchHead; Mixture-of-Depths; MoE-LLaVA.

M6.9 — How can one model choose and combine several task-specific updates?

Begin with adapters trained together or separately, then compare ways of selecting, sharing and merging them. Adding tasks over time shows why keeping old weights unchanged does not necessarily preserve old skills.

Main sources: A22; A29; LoRAMoE, Mixture-of-LoRAs and MoLE; HydraLoRA; TIES and DARE; O-LoRA and AM-LoRA.

M6.10 — How can one model teach another?

Compare teaching through answer probabilities, internal vectors and generated examples, including feedback on the learner’s own attempts. Then connect student design, training data and costs to checks that the student can run without the teacher.

Main sources: Hinton knowledge distillation; DistilBERT and TinyBERT; MiniLM; Sequence-Level Knowledge Distillation; MiniLLM and GKD.

M6.11 — What can we remove from a trained model?

Compare removing individual weights or attention heads, narrowing layers and deleting whole layers. Then follow conversion into a runnable model, quality recovery and execution, explaining why fewer remaining weights do not automatically mean faster computation.

Main sources: SparseGPT and Wanda; Movement Pruning; SliceGPT; Sheared LLaMA; ShortGPT and Minitron.

M7.1 — Can faster generation preserve the original output probabilities?

A cheaper draft proposes several tokens; the main model checks them together. Examples explain acceptance and correction, then follow saved history and timing to show what can be kept and when the extra work pays off.

Main sources: A30; Leviathan speculative decoding; Chen speculative sampling; Hugging Face assisted decoding.

M7.2 — Can one model propose several continuations without a separate draft model?

Medusa adds future-token predictors and checks their combinations as a tree. Follow training labels, allowed branches and saved state, then contrast exact top-choice matching with sampling rules that need not preserve output probabilities.

Main sources: A31; Medusa paper and implementation; Blockwise Parallel Decoding; SpecInfer.

M7.3 — Can the unchanged model reuse its own unfinished guesses?

Lookahead updates several guesses together and saves useful short sequences. Follow those sequences into checks by the main model, then compare memory use, history updates and total work to see whether fewer generation rounds actually save time.

Main sources: A32; Santilli on parallel decoding; Lookahead Decoding paper and implementation.

M7.4 — Why train a model to predict several future tokens?

Start with labels and learning signals, then compare prediction heads and ways to keep training memory manageable. Finally, separate two possible benefits: a better-trained model, and useful guesses that still need checking before they can speed up generation.

Main sources: A33; Better & Faster Large Language Models via Multi-token Prediction; ProphetNet; Blockwise Parallel Decoding; Medusa.

M7.5 — Can a small drafter reuse the main model’s internal vectors?

EAGLE pairs an internal vector with a chosen token to make the next guess. Follow training and verification, then compare changes to branch selection, training with imperfect predicted vectors, and predicting tokens without reconstructing the main model’s vectors.

Main sources: A33; EAGLE; EAGLE-2; HASS; EAGLE-3; Official EAGLE implementation.

M7.6 — How does DeepSeek-V3 train extra predictions into the model itself?

Follow a prediction block from token inputs through training labels and shared weights. Then distinguish the documented single block from repeated drafting steps, and trace the verification and saved state needed to use it during generation.

Main sources: A33; DeepSeek-V3 Technical Report; Official DeepSeek-V3 weights and code; vLLM DeepSeek MTP implementation.

M7.7 — Where should guesses come from, and how many are worth checking?

Compare smaller models, repeated prompt text, external text collections and predictors using earlier guesses. Then explain why checking rules depend on the proposal method, and how useful output and total running time guide the choice of branches.

Main sources: A30; A31; A32; A33; REST; Draft & Verify and Kangaroo; Hydra and Clover; SpecTr and SpecInfer; Sequoia.

M7.8 — When should a server turn speculative generation on or off?

Follow uneven request progress and repair of saved state, then choose draft size from measured costs and load. Finish with fair comparisons, gradual rollout and a return to ordinary generation that preserves already published tokens.

Main sources: A30; A31; A32; A33; Synergy of Speculative Decoding and Batching; TurboSpec; Batch Speculative Decoding Done Right; SPECTRE; vLLM and SGLang implementations.

M8.1 — What changes when numbers use fewer bits?

Work through rounding and reconstructing values, then see how the resulting numbers are multiplied and packed into memory. Examples count the extra information and temporary storage needed, explaining why smaller model files do not automatically mean faster generation.

Main sources: A34; A35; Jacob et al. on integer-only inference; TensorRT-LLM precision documentation; vLLM and llama.cpp implementations.

M8.2 — How do we choose number ranges and decide which layers need more precision?

Start with examples that represent real use, measure the damage from fewer bits, then compare extra precision with small corrective calculations. Worked allocation examples and separate evaluation data show how to make and check those choices under a cost budget.

Main sources: A34; A35; A36; ACIQ; HAWQ and HAWQ-V2; ZeroQuant-V2; LLMC; MBQ.

M8.3 — What if a few extreme values spoil rounding for the rest?

Locate those values, then compare keeping them separate, rescaling values and mixing vector components. Worked transformations show what preserves the original calculation before quantization, and which extra operations remain when running the model.

Main sources: A35; A36; LLM.int8(); SmoothQuant; Outlier Suppression+ and OmniQuant; QuaRot, DuQuant and SpinQuant; FlatQuant.

M8.4 — How should a trained model’s weights be rounded?

Begin with nearest-number rounding, then follow GPTQ as it adjusts still-unrounded weights to compensate for errors. AWQ instead uses input measurements to guide rescaling. Finally, preserve those choices in a packed model that the running software can use.

Main sources: A35; A36; Optimal Brain Compression; GPTQ; AWQ; Marlin.

M8.5 — What if rounding each weight separately is no longer enough?

Explore learned lookup values, separately stored exceptions and codes that represent groups of weights. Follow each method from fitting to storage and weight reconstruction, so “two-bit” labels can be compared by what they preserve, cost and require to run.

Main sources: A35; A36; QLoRA; SqueezeLLM and SpQR; AQLM; QuIP and QuIP#; VPTQ and QMoE.

M8.6 — What about the values created while a request runs?

Start with temporary intermediate values, then follow longer-lived keys and values reused during generation. Comparing their numerical scales, memory costs and updates explains why compressing saved attention data needs more than choosing a smaller number format.

Main sources: A35; A36; SmoothQuant; KIVI and KVQuant; IntactKV and PrefixQuant; QServe; vLLM FP8 documentation.

M8.7 — Can training teach a model to tolerate lower precision?

Follow simulated rounding and the substitute gradients used for learning, then compare full-model, teacher-guided and lower-cost training routes. Finish by checking that exported numbers preserve the choices learned during training.

Main sources: A34; A35; A36; Straight-through gradient estimators; PACT; Learned Step Size Quantization; LLM-QAT; LR-QAT and EfficientQAT.

M8.8 — What if weights use only two or three possible values?

Derive BitNet’s calculations and explain its training from scratch, then contrast OneBit’s teacher-guided conversion of an existing model. Later variants show why precision for intermediate values, real storage formats and specialized execution code still matter.

Main sources: A36; BitNet and BitNet b1.58; OneBit; BitNet a4.8, v2 and 2B4T; bitnet.cpp.

M8.9 — How does a compressed model become something a server can run?

Follow its numbers into files, through loading and into the chosen execution code. Packing examples and increasingly complete checks explain why opening a small file is not enough to establish correct results or low-memory execution.

Main sources: A34; A35; A36; Safetensors; GGUF; ONNX; TorchAO; vLLM, TensorRT-LLM and llama.cpp.

M8.10 — When is a compressed model ready for users?

For a fixed model-and-server setup, compare the same cases, test realistic traffic, and count memory and energy without hiding failures. Finish with gradual rollout and safe recovery, so release depends on useful, reliable service.

Main sources: A34; A35; A36; lm-evaluation-harness; MLPerf and vLLM; Prometheus; Zeus; Google SRE and Kubernetes.

Read X1.1 after M2.2, X1.2 after M3.2 and X1.1, and X1.3 after M5 and X1.2. This is a parallel route, not a prerequisite for the text-only modules.

X1.1 — What should count as one piece of an image, recording, video or action?

Work through patches, words, time windows and action codes, keeping their locations and origins attached. Starting before the encoder shows how preprocessing decides what evidence survives and how much work the model must do.

Main sources: A6; A7; Vision Transformer; LayoutLMv3; Audio Spectrogram Transformer; ViViT; FAST.

X1.2 — How can a language model use what an image encoder sees?

Compare learning a translation between image and language vectors, condensing many image vectors into a few summaries, and letting language read separate visual memory. LLaVA, Flamingo and BLIP examples connect each choice to its training, information loss and running costs.

Main sources: A2; A4; A7; Vision Transformer and CLIP; LLaVA and LLaVA-1.5; Flamingo and Perceiver; BLIP-2 and InstructBLIP.

X1.3 — Which parts of a mixed image-and-text conversation can the model read and learn from?

Expand media placeholders, separate allowed connections from learning targets, then follow the resulting state through training and generation. Concrete examples show why the right media must stay attached when requests are grouped, saved work is reused, or generation is cancelled.

Main sources: A3; A5; A11; LLaVA; Flamingo; BLIP-2 and InstructBLIP; Hugging Face processors; vLLM multimodal inputs.