|

The first question in model compression is usually “how much smaller is it?” The last question must be “should this exact system receive user traffic?” Those questions are separated by quality statistics, workload behavior, resource limits, operational state, and reversibility. The A34 introduction motivates quantization through memory pressure, latency, throughput, power, edge deployment, and long context constraints. A35 and A36 then compare methods by quality and acceleration evidence. That background is ... Read Full Text

Quantization papers often end after reporting reconstructed weights, perplexity, and a kernel benchmark. Deployment begins after that point. The chosen numerical representation must cross a software supply chain without changing meaning: text trained or calibrated state converted logical tensors packed physical layout serialized container and shards loader validation and placement graph lowering and kernel dispatch stateful serving execution Each arrow can introduce a new error class. A converter can use the wrong ... Read Full Text

Most of M8 begins with a trained floating point Transformer and asks how to approximate it. Binary and ternary networks begin with a more radical proposal: make the restricted arithmetic part of the model that optimization is allowed to learn. This idea predates large language models. BinaryConnect used binary weights during forward and backward propagation while retaining precise stored weights for gradient accumulation. XNOR Net extended binary approximations to weights and convolution inputs, tying the ... Read Full Text

Post training quantization treats the pretrained model as fixed. It can choose ranges, transform coordinates, compensate local errors, learn codebooks, and reconstruct blocks, but it begins after the model has already organized its computation for floating point arithmetic. At sufficiently low precision, that restriction becomes the main problem. Quantization aware training changes the direction of adaptation. Instead of asking a converter to approximate the final floating point parameters, it exposes the model to ... Read Full Text

Weight compression solves only the state that exists before the first request arrives. A serving Transformer continuously creates new tensors: normalized activations, projection inputs, queries, attention scores, MLP intermediates, and one key and value row per layer for every retained token. Some survive for microseconds. K/V can survive for the entire request and be read hundreds or thousands of times. This distinction is easy to miss because papers often place weights, activations, and cache under one bit width ... Read Full Text

At eight bits, a uniform scalar grid has enough levels that careful scaling often preserves a useful approximation. At four bits, grouping, clipping, equivalent transforms, and second order compensation can still make the same representation practical. At two or three effective bits per weight, the scalar grid itself becomes a dominant restriction. This is not a magical threshold at exactly four bits. It is a rate distortion transition. As the number of scalar levels shrinks, each independent weight decision ... Read Full Text

The first deployment target for large model quantization is often not integer only inference. It is a mixed operator: weights are stored at low bit width, activations remain FP16 or BF16, and a specialized kernel reconstructs weight tiles close to the multiply accumulate units. That choice follows from the shape of autoregressive inference. During low batch decode, every generated token reuses the model weights while processing only a small number of activation rows. Moving the weight matrices can dominate the ... Read Full Text

The move from ordinary neural network quantization to large Transformer quantization was not only a move to larger matrices. The numerical distributions changed in a structured way. The A35 article devotes substantial space to this transition because a small number of activation dimensions can become much larger than the surrounding values, recur across tokens, and appear through many layers. LLM.int8() reported that these systematic emergent features dominate the failure of naive INT8 matrix multiplication at ... Read Full Text

Once a finite code set exists, the difficult question is no longer how to write the quantization equation. It is how to choose the equation's free state for a model that will see inputs not yet observed. Imagine two calibration sets, each with 100,000 observed tokens. One contains short chat prompts; the other contains long code sequences and sampled late decode traces. The equal token count does not show that their fitted ranges represent the same serving workload. Before choosing a range, ask which inputs, ... Read Full Text

Quantization is often introduced as a cast from a large dtype to a smaller one. That description starts too late. The real problem begins with the way a Transformer turns model scale into physical state. A model with parameters stored at bits per parameter needs, for the raw parameter payload alone, For a 70 billion parameter model, that idealized payload is about 280 GB at FP32, 140 GB at FP16/BF16, 70 GB at eight bits, and 35 GB at four bits. Those numbers exclude tensor headers, scales, zero points, padding, ... Read Full Text

The previous M7 articles established algorithms that can reduce the number of serial target model rounds. They did not establish that enabling speculation on a shared server improves latency or capacity. At production scale, speculation competes with batching: The first source of work always advances another live request. The second source is useful only when candidates are accepted. Once the GPU is already well utilized by concurrent requests, speculative positions can displace useful request work and increase ... Read Full Text

M7.1 derived exact draft and verify sampling. M7.2 built a tree from same model prediction heads. M7.5 moved drafting into feature space, and M7.6 followed a causal MTP module chain. Those mechanisms leave a broader systems question unanswered: Which proposal source and candidate topology should a runtime use once draft latency, proposal dependence, tree verification, exactness, cache state, batch size, and the target hardware are all counted? A speculative decoder has five separable design decisions: Changing the ... Read Full Text

M7.4 derived independent future token heads. M7.5 followed a post hoc feature drafter around a frozen target. DeepSeek V3 uses a third design: text main model hidden state shifted ground truth token embedding during training normalize both paths concatenate project back to model width run an additional Transformer block use the shared output head predict one additional future token The owner question is: How does this module preserve a complete causal prediction chain, how are its tensors shifted and trained, and ... Read Full Text

M7.4 showed why independent future token heads do not define a coherent autoregressive continuation. EAGLE chooses a different state variable: text do not draft only future token ids; autoregress over the target model's internal feature space; condition each feature step on the token that was actually sampled; then let the target model verify the resulting token tree. This creates a deeper question than "is a feature easier to predict than a token?" Which target state is reused, how is sampling uncertainty ... Read Full Text

Next token prediction asks one hidden state to explain one future label. Multi token prediction (MTP) asks the same causal prefix to explain several future labels. The idea has two histories that should not be collapsed. One line uses future token heads to reduce serial decoding work: blockwise parallel decoding trained auxiliary predictors and accepted the longest continuation validated by the base model. A second line uses future targets as a representation learning objective: ProphetNet predicted future n grams ... Read Full Text

Autoregressive decoding is not serial because a Transformer is unable to process many rows at once. It is serial because row needs a token selected from the distribution at row , and that token normally does not exist until the target forward has finished. M7.1 introduced one way to manufacture those missing inputs: run a separate drafter. M7.2 introduced another: attach future offset heads and compile their candidates into a tree. Lookahead decoding starts from a different observation. A causal target model can ... Read Full Text

Owner question: How can one target backbone propose several future token alternatives, pack their shared prefixes into one verification tree, and publish only a path whose tokens, positions, and K/V rows agree? M7.1 established the exact one sequence draft/verify contract. It deliberately left a harder construction problem unresolved. A separate autoregressive drafter proposes one path because each draft token becomes context for the next. A set of parallel future token heads does not have that property: every ... Read Full Text

A decoder only language model does not produce a sentence in one invocation. It advances a state machine one sampled token at a time. For a request whose committed prefix ends at token , one ordinary target model round does this: The newly sampled token is output state, but it is not yet a target K/V row. Its row is created when that token becomes an input to a later target call. This one token offset is already important in ordinary decoding; it becomes a correctness boundary when speculation creates provisional ... Read Full Text

A large Transformer has already been trained. Its quality is useful, but its deployment graph is too expensive. We may want fewer stored bytes, a lower memory bandwidth requirement, smaller matrix multiplications, fewer blocks, or a model that fits on fewer devices. This creates a question that sounds simple: Which parts of the trained model can be removed without destroying the behavior that made the model useful? The word pruning covers the family of methods that answer this question by removing parameter values ... Read Full Text

A large model has been trained, evaluated, and found useful. It is also too expensive for the intended deployment. Perhaps it misses a latency target, cannot fit on the available device, or consumes too much memory when many replicas are required. A natural question follows: Can the behavior of the large, well trained model be transferred into a smaller model that is cheaper to run? That question sounds similar to quantization, pruning, or low rank adaptation, but the operation is different. Quantization changes ... Read Full Text

M6.3 derived the LoRA operator and its gradients. M6.4 compared changes to rank, scaling, initialization, precision, and weight parameterization. M6.5 treated independently trained adapters as serving objects that must be loaded, batched, pinned, and associated with request K/V state. M6.7 then separated factor rank from objective intrinsic dimension and showed why factor coordinates are not unique. Those articles intentionally stop before a different family of questions: If several low rank updates exist at the ... Read Full Text

M6.1 and M6.2 route token rows among FFN experts. M6.6 turns those logical routes into distributed execution. The source Transformer Series also points to a broader design space: attention heads can be experts, tokens can be routed around whole blocks, and image/text tokens can share a sparsely activated multimodal expert pool. These systems all use words such as router, expert, top k, capacity, and load balance. Their tensor contracts are not the same. What changes when routing selects attention components, token ... Read Full Text

M6.3 derived the LoRA operator, tensor shapes, gradients, initialization, memory ledger, and merge path. M6.4 compared methods that change rank allocation, scaling, initialization, precision, optimizer dynamics, or weight parameterization. This article steps back from method selection and asks the harder question: Under what assumptions can a low rank parameterization recover useful fine tuning behavior, and where do those assumptions fail? The popular answer is often one sentence: text Fine tuning has a low ... Read Full Text

M6.1 followed a token through a router, capacity check, expert buffer, and weighted combine. M6.2 moved the same computation across expert parallel ranks and introduced all to all traffic, device limited routing, shared experts, and load balancing. Those mechanisms define the mathematical layer, but a production runtime must answer a more physical question: How does a logical top k route become executable distributed work under device placement, variable token counts, kernel shape, precision, topology, and latency ... Read Full Text

M6.3 defined a low rank update. M6.4 compared ways to train and parameterize it. Neither article answers the production question: How can one resident base model serve requests for many adapters without corrupting request state or fragmenting every batch by adapter identity? 0x00A The Runtime Problem Predates LoRA Parameter efficient training and parameter efficient serving are different problems. Reducing the task specific artifact from a complete model copy to a small parameter set does not tell an inference ... Read Full Text

M6.3 specified vanilla LoRA: with frozen , trainable and , and usually: That baseline contains several independent design decisions: text base representation: full precision or quantized rank: fixed per matrix or allocated dynamically scaling: alpha/r or another rank dependent rule optimization: same or different learning rates, or one frozen factor initialization: random/zero, spectral, gradient aligned, or quantization aware weight parameterization: additive update only or magnitude/direction decomposition ... Read Full Text

Suppose one pretrained model must support a legal document workflow, a customer support assistant, and a code review tool. All three applications can reuse the base model's language and reasoning capabilities, but they do not need exactly the same behavior. Each has its own data, output contract, vocabulary emphasis, and failure policy. The direct solution is to copy the pretrained checkpoint and fully fine tune one copy per task. That gives every task the freedom to change any selected weight. It also turns ... Read Full Text

M6.1 ended with a complete logical MoE layer: The owner question for this article is: How do routing objectives, expert structure, and placement constraints co evolve when a sparse MoE layer is trained and distributed at scale? That pipeline can look like a clean way to buy more parameters without paying for every expert on every token. It hides the harder question: what must remain true for conditional computation to work after the router, experts, and hardware begin learning and operating together? A29 ... Read Full Text

The usual way to increase a neural network's capacity is to make every layer wider or to add more layers. In a dense Transformer, that also enlarges the path executed by every token. The same expanded feed forward matrices process punctuation, copied text, a rare technical term, and a difficult reasoning step: For a SwiGLU FFN with and , the three projection matrices contain approximately: parameters. Under a two FLOP multiply add convention, one token's three projections account for roughly 352 million arithmetic ... Read Full Text

M5.7 reduced K/V cache by changing the number of K/V heads: Multi Head Latent Attention changes a deeper object. Instead of deciding only how many full K/V heads exist, it asks whether historical content must be stored as full per head K/V at all. DeepSeek V2's full architecture figure locates that change inside the block. The lower panel is the relevant one here: the hidden row is down projected into a latent content object and a RoPE key path before multi head attention, and the hatched latent/positional state ... Read Full Text

The first six M5 articles treat K/V as persistent serving state. They ask when that state is created, how requests are scheduled around it, where exact blocks live, which history remains resident, and whether prefill and decode should own the state on different workers. M5.7 begins one step earlier. The cache allocator does not decide how many K/V heads exist. That count was fixed by the model architecture, encoded in the checkpoint, and inherited by every later serving component. Understanding why architectures ... Read Full Text

Owner question: when should prefill and decode run on different workers, where should their K/V state live, and what makes the handoff correct and worthwhile? The question did not begin with a network. It began with a property of autoregressive generation: one request first consumes many prompt rows in parallel, then repeatedly advances by one new token. The same model therefore exposes two execution shapes inside one request. A scheduler can change when those shapes run; chunking can change their service quanta; ... Read Full Text

The K/V cache began as a computation saving device. During autoregressive decode, the new query at layer attends to historical keys and values that were already projected during prefill or earlier decode iterations. Keeping those projections avoids recomputing the complete prefix for every generated token. That optimization changes the systems problem. The checkpoint weights are fixed size, but committed K/V grows with request length, live sequences, layers, and stored K/V heads. The first serving design is ... Read Full Text

M5.1 established why autoregressive serving materializes K/V state in the first place. Without a cache, every decode step would project keys and values again for the entire accepted prefix. With a cache, each committed token contributes one persistent K/V row per layer and later queries reuse those rows. That exchange removes repeated projection work; it does not make a long history free. Under full cache attention, the persistent state grows with the accepted sequence, and every new query remains logically able ... Read Full Text

M5.1 established that a K/V cache is persistent request state rather than an anonymous optimization tensor. M5.2 then made request membership an iteration level decision: That loop still lacks a physical answer. A scheduler may estimate future K/V pressure and decide that a request should run, but those decisions do not create an address, reserve a real block, establish ownership, or prove that an old prefix is reusable. The owner question is: How does a serving runtime turn one logically continuous K/V sequence ... Read Full Text

0x00A From A Rectangular Batch To A Repeated Decision Batching is often introduced as a tensor shape operation. A collection of examples with compatible dimensions becomes one leading batch axis: That view is sufficient when the examples enter together, execute the same graph, and leave together. Training batches and many offline inference jobs have this contract. Padding may waste work, but the membership of the batch is not itself a decision repeated inside the forward pass. Autoregressive serving breaks that ... Read Full Text

0x00A From One Cached Loop To A Stateful Service The mathematical Transformer describes a mapping from visible token rows to new hidden rows. It does not, by itself, specify how a service should preserve the intermediate state of an unfinished request. That distinction is easy to miss when inference is introduced with one prompt and one generation loop. M3.10 owns that loop. It derives the single request state transition from prompt prefill to a token decision, the next decode input, cache append, position ... Read Full Text

M4.7 derives an allocation under a training compute constraint. M4.8 asks whether the fitted surface survives hidden scales, alternative curve families, repeated data, and dependence. Neither article yet decides which artifact should be trained and operated for a real workload. That last step changes the objective. A model can be suboptimal for minimizing training loss at one fixed training budget and still be preferable over its operational lifetime. A smaller model trained on more tokens may cost more to train ... Read Full Text

M4.7 ends with a fitted estimator and a certified optimization procedure. Those are necessary outputs, but they are not yet evidence that the estimator predicts a scale it did not observe. A fitting routine can converge to the declared optimum while the declared functional form is wrong. A curve can interpolate every measured run and still choose the wrong target scale model. A dataset can contain billions of processed tokens while exposing far less independent information because the same examples, sources, or ... Read Full Text

M4.1 starts from a model configuration and derives an analytic resource ledger: That ledger answers what a proposed run should cost. It cannot answer what loss the run will reach. Before a large training run exists, a team must still choose its model size, data volume, training horizon, optimizer policy, and compute budget. The final run is often too expensive to use as the first experiment, yet its configuration must be selected before its result can be observed. This creates M4.7's owner question: Given a ... Read Full Text

M4.3 derived how tiled online softmax removes the quadratic score and probability matrices from HBM. M4.4 then showed that an IO aware algorithm can still leave the GPU underused because work partition, warp ownership, and non matrix instructions matter. M4.5 moved to the asymmetric one query decode shape. This final article in M4's execution/kernel branch asks a different question: Once HBM traffic and coarse work partitioning have improved, how must attention be rescheduled around the asynchronous engines, on ... Read Full Text

The owner question is: When query length collapses to one, which axis can a kernel parallelize, how are partial softmax states recombined exactly, and how do masks, K/V layout, GQA, and framework dispatch constrain the implementation that actually runs? This is not a minor epilogue to FlashAttention 2. It is the point where the shape of autoregressive inference invalidates the most abundant work axis used by prefill and training kernels. The logical attention equation has not changed. The materialized schedule, ... Read Full Text

M4.3 established a result that is easy to state and difficult to implement: exact dense attention does not need to materialize its complete score and probability matrices in HBM. Tiled execution can stream , , and , carry compact row state, and recompute local score tiles in backward. That result removes a major memory schedule bottleneck. It does not prove that the remaining kernel uses the GPU efficiently. Owner question: after FlashAttention 1 removes global traffic, how does FlashAttention 2 change state ... Read Full Text

M4.1 separated a Transformer into parameter, state, activation, operation, transfer, and K/V ledgers. M4.2 then showed why those logical ledgers do not determine one GPU runtime: the selected kernel still decides which objects cross HBM, which tiles remain on chip, and how much parallel work reaches the SMs. M4.3 applies those contracts to one exact attention algorithm. Its owner question is: How can exact dense attention stream score and value tiles without globally materializing the score or probability ... Read Full Text

M4.1 converted a Transformer into several ledgers: Those ledgers are necessary, but they do not execute themselves. For a long time, operation count was the natural first approximation to machine cost. If one algorithm performed fewer operations than another, the first one looked cheaper. That approximation remains useful when both implementations use the same machine path and keep the same hardware busy. It breaks when an accelerator can execute arithmetic much faster than it can supply operands, or when a ... Read Full Text

A neural network description once fit comfortably into a short sentence: name the architecture, count its learned parameters, and report how many arithmetic operations one forward pass requires. Parameter count became a useful common language because it describes a stable property of the learned state. It does not depend on which GPU happens to load the checkpoint, and it gives a first approximation of how much capacity and weight storage the model carries. That shorthand remains useful. It is not a cost model. ... Read Full Text

M3.10 closed one autoregressive request: a selected vocabulary token is appended, model state advances, and the loop repeats until a stopping contract closes the output. That is only one way to consume Transformer state. An encoder style call can expose all valid contextual rows at once: This is an explicit input contract. Tokenization is complete, M2.2 has already constructed the initial rows, and M3.1 M3.7 have already contextualized them. M3.11 receives plus row/task metadata; it does not reopen token ID ... Read Full Text

M3.9 followed the training path from batches to loss. Training has a rectangular target tensor, so the code can build: Inference removes the most important object in that list: there is no . The model does not receive the answer sequence and then compare many predicted rows against known labels. It receives the current context, predicts one token, appends that token, and then uses the enlarged prefix as the next input. The local contrast is: The target side object is now a runtime prefix. In the Harvard encoder ... Read Full Text

There is a sentence that is easy to remember and easy to misunderstand: If we read it too quickly, it sounds as if training has stopped being autoregressive. That is not the case. The supervised objective is still a next token objective: For encoder decoder training, the decoder row that predicts should only use the source sequence and the previous target tokens . What changes during training is not the probability factorization. What changes is that the full gold target sentence is already available in the ... Read Full Text

M3.7 ended with final normalized state rows: Each row is a learned continuous representation. It is not yet: The output head, also called the language model head, generator, vocabulary projection, or unembedding, changes the final axis: For all sequence rows: For one selected inference row: M3.8 owns both the projection and the local decoding policy that consumes its vocabulary scores. M3.10 owns the surrounding autoregressive state machine: appending IDs, updating cache state, stopping, and invoking the next ... Read Full Text

M3.1 opened one attention call. M3.2 added projection, head layout, concatenation, and the output projection. M3.6 reconstructed the dense and gated MLP. Those sublayers share an important output contract: But a Transformer block is not merely: Each sublayer updates a persistent state tensor through a residual branch, normalization, and sometimes dropout: M3.7 owns that block update. Its local questions are: These choices often preserve every visible tensor shape. A shape correct implementation can still be a ... Read Full Text

M3.1 opened the attention score path. M3.2 wrapped that path in Q/K/V projections, head layout, concatenation, and the output projection. The attention sublayer returns model width state rows: Those rows have already exchanged information across the sequence axis. The other main Transformer sublayer performs a different operation: The local object in M3.6 is therefore: The sequence length does not change. The batch size does not change. No token row reads another token row inside this dense FFN call. How the two ... Read Full Text

M3.3 added coordinates to state rows and pairwise scores. M3.4 showed how RoPE rotates Q and K so their dot product carries position structure. The historical problem is older than today's configuration. Early sequence to sequence systems compressed a source into bounded recurrent state; attention made source rows addressable, but the first Transformer still trained and evaluated finite segments with explicit position signals. Later work separated several different ways of crossing that boundary: Period ... Read Full Text

M3.3 put coordinates into the state rows: M3.3 also put pairwise geometry into the attention logit matrix: RoPE uses a third implementation boundary: In code object terms: RoPE is not: and it is not: It is: The overview figure from the handwritten RoPE article is the right mental model: Q and K are transformed by position dependent rotations before attention scoring. V is not rotated. The RoFormer paper frames the same idea at the method level: absolute positions are encoded with rotation matrices while the self ... Read Full Text

Historical pressure: parallel execution removed an implicit clock Sequence models did not begin by inventing a position vector. In a recurrent network, the update at step consumes the state produced at step . The traversal order is part of the computation graph. In a temporal convolution, a kernel is applied at specific offsets, so locality is part of the operator. Those mechanisms can still lose long range information or make training difficult, but neither begins from an unordered set of rows. Self attention ... Read Full Text

M3.1 opened one scaled dot product attention call: That function assumes Q/K/V are already projected and head shaped: M3.2 opens the caller that creates those tensors. The local object is the Harvard module from A12, reconciled against the official Annotated Transformer: This article is not another explanation of softmax attention. It is about the tensor layout around the attention primitive. The original Transformer paper separates the inner scaled dot product primitive from the outer multi head wrapper. That ... Read Full Text

M1.1 traced the long pressure that made attention useful: recurrent encoder decoder systems had to expose a source sequence to a decoder without forcing every target decision through one fixed summary. M3.1 begins later and asks a narrower implementation question. Once a caller has produced query, key, and value tensors, what exact operation turns them into contextualized value rows? The active input route is not M2.1 through M3.5. It is: M3.1 deliberately studies the local primitive before its caller. It accepts ... Read Full Text

In M2.1, we saw how a tokenizer divides text into tokens and assigns each token an integer ID. The next step is to convert these IDs into the vectors used in the Transformer's calculations. This is the purpose of the input embedding layer , and the second stage of the text to vector conversion introduced at the beginning of Module 2. Consider again the sentence: The DeepSeek V3 tokenizer gives us four tokens and their IDs: The ID identifies the token in the vocabulary. Its numerical value comes from the ... Read Full Text

When we ask a language model to translate a sentence or continue a paragraph, we give it written text. The layers of a Transformer perform calculations on vectors, so we need to decide how that text will be represented before it reaches those layers. One of the first choices is what to represent separately: whole words, parts of words, or smaller units such as individual characters and bytes. These units are called tokens . A token can correspond to a word, a word fragment, punctuation, or a sequence of bytes. A ... Read Full Text

Modeling a sequence requires more than assigning a representation to each item. A prediction may depend on information several positions away, and the task determines which information it may use. In translation, source and target can differ in length and word order; during autoregressive generation, the available output prefix grows one token at a time. Recurrence, convolution, and attention provide different answers to this information access problem. They differ in where earlier information is stored, how a ... Read Full Text

A technical study of foundation models, from learning objectives and Transformer architecture to training, execution, and deployment, followed by autonomous agent infrastructure around a fixed trained model. ... Read Full Text

A reading guide to Transformer System: what each module explains, the questions behind its articles, and where to find their main sources. ... Read Full Text

Transformer System began as a rewrite of the Transformer Series. The early assumption was that a better structure would make the same material easier to understand. That was only partly true. Reorganization removed repetition, but it also exposed a more dangerous failure: a concise synthesis can preserve conclusions while deleting the derivations, tensor transitions, implementation choices, training state, runtime consequences, and paper lineage that made the conclusions useful. The production method changed in ... Read Full Text