M8.6 - Live-Data Quantization: Activations and KV Cache

M8.6 - Live-Data Quantization: Activations and KV Cache

0x00 From Static Model Memory To Live Request State

Weight compression solves only the state that exists before the first request arrives. A serving Transformer continuously creates new tensors: normalized activations, projection inputs, queries, attention scores, MLP intermediates, and one key and value row per layer for every retained token. Some survive for microseconds. K/V can survive for the entire request and be read hundreds or thousands of times.

This distinction is easy to miss because papers often place weights, activations, and cache under one bit-width table. Their operational problems are different:

weight:
    known offline, quantized once, shared by all requests

activation tile:
    unknown until an operator runs, usually consumed once,
    scale computation may lie on the latency-critical path

K/V cache row:
    created online, retained across decode steps,
    participates in page allocation and request state transitions

The A34 article introduces static and dynamic activation quantization. A35 explains why LLM activation distributions and prefill/decode behavior make that choice harder. M8.6 extends the same background into stateful serving, where a quantized value is not complete until its scale lifetime and cache transition semantics are defined.

0x00A Why Activation Quantization Is Harder Than Weight Quantization

Weights can be scanned exhaustively. Activations are conditional samples from the model and workload. Their range can change with prompt domain, token position, sequence length, modality, layer, generated history, and batching. Persistent outlier channels can make a tensor-wide scale waste most integer levels, while token-local spikes can defeat a static range.

There are three broad responses:

Dynamic scaling is not automatically slower, because a producer kernel may fuse range reduction and quantization while data is already in registers. It is not automatically free either. The answer depends on tensor shape, reduction axis, launch structure, and whether the consumer has a matching low-precision kernel.

0x00B Prefill And Decode Create Different Numerical Economies

Prefill processes many prompt tokens at once. Linear layers can expose large GEMMs, many K/V rows are created together, and low-precision tensor-core paths may dominate the opportunity. Decode processes a small number of new token rows per step while repeatedly reading static weights and an expanding cache. It tends to expose memory traffic, launch overhead, cache capacity, and scheduler interactions more sharply.

The same A8 declaration can therefore mean two different performance experiments. A prefill kernel may amortize scale computation over a large tile. A decode kernel may spend a larger fraction of time reducing a few rows. A cache format may improve batch capacity even when its dequantization makes one attention step slightly slower. M8.6 keeps phase-specific ledgers instead of assigning one speed number to the format.

0x00C K/V Quantization Is A State-Machine Change

Without caching, autoregressive attention would recompute keys and values for the full prefix at every step. The cache avoids that compute by retaining per-layer state. Its raw payload grows approximately with layers, retained tokens, K/V heads, head dimension, two tensors, and bytes per element. Long contexts and large batches can make it comparable to or larger than the weight memory left after quantization.

Quantizing this state changes more than arithmetic:

• append must write codes and qparams atomically;
• attention must interpret the same scale axis used by the writer;
• page moves and request reordering must preserve metadata ownership;
• prefix reuse must reject incompatible cache formats;
• speculative branches must fork, commit, and discard codes and metadata together;
• residual high-precision windows must move as tokens age;
• rollback cannot retain cache rows produced under an incompatible schema.

KIVI illustrates why distribution analysis matters: it reports different preferred grouping directions for key and value caches rather than applying one symmetric policy to both. The broader lesson is that KV4 or KV2 names a payload width, not a complete cache ABI.

0x00D The Tensor Does Not Exist Until A Request Runs

M8.1 defined codes, scales, zero-points, packing, and the kernel contract. M8.2 built calibration and sensitivity experiments. M8.3 moved activation difficulty through equivalent transforms. M8.4 and M8.5 then compressed static weights.

None of those articles has yet compressed the data created by a live request.

The owner question is:

How are request-dependent activations and persistent K/V quantized, scaled, packed, updated, and consumed without breaking the M5 serving state machine?

The question contains two different lifetimes.

• A linear-layer activation tile is ephemeral. A normalization or activation kernel produces it, a GEMM consumes it, and the tile can disappear.
• A K/V row is persistent request state. It is appended once, read by every later decode step, moved between cache pages, shared by exact prefixes, forked by speculative branches, truncated after rejection, and finally released.

Both are called activation quantization in papers. They do not have the same quantizer, scale lifetime, metadata, or failure modes.

This article owns the live-data path:

The local A35/A36 posts provide the W8A8, activation-granularity, SmoothQuant, PrefixQuant, and KV-quantization spine. KIVI, KVQuant, PrefixQuant, QServe, and vLLM v0.25.0 behavior are checked against their primary papers or official documentation. All paper quality and speed values are source-reported, not generalized deployment guarantees.

0x01 Two Live-Data Contracts

Use these shapes throughout:

and for one attention layer:

$M$ is a flattened token-row count for the local GEMM. During prefill it may equal a large packed batch of prompt tokens. During decode it can be close to the active request count. $T_{kv}$ is the retained prefix length, so it grows during generation even when $M$ remains small.

0x01A Scale Lifetime Is Part Of The Representation

The bit width does not determine the scale lifetime. FP8 KV can use one calibrated tensor scale, one scale per attention head, or another runtime-specific scheme. INT4 KV can use per-head affine parameters, K-per-channel/V-per-token parameters, or grouped residual state.

0x01B Prefill And Decode Stress Different Parts Of The Contract

The local A35 source makes the broad phase distinction:

prefill:
    large M
    GEMMs can be compute-heavy
    W+A low-precision tensor-core paths can matter
    many K/V rows are produced at once

decode:
    small M for each linear operator
    static weight reads and retained KV reads dominate more often
    one new K/V row is appended per active request
    every retained K/V row can be reread by attention

This is a workload tendency, not a universal hardware theorem. Batch size, model shape, context length, parallelism, kernel implementation, and hardware all move the roofline.

0x01C Precision Has Four Independent Locations

A declaration such as W4A8KV4 names three stored/operand precisions but still omits important state:

W4:
    how weights are grouped, packed, zero-pointed, and decoded

A8:
    which activation rows are quantized, when scales are reduced,
    whether Q/K/V inputs are included, and accumulation dtype

KV4:
    K and V axes, affine/symmetric convention, scale lifetime,
    residual window, RoPE stage, page layout, and attention kernel

accumulator/output:
    integer or floating accumulation, epilogue scaling,
    residual-add dtype, and next-operator contract

M8.6 is about those omitted contracts.

0x02 Dynamic Activation Quantization

0x02A Deriving Per-Token Symmetric Quantization

Take one finite, nonempty activation row $x_m\in\mathbb{R}^{K}$. For the narrow symmetric signed alphabet with integer $b\ge2$, let:

A symmetric scale that maps the largest magnitude to the endpoint is:

The all-zero branch uses zero codes and the exactly representable scale sentinel one; it does not divide by zero. The codes and reconstruction are:

Each row receives its own scale because one token with an outlier should not make every other token’s grid coarse. In matrix form, if $D_x=\operatorname{diag}(s^x_1,\ldots,s^x_M)$:

Renderer-safe form:

s_x[m] = max_k(abs(X[m, k])) / qmax
if the entire row is zero: s_x[m] = 1, Q_x[m, :] = 0
otherwise: round s_x[m] to the declared qparam dtype before encoding
Q_x[m, k] = clip(round(X[m, k] / s_x[m]), -qmax, qmax)
X_hat = diag(s_x) @ Q_x

Those equations first describe real arithmetic. A stored scale is an effective qparam, not an invisible full-precision observer value. The reference below uses ties-to-even and FP16/FP32 scale storage, re-reads that scale before encoding, and rejects a nonzero row whose scale overflows, underflows, or becomes subnormal. This explicit FTZ-safe domain is one policy, not a universal kernel rule; a different minimum-scale or fallback policy must be declared and tested at both producer and consumer.

The reduction is live work. It reads the row, computes an absolute maximum, derives a scale, rounds and clips each element, and emits both codes and metadata. A useful implementation fuses that work into a producer such as RMSNorm, LayerNorm, or an activation kernel rather than writing FP16 $X$ to HBM and rereading it in a separate quantization pass.

0x02B Why Per-Token Activation And Per-Channel Weight Scales Fit GEMM

Suppose weights use one symmetric scale per output channel:

Then:

For output element $(m,n)$:

The integer GEMM produces the reduction term. The epilogue multiplies each output by an outer product of one token scale and one output-channel scale. This is why the axes are operationally attractive: neither scale varies along the reduction dimension $K$.

int_acc[m, n] = sum_k Q_x[m, k] * Q_w[n, k]
Y_hat[m, n] = s_x[m] * s_w[n] * int_acc[m, n]

If a scale varies with $k$, it cannot generally be pulled outside the dot product. The kernel must apply it during the reduction, transform the opposite operand equivalently, or use a finer-grained low-precision instruction that understands the scale blocks.

0x02C Affine Zero-Points Add Reduction Corrections

Let both operands use affine representations:

Expanding one output dot product gives:

The raw integer dot product is only the first term. The kernel also needs row sums, column sums, or an equivalent packed/epilogue strategy. Symmetric activations set $z^x_m=0$, removing two terms and simplifying the online path. That systems advantage is one reason dynamic activation quantizers often choose symmetric INT8 even when an affine grid might reduce local reconstruction error.

0x02D Tensor Trace: RMSNorm To W8A8 GEMM

import numpy as np

def quantize_symmetric_rows(x, bits=8, qparam_dtype=np.float32):
    x = np.asarray(x, dtype=np.float64)         # arithmetic reference only
    dtype = np.dtype(qparam_dtype)
    if type(bits) is not int or not 2 <= bits <= 8:
        raise ValueError("reference stores narrow signed codes in int8")
    if dtype not in (np.dtype("float16"), np.dtype("float32")):
        raise ValueError("declare FP16 or FP32 scale storage")
    if x.ndim != 2 or 0 in x.shape or not np.isfinite(x).all():
        raise ValueError("expected finite, nonempty [M, K] rows")
    qmax = 2 ** (bits - 1) - 1
    amax = np.max(np.abs(x), axis=1)
    zero = amax == 0
    ideal = np.where(zero, 1.0, amax / qmax)
    with np.errstate(over="ignore", under="ignore"):
        scale = ideal.astype(dtype)             # commit effective qparams
    effective = scale.astype(np.float64)
    if (not np.isfinite(effective).all()
            or np.any(effective < np.finfo(dtype).tiny)):
        raise ValueError("nonrepresentable/FTZ-unsafe stored scale")
    codes = np.clip(np.rint(x / effective[:, None]), -qmax, qmax)
    codes[zero] = 0
    return codes.astype(np.int8), scale

def fused_rmsnorm_quantize(x_fp, gamma, eps=1e-6,
                           qparam_dtype=np.float32):
    x = np.asarray(x_fp, dtype=np.float64)
    gamma = np.asarray(gamma, dtype=np.float64)
    if (x.ndim != 2 or 0 in x.shape or gamma.shape != (x.shape[1],)
            or not np.isfinite(x).all() or not np.isfinite(gamma).all()
            or not np.isfinite(eps) or eps <= 0):
        raise ValueError("invalid RMSNorm reference input")
    # Fail explicitly if this simple FP64 oracle exceeds its arithmetic domain.
    with np.errstate(over="raise", invalid="raise", divide="raise"):
        rms = np.sqrt(np.mean(x * x, axis=-1, keepdims=True) + eps)
        normalized = (x / rms) * gamma
    return quantize_symmetric_rows(normalized, qparam_dtype=qparam_dtype)

The following GEMM adapter is separate pseudocode; its accumulator must also be proven wide enough for the reduction and zero-point corrections.

def w8a8_linear(x_q, x_scale, w_q, w_scale, bias=None):
    # x_q: [M, K], w_q: [N, K]
    acc_i32 = int8_gemm_i32(x_q, w_q.T)          # [M, N]
    y = acc_i32 * x_scale[:, None] * w_scale[None, :]
    if bias is not None:
        y = y + bias[None, :]
    return y

The NumPy path is a numerical oracle, not a fused kernel or a claim of bit-exact CUDA arithmetic. An optimized implementation may accumulate or scale in another supported dtype, but it must preserve the declared axes, effective qparams, zero policy, and an explicit arithmetic tolerance.

0x02E Static, Dynamic, And Equivalent-Transform Choices

SmoothQuant does not eliminate M8.6. M8.3 derived the offline diagonal migration that makes activation ranges easier. A runtime still needs to produce activation codes and scales, invoke a compatible GEMM, and measure prefill/decode cost.

0x02F Prefill And Decode Need Separate Traces

During prefill, a fused quantizer can amortize launch and reduction overhead over many rows, and an INT8 GEMM can exploit a large $M$. During decode, $M$ may be small enough that weight-only kernels or GEMV-specialized paths win despite nominally higher activation precision. A runtime can therefore use different operator families by phase.

valid phase-specialized deployment:
    prefill: W8A8 or W4A8 GEMM
    decode:  W4A16 GEMV/GEMM
    KV:      independent FP8/INT4 cache policy

invalid inference:
    "the model is W4A8" therefore every layer and phase uses one kernel

0x03 Scale Placement Inside Attention

Attention is:

The scale axis determines whether dequantization can happen after a matrix product, inside its reduction, or through an equivalent transform.

0x03A Per-Query And Per-Key-Token Scales Form A Logit Outer Product

For one head, suppose:

Then:

The scale pair depends on output coordinates $(t,j)$, not on reduction coordinate $d$. An integer dot product followed by an outer-product scale is legal.

0x03B Per-Channel Key Scales Stay Inside The Dot Product

KIVI and KVQuant exploit fixed key-channel structure. If:

then:

There is no single $s^k_j$ to multiply after the dot product. The implementation must do at least one of the following:

1. dequantize key elements or groups during the reduction;
2. multiply query channels by $s^k_d$ before an integer/low-precision dot;
3. use an equivalent offline transform that changes key and query range together;
4. invoke a kernel whose low-precision primitive natively consumes block scales.

0x03C Value Scales Are Weighted Before The Output Sum

With one value scale per cached token:

The output is:

The value scale varies along the reduction over cached tokens. It must be applied to each value row, folded into each attention weight, or consumed inside a fused kernel. One global post-output scale is not equivalent.

This also explains why V error is not uniform across tokens. If $E_j=\widehat V_j-V_j$:

A badly quantized value row with a large attention weight can dominate the output error. KIVI’s paper uses this observation to motivate per-token V quantization: one token’s range does not coarsen every other token’s values.

0x03D Softmax Makes Logit Error Nonlinear

Let quantization perturb logits by $\delta S$. The softmax Jacobian for one query row is:

For small error:

The same logit error can matter differently depending on the attention distribution. Errors near competing logits can change probability mass more than errors far below the maximum. This is why cache evaluation cannot stop at elementwise K/V MSE.

0x03E RoPE Stage Is Part Of The Cache ABI

For position $n$, RoPE applies an orthogonal block rotation:

A conventional cache stores $\widetilde K_n$. KVQuant instead studies quantizing and storing pre-RoPE $K_n$, then applying $R_n$ after dequantization inside the attention kernel. The two artifacts are not interchangeable.

post-RoPE cache row:
    payload decodes directly to rotated K_tilde[n]

pre-RoPE cache row:
    payload decodes to K[n]
    consumer also needs position n, RoPE parameters, pairing convention,
    and a fused or explicit R_n application

Prefix reuse, offload, and transfer metadata must therefore identify the RoPE stage and exact rotary configuration, not merely dtype and shape.

0x04 KV Cache Rate Is Payload Plus Persistent State

M5.1 derived the unquantized cache payload:

where $b$ is bytes per scalar. With a $b_q$-bit payload:

These two products assume homogeneous full K/V rows, the same retained length in every layer, and uniform scalar width. The quantized formula is only the code stream, and only for rows actually encoded at that width—not an additional charge for rows retained solely in a full-precision tail.

0x04A Metadata Depends On The Quantization Axis

Suppose one affine group stores a scale and zero-point using $b_s+b_z$ bytes.

For per-token groups of $G_d$ head-dimension values, K and V qparams cost approximately:

For K grouped per channel over $G_t$ tokens:

For one scale per head, the metadata can be tiny, but the scale must cover more values and may require calibration. For one scale per token group, metadata grows linearly with sequence length but each row adapts independently.

An exclusive logical representation ledger is:

Renderer-safe ledger:

M_served =
    quantized K/V codes
  + scales and zero-points or codebooks
  + recent full-precision residual rows
  + optional sparse outliers / sink rows
  + page tables, valid lengths, alignment, and headers

Each byte belongs to exactly one term: do not add qparams again if a format’s measured payload size already includes them, or count the same sink/tail row as both dense and exceptional state. For MLA, mixed layers, sliding windows, distinct K/V frontiers, or shared pages, enumerate the actual per-layer segments and retained lengths instead of substituting a fictional $H_{kv}$ into the full-KV product. Physical resident memory is the union of live allocations, with shared allocations counted once; allocator slack and temporary workspaces are separate categories. M8.10 owns measurement of that physical ledger, while M5 owns retention and sharing.

0x04B Numeric GQA Ledger

Use the M5 example:

L = 32 layers
H_kv = 8 KV heads
d_h = 128
T = 4096 retained tokens

The payload-only sizes are:

For an illustrative synchronized-tail INT2 layout with $G_t=G_d=32$, FP16 scale plus FP16 zero-point, and exactly 128 retained FP16 tokens, only $4096-128=3968$ tokens have codes and qparams:

3968-token INT2 K+V payload:               62 MiB
K per-channel/token-group qparams:       15.5 MiB
V per-token/head-dimension-group qparams:15.5 MiB
128-token FP16 K+V residual:               16 MiB
illustrative subtotal:                    109 MiB
plus page/layout/bookkeeping state

This is a transparent shape ledger, not a claim about KIVI’s exact packing or closure schedule. In the pinned KIVI implementation, K closes a residual chunk while V advances a rolling tail; their quantized frontiers need not agree. Count their actual lengths separately. Implementations may also use different qparam precision, grouping, padding, or fused headers.

0x04C Stored Bytes And Decode-Read Bytes Are Different Metrics

For the homogeneous full-prefix, uniform-width example only, at decode step $t$ a first-order code-payload read ledger is:

Across $N$ generated tokens after prompt length $T_0$:

Reducing payload precision can reduce both capacity and read traffic. But the realized latency depends on dequantization, qparam reads, softmax, masking, layout, kernel occupancy, and whether lower precision moves the kernel from bandwidth-bound to compute-bound. QServe demonstrates the failure of the naive inference: a nominal halving from KV8 to KV4 did not automatically yield a 2x attention speedup.

0x04D Architecture Reduction And Quantization Multiply But Do Not Substitute

M5.7 changes $H_{kv}$, while M5.8 changes the cached representation itself. M8.6 changes the bytes used for the resulting state. Within a homogeneous full-KV example, the head-sharing and scalar-width levers multiply in the payload formula:

GQA plus FP8 can reduce more than either alone, but GQA is an architecture/training choice and FP8 cache is a runtime representation choice. Their quality, kernel, and compatibility effects remain separate.

0x05 A Quantized Cache Is A State Machine

M5.1 owns logical cache publication and generation fencing; M5.3 owns physical layout and reuse admission. Quantization extends the state included in every transition. It does not create a second cache owner.

0x05A Why Online Per-Channel Scaling Can Rewrite History

Suppose one key scale per channel uses the maximum observed through time $t$:

If a new row introduces a larger maximum:

Historical integer codes were created under $s_d^{(t)}$. Interpreting them with $s_d^{(t+1)}$ changes their reconstructed values. A correct online scheme must therefore choose one of four designs:

1. requantize historical rows whenever the scale expands;
2. freeze a calibrated per-channel scale offline;
3. partition time into groups, each with its own scale;
4. retain a recent FP16 window until a group is closed and quantized.

KVQuant chooses calibrated per-channel K scales. KIVI chooses grouped K state plus a recent full-precision residual. These are different solutions to the same append problem.

0x05B Per-Token Scaling Is Append-Local

For a new value row $V_t$:

The scale depends only on the new row. Appending token $t+1$ does not invalidate token $t$. This is operationally convenient for a streaming cache, at the price of one scale or scale group per token and live reductions on every append.

0x05C Page Schema

A quantized page needs more than data_ptr and valid_tokens:

QuantizedKVPage:
    request_or_prefix_id
    layer_id
    logical_token_start
    valid_token_count

    key_payload
    value_payload
    key_qparams
    value_qparams
    optional_sparse_outliers
    optional_full_precision_residual

    bit_width / floating format
    signedness and zero-point convention
    K axis and group size
    V axis and group size
    qparam dtype and ordering
    pre_rope_or_post_rope
    rope schema/version
    packing layout/version
    attention kernel family/version

0x05D Append, Fork, Reorder, Truncate, Release

def append_quantized_kv(reference, txn, layer, position, k_fp, v_fp):
    # PREPARE ONLY: transaction-private K AND V, including all metadata.
    reference.prepare_layer(txn, layer, k_fp, v_fp, position)


def publish_quantized_position(reference, txn):
    # Called once, after every required layer/segment is prepared.
    return reference.publish(txn)


def fork_cache(owner, child_ids):
    # Immutable closed pages may be shared by reference.
    # Mutable residual rows need copy-on-write or independent child state.
    return owner.fork_with_cow(child_ids)


def reorder_cache(owner, beam_order):
    # One M5 transition moves row mapping, positions and all attached state.
    # The successor permutes request_page_tables[beam_order] and
    # mutable_residuals[beam_order] together: residual rows follow their beam.
    return owner.publish_reorder(beam_order)


def truncate_cache(owner, new_length):
    # Build/validate a complete successor; stale append transactions must fail.
    # Besides pages, the successor trims mutable residual rows to new_length.
    return owner.publish_truncate(new_length)


def release_cache(owner):
    receipt = owner.close_logically()  # fences all future publications now
    owner.enqueue_quiescent_reclamation(receipt)
    return receipt

These are owner-adapter sketches, not a new implementation of M5. A position becomes public only when all required K/V codes, qparams, residual/sparse state, validity, positions, row mappings and architecture-specific segments form one validated successor. The committed frontier moves inside the same conditional publication; incrementing a counter afterward is not atomic. Current-forward attention may read its own prepared layer through a transaction-private view, never through a partly updated public cache. Section 0x0CA makes this prepare/publish split explicit.

Truncation can land inside a quantization group. A format must support a valid-length mask, partial boundary group, or deterministic rebuild. It invalidates stale generations across all layers. Speculative decoding cannot commit K/V for rejected draft tokens merely because their packed bytes exist; an emitted correction/bonus token is not automatically materialized in the cache. Logical release is immediate, but physical reclamation waits for readers/kernels/transfers to quiesce and ownership to be rechecked. Shared pages remain live for other owners; a timeout is not proof they can be freed.

0x05E Prefix Reuse Requires A Quantization Schema Match

M5.3 owns the authoritative compatibility record and admission receipt. Quantization contributes the following diagnostic fields to that owner’s schema; this is neither an independently complete cache key nor a rule that every provenance field must compare equal:

model and adapter identity
token ids and positions
attention architecture and KV head mapping
KV precision and code convention
K/V quantization axes and group sizes
scale/zero-point or codebook versions
pre-RoPE versus post-RoPE storage
packing/page layout and kernel ABI
calibration artifact identity when scales are static

Two caches containing the same prompt are not interchangeable if one stores pre-RoPE per-channel keys and the other stores post-RoPE FP8 keys.

0x06 KIVI: Asymmetric Axes And A Residual Window

KIVI begins from an empirical asymmetry:

keys:
    persistent outlier channels across tokens
    quantize per channel to isolate their range

values:
    no equally stable channel pattern
    attention mixes value rows with token-dependent weights
    quantize per token so one difficult row does not coarsen every row

The paper’s central result is not merely KV2. It is K per-channel, V per-token, plus a streaming representation that makes per-channel K possible without rewriting the entire cache.

0x06A Deriving The Axis Choice

For key cache $K\in\mathbb{R}^{T\times d_h}$, per-token quantization gives one scale across all channels of a token:

If channel $d^*$ is consistently much larger, it determines $s^K_t$ for nearly every token and wastes levels on ordinary channels. Per-channel scaling instead uses:

so the outlier channel owns only its own coarse grid.

For values, the attention output error is:

Per-token V scales confine a difficult row’s range and error to that row. KIVI reports that K-per-channel and V-per-token is the best low-bit combination in its experiments. This is source-reported evidence for the method, not a universal claim for every architecture.

0x06B Why A Full-Precision Residual Solves The Streaming Group Problem

Per-channel K needs several token rows before it can compute a local channel range. Let the closed group size be $G$ and recent residual length be $R$. Split the cache:

where $K_q,V_q$ are old quantized rows and $K_r,V_r$ are recent full-precision rows.

When the residual exceeds its target, close the oldest $G$ rows:

class KIVILayerState:
    def __init__(self, group_size=32, residual_len=128):
        self.group_size = group_size
        self.residual_len = residual_len
        self.k_groups = []       # K: per-channel within token groups
        self.v_rows = []         # V: per-token, optionally grouped along d_h
        self.k_residual = []     # recent FP16/BF16 rows
        self.v_residual = []

    def append(self, k_new, v_new):
        self.k_residual.append(k_new)
        self.v_residual.append(v_new)

        while len(self.k_residual) > self.residual_len:
            k_old = stack(self.k_residual[:self.group_size])
            v_old = stack(self.v_residual[:self.group_size])

            k_codes, k_qparams = quantize_per_channel(k_old)
            v_codes, v_qparams = quantize_each_token(v_old)

            self.k_groups.append((k_codes, k_qparams))
            self.v_rows.append((v_codes, v_qparams))
            del self.k_residual[:self.group_size]
            del self.v_residual[:self.group_size]

This synchronized-tail class is a teaching variant, not a transcription of KIVI. The pinned implementation closes a K residual chunk at its threshold and moves V rows through a rolling tail, so the two frontiers differ. The class must run on transaction-private state; a failed quantizer cannot expose half an append. The shared invariant is that closed K groups retain stable per-channel qparams; the mutable tail stays full precision until it can be closed without rewriting older groups.

0x06C Attention Must Combine Quantized History And Exact Tail

For one query, compute logits over both regions under a shared softmax:

def kivi_attention(q, state):
    logits_old = qk_dot_dequant_k_per_channel(
        q, state.k_groups
    )
    logits_recent = q @ stack(state.k_residual).T

    logits = concat([logits_old, logits_recent], axis=-1)
    probs = softmax(logits / sqrt(d_h), axis=-1)

    p_old, p_recent = split_at(probs, old_token_count(state))
    out_old = pv_dequant_v_per_token(p_old, state.v_rows)
    out_recent = p_recent @ stack(state.v_residual)
    return out_old + out_recent

The two logit regions cannot be softmaxed independently and then naively added. They compete in one normalization. A tiled implementation may merge online-softmax statistics across regions, but it must be mathematically equivalent to the concatenated softmax.

0x06D Prefill Produces Exact Inter-Layer Activations But Quantized Retention

KIVI distinguishes the current forward computation from retained state. During prefill, a layer computes K/V in full precision and uses the exact tensors for that layer’s attention and subsequent block computation. The rows retained for future decode are then quantized according to the cache policy. Quantizing retention does not require feeding quantized K/V to the next transformer layer as ordinary activations.

current layer computation:
    produce K_fp, V_fp
    compute current attention under method's forward contract

retention path:
    append/quantize K_fp, V_fp into persistent cache state
    keep recent residual in full precision

future decode:
    consume quantized history + full-precision residual

0x06E KIVI Design Choices

The paper uses group size 32 and residual length 128 in its reported setup. Those are source-reported settings, not universal defaults.

0x07 KVQuant: Calibrate The Key Axis, Quantize Values Online

KVQuant starts from a similar K/V distribution observation but chooses a different streaming solution. It calibrates stable K channel state offline, quantizes keys before RoPE, derives a layer-specific nonuniform datatype, optionally isolates sparse outliers, and computes value-row state online.

0x07A KIVI And KVQuant Solve The Same Rewrite Problem Differently

Recall the online per-channel scale:

KIVI prevents historical rewrite by closing finite token groups and keeping a recent residual. KVQuant instead estimates a fixed channel scale $\bar s_d$ from calibration data:

Every request then uses $\bar s_d$, so a new token does not change the interpretation of previous codes.

Offline calibration turns a streaming update problem into a calibration-coverage problem. Neither choice is free.

0x07B Why Pre-RoPE Keys Recover Stable Channels

RoPE mixes coordinate pairs by a position-dependent angle. For one pair $(a,b)$:

If $K_a$ is a stable high-magnitude channel and $K_b$ is small, the rotated magnitude moves between the two coordinates as $\theta_n$ changes with position. A post-RoPE per-channel observer sees a less stationary distribution. KVQuant therefore stores quantized pre-RoPE keys and applies the rotation after dequantization.

def append_kvquant(layer_state, position, k_pre_rope, v):
    k_codes, k_sparse = quantize_key_with_offline_channel_state(
        k_pre_rope,
        scale=layer_state.k_channel_scale,
        zero=layer_state.k_channel_zero,
        codebook=layer_state.k_nuq_codebook,
        thresholds=layer_state.k_outlier_thresholds,
    )

    v_scale, v_zero, v_thresholds = observe_value_row_online(v)
    v_codes, v_sparse = quantize_value_row(
        v, v_scale, v_zero, v_thresholds,
        layer_state.v_nuq_codebook,
    )

    layer_state.append(
        position=position,
        k_codes=k_codes,
        k_sparse=k_sparse,
        v_codes=v_codes,
        v_sparse=v_sparse,
        v_qparams=(v_scale, v_zero),
    )


def attend_kvquant(q_pre_rope, query_position, layer_state):
    q_rot = rope(q_pre_rope, query_position)
    # Kernel lookup-decodes K, adds sparse exceptions, then applies each
    # cached token's RoPE position before the QK reduction.
    logits = fused_pre_rope_key_lookup_rope_qk(q_rot, layer_state)
    probs = softmax(logits / sqrt(d_h))
    return fused_value_lookup_sparse_pv(probs, layer_state)

The cache must preserve each key row’s logical position. Moving RoPE to the consumer is not a local quantizer detail; it changes the attention-kernel ABI.

0x07C Sensitivity-Weighted Nonuniform KV Codes

KVQuant observes that a uniform grid can be wasteful at very low precision. It derives one nonuniform datatype per K or V layer offline. First, each vector is normalized to a common range so a shared codebook can be learned. Let flattened calibration activations be $A_i$, reconstruction be $Q(A_i)$, and $F_{ii}$ be a diagonal Fisher sensitivity. The paper’s objective is:

This produces layer-specific signposts through sensitivity-weighted k-means. Runtime does not run k-means for each value row. It uses the offline codebook, rescaled by the row’s online qparams.

offline:
    collect normalized K/V calibration values
    estimate diagonal Fisher sensitivity
    solve weighted k-means for layer codebook
    calibrate K per-channel scales/zeros and thresholds

online key append:
    use fixed per-channel state + layer codebook

online value append:
    compute per-token scale/zero/outlier threshold
    use fixed layer codebook after row normalization

The method inherits the M8.5 distinction between code index and reconstruction value. A low-bit code is a lookup index into the nonuniform datatype, not an ordinary affine integer unless the schema says so.

0x07D Per-Vector Dense-And-Sparse State

A small number of outliers can stretch one vector’s range. KVQuant uses thresholds aligned with the quantization vector:

K per-channel quantization:
    one outlier threshold per K channel

V per-token quantization:
    one outlier threshold per V token row

Write a row or channel vector as:

The decoder must choose an explicit sparse semantic, as in M8.5. With full-value replacement, the dense decoded path is masked to exactly zero at selected coordinates and the sparse path stores their full values. Alternatively, if the dense reconstruction remains nonzero, an additive sparse path stores the correction relative to that decoded dense value, not the original full value. A low-bit code whose lookup value is nonzero is not an exact hole merely because its code index is zero. The following two-path equations assume exact decoded holes plus full sparse values (or the explicitly tagged decoded-dense correction variant), never full values added to an unmasked dense approximation:

The sparse term can improve low-bit quality, but it adds indices, row/column pointers, irregular reads, and another arithmetic path. KVQuant reports results with one percent outliers; that density and its quality effect are source-reported, not a default budget.

0x07E Attention Sink Exception

KVQuant reports that preserving the first token in FP16 can help at very low precision because later attention layers may allocate disproportionate score to an initial attention sink. This is a distinct exception from a recent residual window:

sink exception:
    fixed logical token, retained for sensitivity reasons

KIVI residual:
    moving recent-token window, retained for grouping and quality

If a sink row is excluded from quantization, calibration must also exclude it from the statistics intended for quantized rows. Otherwise the calibrated codebook/range solves a different population than the runtime encodes.

0x07F Runtime Integration

KVQuant’s paper implements four-bit cache elements as indices into lookup tables that recover FP16 values, uses CSR or CSC for sparse state depending on the append/multiply orientation, and applies RoPE on the fly for pre-RoPE keys. It also separates calibration work:

The paper describes overlapping CPU top-k work for value outlier thresholds with GPU projection work. That is a specific scheduling design, not proof that online outlier detection is free in another runtime.

0x07G IntactKV: Preserve Pivot-Token State Before Quantized Decode

IntactKV identifies a different token exception. In the paper’s studied LLaMA and Vicuna models, very large token-specific activations occur at initial pivot tokens such as [BOS] and, for some prompts, punctuation or fixed system-prompt tokens. Attention can concentrate on those rows, so quantization error in their K/V state can propagate through every later decode step.

The method uses the full-precision model offline to process a fixed prefix $p_{1:P}$ and save its exact per-layer K/V tensors:

At serving time, a compatible quantized model loads that artifact as immutable prefix state and appends the request-dependent cache after it:

All rows still participate in one attention normalization:

The intact rows are not a separately softmaxed correction. They are ordinary causal prefix positions with a different storage/precision provenance.

def build_intact_kv(fp_model, fixed_prefix, identity):
    with no_grad():
        kv = fp_model(fixed_prefix, use_cache=True).past_key_values
    return ImmutablePrefixKV(
        tensors=kv,
        model_hash=identity.model_hash,
        tokenizer_hash=identity.tokenizer_hash,
        prefix_token_ids=tuple(fixed_prefix),
        rope_policy=identity.rope_policy,
        cache_layout=identity.cache_layout,
    )

def begin_intactkv_request(quant_model, artifact, visible_suffix):
    assert artifact.compatible_with(quant_model)
    state = clone_readonly_prefix_pages(artifact)
    return quant_model.generate(
        visible_suffix,
        past_key_values=state,
        position_offset=artifact.prefix_length,
    )

The source paper reports no additional inference computation after the reusable prefix has been prepared. That statement has boundaries: the prefix consumes persistent K/V bytes, must be loaded or mapped, and only applies when requests share the exact fixed prefix. If each request has a different prefix, the full-precision preparation cannot be amortized in the same way.

0x07H Extraction And Calibrated IntactKV Are Different Variants

IntactKV also treats the saved prefix cache as a small trainable parameter set and calibrates it to reduce layer-output MSE between the full-precision and quantized models:

The base quantized model can remain frozen while gradients update only $\Theta_{KV}$. This creates two variants that must not share one vague label:

The paper reports lightweight calibration recipes. Those epochs, sample counts, and times are source-reported, not defaults. A deployment record stores optimizer, objective, corpus, selected checkpoint, held-out error, and whether the final cache remains equal to the original full-precision prefix.

IntactKV and PrefixQuant are related but not interchangeable:

IntactKV:
    keep K/V from an existing fixed prefix exact or calibrated
    visible/system prompt semantics stay tied to that prefix

PrefixQuant:
    discover outlier-inducing token IDs and insert them as method state
    served sequence and hidden-prefix accounting change

both:
    require exact model/tokenizer/position/cache identity
    consume real prefix rows and cannot be omitted from capacity accounting

0x08 PrefixQuant: Turn Token Outliers Into Declared Prefix State

KIVI and KVQuant adapt the representation of K/V rows after tokens have been produced. PrefixQuant starts one step earlier: it identifies token-wise activation outliers that remain difficult after channel-oriented smoothing and materializes selected outlier-inducing tokens as a fixed prefix in the K/V cache.

This addresses a different axis than SmoothQuant or a Hadamard rotation:

The prefix does not delete a difficult token from the model. It changes where that token’s state is introduced and which rows are included in the quantized population.

0x08A Why Rotation Alone Can Leave A Token Stripe

Let $X\in\mathbb R^{T\times K}$ and $R$ be orthogonal. A rotation preserves every row norm:

It can reduce a row’s coordinate maximum, but it cannot remove exceptional row energy. If one token row has a much larger norm than ordinary rows, a static tensor or token-group scale can remain dominated by that row after rotation. PrefixQuant’s source figure gives one concrete case: in its LLaMA2-7B example, the paper reports 4-bit error values of $54.63$ before rotation, $7.88$ after Hadamard rotation, and $0.04$ after adding the selected prefix tokens. These values belong to that figure’s model, tensor, error definition, and setup; they are not universal error rates.

Source: PrefixQuant, Figure 2, arXiv:2410.05265, CC BY 4.0. The displayed errors are source-reported for the illustrated LLaMA2-7B case.

0x08B Prefix Discovery And The Training Boundary

The paper’s prefix-token identification is training-free: calibration traces reveal tokens that repeatedly induce token-wise outliers, and selected token IDs are placed before ordinary inputs. The full PrefixQuant pipeline also introduces trainable parameters for block-wise compensation. These are separate claims:

prefix isolation:
    calibration statistics -> select recurring outlier tokens
    no base-model retraining required for token discovery

optional/full method compensation:
    block-wise optimization -> fit added quantization parameters
    calibration objective and learned state enter the artifact

An implementation should preserve that ablation boundary. Otherwise a quality result from the full optimized method may be incorrectly attributed to the prefix alone.

def discover_prefix_tokens(model, calibration_requests, candidate_token_ids):
    score = {token_id: 0.0 for token_id in candidate_token_ids}

    for request in calibration_requests:
        trace = collect_intermediate_activation_ranges(model, request)
        for token_id in candidate_token_ids:
            score[token_id] += token_outlier_contribution(trace, token_id)

    selected = select_stable_high_score_tokens(score)
    return tuple(selected), score

Token strings are insufficient artifact identity. The same visible string can map differently under another tokenizer, special-token policy, or chat template. Store token IDs and the tokenizer/template hashes that produced them.

0x08C Prefix State Changes The Cache ABI

For a user sequence $x_{1:T}$ and fixed prefix $p_{1:P}$, the served causal sequence is

The user-visible prompt can remain $x_{1:T}$ while the attention state contains $P+T$ positions. Therefore the implementation must define:

prefix token IDs and tokenizer/template hashes
prefix length and logical-position convention
attention-mask and position-ID policy
whether prefix K/V is precomputed or rebuilt
model, transform, quantizer, and scale artifact hashes
prefix-row dtype and body-row dtype
whether prefix rows are excluded from body calibration
cache-reuse identity and accounting of hidden prefix tokens

A short hash of that list is not a complete reuse contract. The M5.3 owner resolves the full request/artifact compatibility record, distinguishing effective execution identity from provenance, and returns a receipt bound to the exact admitted state. Equal visible text alone never authorizes sharing. A runtime must also declare how max_model_len, rate limits, token accounting, and position-dependent logic include the hidden prefix length.

def begin_prefixquant_request(request, artifact, owner):
    # Adapters consume M5.3's full schema; no local hand-maintained hash subset.
    compatibility = owner.resolve_prefix_compatibility(request, artifact)
    receipt = owner.admit_exact_prefix(compatibility)
    # Published prefix snapshot already carries its positions/frontier/ABI.
    state = owner.fork_admitted_prefix(receipt)
    # Token intake is not a claim that their K/V has already been materialized.
    return owner.enqueue_suffix(state, request.token_ids)

If prefixed rows are retained in higher precision while body rows are quantized, the cache is a mixed-schema state machine. If they are quantized, their exact qparams and calibration treatment must be declared. Either design is legitimate; silently treating the prefix as ordinary user-token state is not.

0x08D PrefixQuant Failure Modes

PrefixQuant reports W4A4KV4/W4A8KV4 quality and kernel speedups. Those measurements are source-reported for its implementation, models, and hardware. The transferable result is that token-wise outliers can be converted into explicit, reusable runtime state, provided the cache ABI and calibration population change with them.

0x09 QServe: W4A8KV4 Is A Co-Designed System

QServe combines three runtime paths:

W4:
    progressively quantized weights decoded for INT8 execution

A8:
    per-token symmetric INT8 activations

KV4:
    per-token/per-head asymmetric INT4 cache consumed by a fused attention kernel

The weight representation and activation rotation details connect to M8.3-M8.5. M8.6 owns QServe’s live A8/KV4 state, SmoothAttention scale placement, and the kernel result: fewer cache bits help only if unpack/dequantization remains below the machine’s compute roof.

0x09A SmoothAttention Moves Key Range Into Queries

Let $\Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_{d_h})$ with positive entries. Then:

The attention logits are unchanged before quantization. QServe chooses a key-channel statistic:

with $\alpha=0.5$ in its reported setup. Multiplying $Q_i$ by $\lambda_i$ and dividing $K_i$ by $\lambda_i$ reduces key-channel outlier magnitude while moving range into queries. In QServe’s KV4 discussion, queries are not stored in the low-bit cache, so this trade can make persistent K easier without paying persistent Q memory.

0x09B RoPE Restricts The Diagonal Transform

A diagonal scale does not generally commute with a rotation:

For a RoPE coordinate pair, equal scales do commute with that pair’s rotation because the pairwise diagonal is a scalar multiple of identity. Under QServe’s indexing convention, coordinates $i$ and $i+d_h/2$ are paired, so it constrains:

The paired statistic becomes:

For a pair whose observed maximum is exactly zero, choose the identity scale one instead of evaluating a zero scale. Nonzero paired scales must remain finite, positive and invertible in the declared storage/arithmetic; reject or explicitly fall back when they do not. This is not cosmetic. Independent scaling of the two coordinates would change the RoPE-transformed dot product.

With the row-major projection convention $Q=XW_q^\top$ and $K=XW_k^\top$, the transform can be fused offline into projection rows:

Then $Q'=XW_q'^\top=Q\Lambda$ and $K'=XW_k'^\top=K\Lambda^{-1}$, subject to the chosen row/column convention and RoPE pairing. An exporter must unit-test unquantized attention equivalence before measuring KV4 error.

0x09C Per-Head Asymmetric KV4

For one token’s head vector $A$ with unsigned INT4 codes $q\in[0,15]$ and nonzero range, QServe’s paper-level affine equation uses a rounded offset without clamping that offset to the code alphabet. The pinned cache kernel has a different concrete ABI: a floating FP16 offset, without integer rounding or clamping. Before storage rounding its parameters are:

For a positive-only range $[1,2]$, the ideal offset is $-15$, outside the unsigned code range. Clamping it to zero would reconstruct both endpoints near one rather than represent the range. The codes are clipped; the floating offset is not. This offset ABI is not M8.1’s integer-zero-point convention: the encoder rounds after adding its floating offset.

The pinned implementation stores separate FP16 scale/offset pairs at each (token, head) within a page and re-reads the stored qparams before encoding. Appending another token writes another metadata slot; it does not replace a page-wide scale and reinterpret history.

QServe page-local KV state:
    packed INT4 K/V features
    FP16 scale per (token, head)
    FP16 floating offset per (token, head)
    append writes its own metadata slot; old slots stay unchanged

The following NumPy arithmetic model isolates that ABI. It is not a bit-exact reproduction of CUDA reduction, fused arithmetic, rounding, or packing. It deliberately rejects constant heads and nonrepresentable scales: the cited source fragment divides by the range and does not establish a constant-head fallback. A production extension must specify and validate its own such policy, not attribute this rejection rule or M8.6’s symmetric zero sentinel to QServe.

def qserve_head_reference(a):
    a = np.asarray(a, dtype=np.float64)
    if a.ndim != 1 or a.size == 0 or not np.isfinite(a).all():
        raise ValueError("expected one finite nonempty token/head vector")
    with np.errstate(over="ignore", invalid="ignore"):
        width = np.max(a) - np.min(a)
    if not np.isfinite(width) or width <= 0:
        raise ValueError("constant/nonrepresentable range needs a separate policy")
    ideal_scale = width / 15.0
    with np.errstate(over="ignore", under="ignore", divide="ignore"):
        scale = np.float16(ideal_scale)
        offset = np.float16(-np.min(a) / ideal_scale)
    if (not np.isfinite(scale) or scale < np.finfo(np.float16).tiny
            or not np.isfinite(offset)):
        raise ValueError("unsupported stored qparams")
    codes = np.clip(np.rint(a / float(scale) + float(offset)), 0, 15)
    decoded = (codes - float(offset)) * float(scale)
    return codes.astype(np.uint8), scale, offset, decoded

0x09D Why A Naive KV4 Kernel Can Lose To KV8

Nominally, INT4 reads half the payload of INT8. But each packed nibble requires extraction, conversion, zero-point subtraction, and scaling. A fused attention kernel also performs masking, online softmax, value accumulation, cache update, and address calculations.

QServe reports that directly replacing KV8 accesses with dynamic per-head KV4 was faster on L40S but slower on A100. Its analysis attributes this to CUDA-core arithmetic during dequantization reaching the compute roof. The optimized path uses lower-cost bit conversion, FP16 rather than FP32 arithmetic where valid, prefetched qparams, and simplified control/address logic.

incorrect roofline argument:
    KV4 bytes = KV8 bytes / 2
    therefore latency = latency_KV8 / 2

required argument:
    bytes saved
    + code/qparam bytes
    + bit extraction and dequant operations
    + softmax/mask/value work
    + target GPU compute and bandwidth roofs
    + occupancy and sequence-shape effects

Published QServe speedups are source-reported for its hardware, models, shapes, and software. The general result is the co-design requirement, not the exact multiplier.

0x09E Live Activation Fusion

QServe fuses dynamic activation quantization into producer kernels so the W4A8 GEMM receives INT8 activation rows and their scales. The online trace is:

RMSNorm / activation producer
    -> transform/rotation if required by offline graph
    -> per-token absmax reduction
    -> INT8 codes + row scale
    -> W4A8 kernel decodes weight tile into protected INT8 path
    -> integer accumulation
    -> token-scale / output-scale / zero correction epilogue

The precise weight progressive-quantization decoder belongs to M8.5/M8.9. The live-data lesson is that A8 and KV4 are not independent checkboxes: producer fusion, projection transforms, cache packing, and attention consumption all target one runtime.

0x0A Production FP8 Cache Schemes

Research INT2/INT4 methods maximize compression and expose difficult axis choices. Production runtimes often begin with FP8 because it halves FP16/BF16 payload while retaining a floating representation and simpler dequantization semantics.

As of the vLLM v0.25.0 documentation checked for this article, the official quantized-KV-cache interface documents:

• one scale for each Q, K, and V tensor ([1] scale shapes);
• per-attention-head scales (Q: [H_q], K/V: [H_{kv}]) on the documented FlashAttention path with llm-compressor calibration;
• no calibration, which uses scales of 1.0;
• warmup calibration from a batch of random tokens, after which scales are fixed;
• dataset calibration through llm-compressor, which the docs recommend for highest quality;
• FP8 E4M3 and E5M2 options under documented hardware/software constraints;
• selective layer or layer-type skipping when some attention layers are more sensitive;
• a FlashAttention 3 path where Q can also be quantized and attention runs in the FP8 domain.

These are version-specific runtime capabilities, not a universal description of vLLM releases or every attention backend.

0x0AA FP8 Still Needs A Scale Manifest

FP8 supplies exponent and mantissa fields, but a serving implementation can still apply an external tensor/head scale. A cache manifest needs:

fp8 format: E4M3 or E5M2
scale convention and dtype
scale shape: tensor, head, layer, or another supported axis
calibration method and artifact identity
layers skipped from quantization
whether Q is quantized in the attention path
attention backend and minimum runtime version

Setting all scales to 1.0 is a representation choice, not calibration. It can work for a compatible model distribution, but it should not be silently presented as an observed range fit.

0x0AB Calibration Choices Change Deployment Reproducibility

The chosen scale artifact should travel with the model/runtime manifest. A prefix cache produced under one scale set must not be decoded under another.

0x0AC Mixed Layer Policies Are A First-Class Design

If a sliding-window layer or a small subset of layers is sensitive, leaving those layers in native precision can produce a better memory-quality point than forcing one global cache dtype. Let $\mathcal Q$ be quantized layers and $\mathcal F$ be full-precision layers:

This displayed product still assumes the same retained $T$, $H_{kv}$ and $d_h$ in every layer, disjoint full/quantized row sets, no separate tail, and metadata not already included in scalar bytes. For an actual sliding-window or MLA policy, sum its real per-layer representation segments and unique allocations as in 0x04A. This is the live-state analogue of M8.2’s precision map; the allocation unit is persistent cache state and backend support, not static weight blocks.

0x0B Method Comparison By Design Decision

The table is a map, not a ranking. KIVI avoids offline tuning but carries residual/group state. KVQuant spends calibration and metadata to avoid online K-scale evolution. IntactKV preserves an existing prefix’s K/V and optionally calibrates that state. PrefixQuant converts selected token outliers into inserted prefix state and therefore changes cache identity differently. QServe gives up method modularity in favor of end-to-end kernel co-design. FP8 accepts a higher payload rate for simpler production support.

0x0BA Selection Procedure

1. Establish the phase bottleneck.
   Is capacity, KV read traffic, activation GEMM compute, or kernel overhead limiting?

2. Fix the attention architecture and runtime.
   H_kv, d_h, RoPE convention, page size, supported backend, and hardware.

3. Choose K and V axes from observed distributions.
   Do not assume one axis is correct for both tensors.

4. Choose scale lifetime.
   offline fixed, per-head fixed, per-token online, token-group, or residual closure.

5. Account for every persistent byte.
   codes, qparams, residual, sparse state, pages, alignment, and mixed layers.

6. Implement a reference cache and transition tests.
   Append, fork, reorder, truncate, reuse, offload/transfer, and release.

7. Implement or select the consuming kernel.
   Prove scale/RoPE/packing parity before performance testing.

8. Evaluate quality across long contexts and tasks.
   Element MSE alone is not an attention-quality gate.

9. Benchmark prefill and decode separately.
   Include startup/calibration, memory capacity, throughput, latency, and energy.

0x0C Reference Implementation And Validation

0x0CA A Reference Quantized Cache

class QuantizedCacheReference:
    def __init__(self, schema, owner):
        self.schema = schema
        self.owner = owner                     # M5.1/M5.3 adapter

    @property
    def next_position(self):
        return self.owner.snapshot().committed_length

    def begin_append(self):
        return self.owner.begin_append(count=1)

    def prepare_layer(self, txn, layer, k_fp, v_fp, position):
        self.owner.require_open_txn(txn)        # correct owner/base/ABI/phase
        try:
            if (type(layer) is not int or not 0 <= layer < self.schema.num_layers
                    or position != txn.start or layer in txn.pending_layers):
                raise ValueError("invalid interval, layer, or duplicate preparation")
            # Clone/COW everything mutable, not just the code arrays.
            private = txn.base.layer_handles[layer].clone_private()
            private.append_reference(k_fp, v_fp, position)
            private.validate_complete(self.schema, start=txn.start, count=1)
            txn.pending_layers[layer] = private.freeze()
        except Exception:
            self.owner.abort_append(txn)       # discard only private state
            raise

    def publish(self, txn):
        self.owner.require_open_txn(txn)
        if set(txn.pending_layers) != set(range(self.schema.num_layers)):
            raise ValueError("all required layers must be prepared")
        # Owner revalidates every segment and conditionally publishes ONE
        # successor with committed_length=start+1 inside that same operation.
        # It rejects any stale generation, cancellation, truncation or ABI.
        return self.owner.publish_append(txn)

    def decode_layer(self, layer, txn=None):
        if txn is None:
            state = self.owner.snapshot().layer_handles[layer]
        else:
            self.owner.require_open_txn(txn)
            state = txn.pending_layers[layer]  # visible only to this forward
        k, v, positions = state.decode_all_reference()
        if self.schema.key_stage == "pre_rope":
            k = rope(k, positions, self.schema.rope)
        return k, v

    def attention(self, layer, q, q_position, mask, txn=None):
        k, v = self.decode_layer(layer, txn)
        q = rope_if_required(q, q_position, self.schema.rope)
        scores = (q @ k.T) / sqrt(k.shape[-1])
        return softmax(scores + mask) @ v

This is an orchestration reference with declared adapters, not a production concurrency implementation. clone_private must isolate codes, qparams, residual/sparse rows, validity, positions and mappings; freeze makes the prepared representation immutable. validate_complete checks all schema-required segments and their ABI/interval. The owner supplies immutable snapshots, authoritative transaction identity, fencing, complete-successor construction, atomic conditional publication, durable/idempotent receipts and quiescent reclamation as specified in M5.1/M5.3. A real implementation must test that owner boundary—not substitute a sequential dictionary assignment and call it lock-free publication.

For one token: begin a transaction, prepare each layer, use its private view for current-forward attention, then publish once after every required layer is complete. On failure, abort and start a fresh transaction from the still-current snapshot; never reuse a failed transaction. Readers see either the entire predecessor or entire successor. No public layer or frontier changes during preparation. This reference may materialize decoded K/V; the optimized attention kernel should not. Its purpose is to separate representation correctness from fused-kernel correctness.

0x0CB Required Unit And Property Tests

quantizer tests:
    endpoint, zero, clipping, zero-point, all-zero row, NaN/Inf policy

axis tests:
    token/head/channel/group qparams broadcast to intended dimensions

attention tests:
    reference decoded attention ~= fused quantized attention
    shared softmax across quantized and residual K regions

RoPE tests:
    pre-RoPE decode+rotate ~= expected post-RoPE row
    paired SmoothAttention transform preserves FP logits

state tests:
    two-token append advances the frontier only with complete publication
    reader between layer preparations sees the entire predecessor
    failed V/metadata preparation leaves K and every public layer unchanged
    duplicate/missing layer, wrong interval/ABI and stale generation reject
    cancellation/truncation fences a late prepared append; retry starts fresh
    fork copy-on-write for mutable residual
    beam reorder moves pages and qparams together
    truncate inside group/page cannot expose stale rows
    release closes publication now; reclaim only after quiescence/owner recheck
    reclamation frees owned code, qparam, sparse, and residual allocations

reuse tests:
    schema/calibration/layout mismatch rejects prefix reuse

serialization tests:
    packed cache transfer round-trip preserves schema and values

0x0CC Quality Evaluation

Measure at least:

• per-layer K/V reconstruction error by token, head, channel, and position;
• attention-logit and attention-output error, not only cache element MSE;
• perplexity over short and long contexts;
• retrieval/needle tasks across positions;
• generation quality and stability over long decode;
• sensitivity of first/sink tokens and recent tokens;
• calibration-distribution shift;
• mixed-layer and residual-window ablations.

All benchmark claims must report model, task, context length, cache policy, group/residual settings, runtime, backend, hardware, and baseline.

0x0CD Runtime Evaluation

capacity:
    measured bytes/request and maximum concurrent tokens

append path:
    scale reduction, outlier detection, packing, group closure latency

prefill:
    activation quantizer time, projection GEMM, cache write bandwidth

decode:
    attention latency versus context length
    cache payload and qparam read bytes
    dequant/unpack instruction cost
    throughput and inter-token latency under continuous batching

state operations:
    fork, reorder, truncate, prefix hit, offload, transfer, release

system:
    energy/token, allocator fragmentation, kernel fallback rate,
    startup/calibration time, and peak memory

0x0D Failure Lab

0x0E Boundaries And Handoffs

M8.6 produces:

activation quantizer axes and live scale equations
GEMM/attention factorization and zero-point corrections
K/V axis, scale lifetime, RoPE stage, and packing schema
payload + qparams + residual + page byte ledger
append/fork/reorder/truncate/reuse/release state contract
reference cache decoder and attention oracle
kernel/backend requirements by phase
calibration provenance and mixed-layer policy
quality and runtime validation matrix

It does not train through the quantizer. M8.7 receives the exact fake-quant insertion points, live scale rules, K/V persistence model, and failure modes needed to discuss LLM-QAT, straight-through gradients, and learned scales. M8.8 receives the serving-state baseline against which native binary/ternary architectures must still be evaluated.

It also does not declare a deployment winner. M8.9 receives the artifact/schema, supported kernel family, calibration identity, and reference tests. M8.10 receives the phase-specific measurements and rollback requirements needed to release one candidate.

0x0F Claim Status

0x10 Publish Gate

• Ephemeral activation tiles and persistent K/V rows have separate contracts.
• Per-token symmetric activation quantization is derived from row absmax.
• Per-token activation and per-output-channel weight scales are factored through GEMM.
• Affine zero-point correction terms are expanded.
• Producer fusion and prefill/decode operator differences are explicit.
• Q/K/V scale placement is derived by reduction axis.
• Value error is connected to attention weights rather than element MSE alone.
• Pre-RoPE and post-RoPE cache artifacts are distinguished.
• KV codes, qparams, residual, sparse, page, layout, and alignment bytes are counted.
• Stored bytes and cumulative decode-read traffic are separated.
• Online per-channel scale rewrite is derived.
• Append, fork, reorder, truncate, release, and exact prefix reuse include quantized state.
• KIVI K/V asymmetry, group closure, residual window, and mixed-cache attention are traced.
• KVQuant pre-RoPE K, offline channel calibration, nonuniform codebook, sparse path, and online V state are traced.
• KIVI and KVQuant alternatives are compared by scale lifetime and runtime state.
• IntactKV separates pivot-token evidence, offline exact K/V extraction, optional prefix calibration, cache identity, persistent bytes, and the fixed-prefix reuse assumption.
• IntactKV and PrefixQuant are compared by whether they preserve existing rows or insert new token state.
• PrefixQuant separates token-wise from channel-wise outliers, prefix discovery from learned compensation, and visible prompt from hidden cache state.
• Prefix token IDs, position policy, calibration exclusion, cache identity, context accounting, and source-reported figure scope are explicit.
• QServe SmoothAttention equivalence and RoPE-pair constraint are derived.
• QServe activation fusion and KV4 kernel roofline caveat are explicit.
• FP8 runtime behavior is bound to vLLM v0.25.0 and calibration-aware.
• A reference decoder, transition test suite, quality gates, and runtime gates are included.
• M8.7 retains training through quantizers, M8.8 retains native low-bit design, and M8.9/M8.10 retain integration and release decisions.

0xFF References

Local Series And Transformer System Sources

• M5.1 - Autoregressive Serving State - logical K/V lifecycle, append semantics, scheduler pressure, and base cache byte ledger. [TRANSFORMER SYSTEM]
• M5.2 - Continuous Batching and Scheduler Pressure - iteration work selection, token/sequence budgets, K/V-aware admission, and preemption policy. [TRANSFORMER SYSTEM]
• M5.3 - KV Layout and Exact Reuse - block/page allocation, fragmentation, physical cache state, layout compatibility, prefix reuse identity, and exactness requirements. [TRANSFORMER SYSTEM]
• M5.7 - MQA and GQA - architecture-level reduction of retained K/V heads. [TRANSFORMER SYSTEM]
• M5.8 - MLA Latent Cache and Absorbed Kernels - latent representation and MLA cache execution. [TRANSFORMER SYSTEM]
• M8.1 - Quantized Numbers and the Kernel Contract - affine quantization, zero-points, scale axes, packing, accumulation, and kernel ABI. [TRANSFORMER SYSTEM]
• M8.2 - Calibration, Sensitivity, and Precision Maps - observers, calibration coverage, sensitivity, and mixed precision. [TRANSFORMER SYSTEM]
• M8.3 - Outlier Geometry and Equivalent Transforms - SmoothQuant, decomposition, rotations, and function-preserving transforms. [TRANSFORMER SYSTEM]
• M8.4 - Scalar Weight-Only PTQ - ordinary scalar weight-only conversion and packed W4A16 runtime. [TRANSFORMER SYSTEM]
• M8.5 - Extreme Weight Compression - nonuniform LUTs, sparse exceptions, structured decoders, and effective-bit accounting. [TRANSFORMER SYSTEM]
• Transformer Series (35) - Fundamentals of Large Model Quantization - local W+A, token-wise activation, KV-cache, phase, and runtime taxonomy. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - local SmoothQuant, W8A8, LLM-QAT, KV quantization, and hardware-granularity treatment. [SOURCE-LOCAL]

Primary Papers And Official Documentation

• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - offline migration of activation-channel difficulty into weights for W8A8 deployment. [PAPER]
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers - group-wise weights, token-wise activations, quantization fusion, and layer-wise distillation. [PAPER]
• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale - mixed-precision decomposition for activation outlier dimensions. [PAPER]
• KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache - K-per-channel/V-per-token asymmetry, token grouping, residual cache, and fused mixed-precision attention. [PAPER]
• Official KIVI Repository - reference implementation, cache state, and kernels. [OFFICIAL CODE]
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization - pre-RoPE per-channel keys, online per-token values, sensitivity-weighted nonuniform datatypes, dense-and-sparse state, and custom kernels. [PAPER]
• Official KVQuant Repository - calibration, packing, sparse state, and CUDA implementation context. [OFFICIAL CODE]
• IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact - pivot-token attention sinks, offline full-precision prefix K/V extraction, optional K/V calibration, and mixed-prefix quantized inference. [PAPER, SOURCE-REPORTED RESULTS]
• Official IntactKV Repository - extraction, calibration, and integration context. [OFFICIAL CODE]
• PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantization - token-wise outlier isolation, prefixed K/V state, block-wise compensation, W4A4KV4/W4A8KV4 experiments, and source Figure 2. [PAPER, FIGURE 2 CC BY 4.0]
• Official PrefixQuant Repository - calibration, prefix-token selection, block-wise optimization, and implementation context. [OFFICIAL CODE]
• QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving - QoQ, per-token INT8 activations, per-head INT4 K/V, SmoothAttention, W4A8 kernels, and KV4 fused attention. [PAPER]
• Official QServe Repository - quantization and serving implementation context. [OFFICIAL CODE]
• vLLM v0.25.0 Quantized KV Cache Documentation - FP8 cache formats, scale shapes, calibration paths, backend constraints, and selective layer skipping at this documentation version, retrieved 2026-09-29. [OFFICIAL DOCS, VERSION-SPECIFIC]