M8.7 - Quantization-Aware Training and Low-Bit Optimization

M8.7 - Quantization-Aware Training and Low-Bit Optimization

0x00 Why Post-Training Repair Eventually Runs Out

Post-training quantization treats the pretrained model as fixed. It can choose ranges, transform coordinates, compensate local errors, learn codebooks, and reconstruct blocks, but it begins after the model has already organized its computation for floating-point arithmetic. At sufficiently low precision, that restriction becomes the main problem.

Quantization-aware training changes the direction of adaptation. Instead of asking a converter to approximate the final floating-point parameters, it exposes the model to the target numerical perturbation while parameters can still move. The optimizer can relocate values away from clipping boundaries, reshape activation ranges, change how information is distributed across channels, and recover task behavior under the simulated deployment grid.

The A34 workflow introduces the central PTQ/QAT distinction. A36 then uses LLM-QAT and native low-bit methods to show that training can cover weights, activations, and K/V simultaneously. M8.7 expands the background that those source sections assume: a QAT model has continuous training state, a discrete or fake-discrete forward path, surrogate gradients, scheduled observer state, and an export path that must reproduce the trained arithmetic.

0x00A QAT Is Not “Store The Training Model In INT4”

Optimizers need high-precision latent state because many useful updates are smaller than one low-bit quantization step. If a trainable parameter itself were rounded destructively after every update, sub-threshold changes could disappear. Most QAT therefore keeps a master weight in FP32, BF16, or FP16 and derives a fake-quantized reconstruction for the forward pass.

That creates two simultaneous models:

optimization state:
    continuous master weights, gradients, optimizer moments,
    observer state, and possibly learned quantizer parameters

forward numerical state:
    values clamped and rounded onto the target grid,
    represented in a floating compute dtype during training

The training memory can be larger than ordinary fine-tuning because fake quantization does not automatically pack weights or remove optimizer state. QAT is a quality-recovery method first; deployment savings only appear after conversion and lowering.

0x00B Discrete Arithmetic Breaks Ordinary Differentiation

Rounding is piecewise constant, and clipping has zero gradient outside its active interval. The true derivative therefore provides little or no signal for moving a latent value across code boundaries. QAT introduces a surrogate gradient, commonly a straight-through estimator, that deliberately uses a backward rule different from the forward operation.

This mismatch is not an implementation nuisance. It defines the optimization algorithm. Different methods decide:

• whether gradients pass through rounding as identity;
• whether saturated values receive zero, clipped, or custom gradients;
• whether scale and zero-point are observed, fixed, or learned;
• how scale gradients are normalized relative to tensor size and code range;
• when observers and fake quantizers turn on or freeze;
• whether a teacher supplies logits or hidden-state targets.

LSQ, for example, learns the quantization step size with a normalized surrogate gradient. PACT learns an activation clipping boundary. LLM-QAT combines synthetic teacher data and distillation with weight, activation, and cache fake quantization. They are all QAT, but their trainable state and backward fields differ.

0x00C Training Earlier Buys Adaptation At A Real Systems Cost

QAT can recover quality that PTQ cannot, but it requires more than adding a fake-quant call:

Efficient QAT variants reduce one of these costs by training low-rank updates, processing blocks sequentially, or restricting which parameters move. They should be described as different optimization problems, not as free approximations to full-model QAT.

0x00D Training Must See The Arithmetic That Serving Will Use

M8.1 defined the quantized numerical object. M8.2 selected ranges and precision maps from calibration evidence. M8.3, M8.4, and M8.5 changed weight geometry and reconstruction. M8.6 carried quantization into request-dependent activations and persistent K/V state.

All of those methods can start from a fixed pretrained model. Quantization-aware training changes the optimization problem itself.

The owner question is:

How can a floating-point optimizer train parameters for a discontinuous low-bit forward path, and how does that training graph become the exact packed arithmetic used by inference?

That question has four graphs, not one:

1. the pretrained floating-point graph that initializes the candidate and often acts as teacher;
2. the QAT graph that stores trainable high-precision state but inserts fake quantization in the forward path;
3. the converted graph whose weights, scales, zero-points, and Q/DQ boundaries represent the target format;
4. the lowered runtime graph that packs codes and fuses dequantization into supported kernels.

Confusing any two of them creates a false deployment claim. A BF16 fake-quant training loop is not an INT4 kernel. A low training loss is not proof that convert preserved the trained scale axes. A packed checkpoint is not useful if the loader silently falls back to a high-precision operator.

This article owns training under an already specified low-bit contract:

The local A34-A36 articles provide the QAT/PTQ distinction, pseudo-quantization workflow, LLM-QAT data-free distillation, and simultaneous W/A/KV spine. The STE, LSQ, PACT, LLM-QAT, LR-QAT, and EfficientQAT mechanisms are checked against their primary papers. The inspected TorchAO and TensorRT documentation is dated, version-specific implementation context, not an evergreen support claim.

0x01 The Four-Graph Contract

0x01A State Inventory

Let one linear layer use:

QAT can carry several representations of the same logical weight:

The QAT optimizer updates high-precision state because small updates must accumulate even when they do not immediately cross a low-bit bin boundary. The forward pass consumes $\widehat W$, so the loss still sees the grid and clipping error.

0x01B Fake Quantization Does Not Reduce Training Storage Automatically

For affine quantization with integer interval $[q_{min},q_{max}]$:

FQ returns a floating-point tensor whose values lie on the deployment grid. It simulates the numerical loss of quantize then dequantize, but it normally does not store the tensor in the target packed representation.

Renderer-safe form:

q = clamp(round(x / s) + z, qmin, qmax)
x_fq = s * (q - z)

If the runtime uses a different rounding rule, grouping axis, clipping interval, zero-point convention, or code range, the training graph simulated the wrong machine.

0x01C Fake-Quant, Converted, And Lowered Parity

The required invariant is numerical, not syntactic:

Exact equality may be impossible across accumulator and epilogue dtypes, but every tolerated difference must be named. A useful parity test isolates:

same input tokens
same quantization parameters
same logical codes after unpack
same scale and zero-point axes
same clipping and rounding semantics
same accumulator width
same bias / residual / activation epilogue order
bounded output difference at each converted operator

0x02 Why Ordinary Backpropagation Fails At Rounding

0x02A The True Derivative

Ignore clipping and zero-point temporarily. A symmetric fake quantizer is:

For fixed $s$, this is a staircase. Between two thresholds, changing $x$ does not change the output, so:

almost everywhere. At a threshold the derivative is undefined. Feeding that true derivative into gradient descent would send no useful gradient through almost every fake-quantized weight or activation.

0x02B The Straight-Through Substitution

The straight-through estimator does not differentiate rounding correctly. It replaces the unusable derivative with a chosen surrogate. The common clipped identity rule is:

With upstream gradient $g=\partial\mathcal L/\partial\widehat x$:

Here the represented endpoints are inclusive: exactly $x/s+z=q_{min}$ or $q_{max}$ passes the upstream gradient. Immediately outside, it is zero. This is a chosen surrogate at the endpoints, not a true derivative; recomputation and distributed implementations must use the same convention. An identity-everywhere STE instead passes $g$ even for clipped values. That can help a clipped parameter move back toward range, but it also hides the fact that the forward operator is saturated. The backward rule is therefore part of the method, not an implementation footnote.

0x02C Gradient Mismatch Is The Central Approximation

Let the true low-bit objective be:

The optimizer follows a surrogate field:

There need not exist a smooth scalar objective whose exact gradient equals this field. QAT works empirically because the surrogate often moves high-precision parameters toward regions whose quantized reconstructions reduce task loss. It is not ordinary gradient descent on the discrete model.

Consequences:

• learning rates that worked for floating-point fine-tuning may be unstable;
• clipping masks can create dead parameters;
• scale updates can dominate weight updates because one scale affects many values;
• gradient checkpoint recomputation must reproduce the same qparams and stochastic choices;
• a numerically closer fake-quant reconstruction is not guaranteed to improve task loss.

0x02D A Minimal Autograd Contract

def check_affine_qparams(scale, zero, qmin, qmax):
    if type(qmin) is not int or type(qmax) is not int or qmin >= qmax:
        raise ValueError("expected ordered integer code endpoints")
    if not torch.isfinite(scale).all() or not (scale > 0).all():
        raise ValueError("expected positive finite effective scale")
    if (not torch.isfinite(zero).all() or not (zero == zero.round()).all()
            or not ((zero >= qmin) & (zero <= qmax)).all()):
        raise ValueError("expected integral in-range zero point")

class FakeQuantSTE(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, scale, zero, qmin, qmax):
        check_affine_qparams(scale, zero, qmin, qmax)
        normalized = x / scale
        u = normalized + zero
        if not torch.isfinite(u).all():
            raise ValueError("nonfinite input or unsupported quotient arithmetic")
        # Ties-to-even applies BEFORE adding the integer zero point.
        q = (torch.round(normalized) + zero).clamp(qmin, qmax)
        ctx.save_for_backward(u)
        ctx.qmin = qmin
        ctx.qmax = qmax
        return scale * (q - zero)

    @staticmethod
    def backward(ctx, grad_output):
        (u,) = ctx.saved_tensors
        in_range = (u >= ctx.qmin) & (u <= ctx.qmax)
        grad_x = grad_output * in_range
        return grad_x, None, None, None, None

This sketch leaves scale and zero-point fixed and returns no gradients for them. Learning them requires a separately derived surrogate; returning the activation gradient as the scale gradient is not valid. The effective scale has already been rounded to its declared storage dtype by the qparam policy. Broadcasting/axis validation belongs to that policy. The arithmetic reference uses float64 inputs; a training kernel’s compute dtype and tolerance remain separately declared.

The order of operations matters even for integer $z$: with ties-to-even, $x/s=0.5$ and $z=1$ give code $\operatorname{round}(0.5)+1=1$, whereas $\operatorname{round}(1.5)=2$. PyTorch 2.8’s rounding contract is ties-to-even. Moving the addition inside rounding silently changes the affine grid at half ties.

0x03 Learning The Quantizer, Not Only The Weight

0x03A LSQ Step-Size Gradient

Use a signed symmetric quantizer with positive endpoint $Q_P$ and negative magnitude $Q_N$:

Inside the clipping interval, LSQ applies the STE only to rounding. Since $v=x/s$:

Then:

Outside the interval, the clipped code is constant with respect to $v$, but the reconstructed endpoint still scales with $s$. Therefore:

The middle term is sensitive to position inside a bin. A value near one side of a rounding transition contributes differently from a value near the other side.

0x03B Why The Scale Gradient Needs Normalization

One shared scale receives contributions from every value in its group:

Its raw magnitude grows with group size $n$ and with the quantizer endpoint. LSQ uses a heuristic gradient multiplier:

The training system applies:

This is not a derivative identity. It is an optimization preconditioner intended to keep relative scale updates in a useful range compared with weight updates. Changing granularity changes $n$, so copying a learning rate while changing per-tensor scales to per-group scales changes the optimization dynamics.

Renderer-safe form:

d(x_hat)/ds =
    -Q_N                     if x/s < -Q_N
    round(x/s) - x/s         if -Q_N <= x/s <= Q_P
    Q_P                      if x/s > Q_P

scale-gradient multiplier = 1 / sqrt(number_of_values * Q_P)

0x03C PACT Learns The Activation Boundary

For a nonnegative activation, PACT learns a clipping parameter $\alpha>0$:

An unsigned $b$-bit quantizer then uses:

Ignoring the dependence of the internal rounding grid for a moment, the clipping path gives:

Reducing $\alpha$ makes the grid finer but clips more mass. Increasing it preserves outliers but enlarges the step. The task loss chooses that tradeoff, often with a regularizer discouraging unbounded $\alpha$.

PACT is naturally aligned with ReLU-like nonnegative activations. Transformer residual streams, normalization outputs, and many projection inputs are signed, so an LLM implementation needs a signed extension, separate positive/negative bounds, or a different learned quantizer. Importing PACT as a name without checking tensor support is not a method.

0x03D Trainable Zero-Points And Discrete State

An affine zero-point is often constrained to integer range:

Training $z_{real}$ needs another surrogate through rounding. It also adds optimizer state and can make conversion ambiguous if training treats $z$ as floating but the artifact stores an integer. Some methods therefore train scales while deriving or freezing zero-points. EfficientQAT reports that its end-to-end quantizer-parameter phase commonly trains only $s$ to avoid the additional state associated with a full-precision trainable $z$; that is a method choice, not a universal rule.

0x03E Quantizer Design Space

0x04 Tensor-Level Insertion Sites

0x04A A Fake-Quantized Linear Layer

Suppose weights are grouped along $K$ with group size $G$ and activations use one scale per token row:

The QAT forward is:

Training usually performs the final matrix multiply in BF16/FP16/FP32. Deployment may use integer accumulation and a fused scale epilogue. The fake-quant values must represent the same grid even though the training compute dtype differs.

class QATLinear(nn.Module):
    def __init__(self, linear, weight_fq, activation_fq):
        super().__init__()
        self.weight = nn.Parameter(linear.weight.detach().clone())
        self.bias = (None if linear.bias is None else
                     nn.Parameter(linear.bias.detach().clone()))
        self.weight_fq = weight_fq
        self.activation_fq = activation_fq

    def forward(self, x):
        x_fq = self.activation_fq(x)       # [B, T, K]
        w_fq = self.weight_fq(self.weight) # [N, K]
        return F.linear(x_fq, w_fq, self.bias)

0x04B Transformer Block Placement

For a decoder block, candidates include:

residual h [B,T,d]
  -> RMSNorm
  -> activation FQ
  -> q_proj / k_proj / v_proj weight FQ
  -> optional Q/K/V FQ
  -> attention, with optional K/V-cache FQ semantics
  -> o_proj activation + weight FQ
  -> residual add in chosen high precision
  -> RMSNorm
  -> activation FQ
  -> gate/up projection weight FQ
  -> SiLU * gate product
  -> down projection activation + weight FQ
  -> residual add

Quantizing every visible edge is not automatically faithful. A target W4A16 kernel does not justify activation fake quantization. A W4A8 kernel does. Softmax, normalization reductions, residual accumulation, embeddings, and LM head may stay at higher precision depending on the actual backend contract.

0x04C The K/V Training-Serving Mismatch

During teacher-forced training, all sequence positions are commonly available in one tensor:

During autoregressive serving, K/V are appended one row at a time and retained. A QAT implementation that fake-quantizes the full tensor with one sequence-wide scale can train a format that cannot be reproduced online without revisiting history.

There are two distinct meanings of a token row here. The pinned LLM-QAT attention implementation applies K/V fake quantization to projected tensors of shape $[B,T,Hd_h]$, before head reshape and before RoPE. Its symmetric quantizer reduces that full last dimension: one scale per token across heads, not one per token per head. It also has its own epsilon arithmetic and clipping-gradient rule. Those source choices are not identical to the following local reference policy.

For a teaching per-token/per-head variant, choose independent rows $K_{b,h,t,:}$ and $V_{b,h,t,:}$. With $a^K_{b,h,t}=\max_d|K_{b,h,t,d}|$, the stored effective scale is:

Consume M8.6’s symmetric-row reference policy: narrow signed codes for integer bits 2–8; all-zero rows use scale 1 and code 0; FP16 or FP32 scale storage is rounded before encoding. Reject nonfinite, nonpositive, or subnormal stored scales (an explicit FTZ-safe supported domain), rather than inventing a decimal epsilon floor. BF16/FP8 scale storage and alternate floors are not covered by this reference. The tensor can be vectorized across $t$, but its axis, insertion edge and state lifetime must still match serving.

def fake_quant_observed_rows(x, bits, qparam_dtype, detached_scale_policy):
    # Adapter: flatten leading axes; run M8.6's row policy on detached x;
    # restore [..., 1] scales and re-read their stored dtype for arithmetic.
    # Reference arithmetic is float64; this is not a GPU observer kernel.
    scale = detached_scale_policy(x.detach(), bits, qparam_dtype).detach()
    qmax = 2 ** (bits - 1) - 1
    return FakeQuantSTE.apply(x, scale, torch.zeros_like(scale), -qmax, qmax)

def fake_quant_kv_per_token_head(k, v, bits, qparam_dtype, detached_scale_policy):
    # Teaching variant only: k, v are [B, Hkv, T, dh], reduce dh.
    if k.ndim != 4 or v.shape != k.shape:
        raise ValueError("expected matching [B, Hkv, T, dh] tensors")
    k_fq = fake_quant_observed_rows(k, bits, qparam_dtype, detached_scale_policy)
    v_fq = fake_quant_observed_rows(v, bits, qparam_dtype, detached_scale_policy)
    return k_fq, v_fq

The observer updates a detached scale; the custom STE supplies the input gradient. Literal round(x / scale) * scale does not supply that gradient: with a detached scale its ordinary input gradient is zero almost everywhere; with a live max-derived scale it instead leaks a different scale path. Learned scales require the separately defined LSQ/other surrogate, not this observer adapter.

For an axis-faithful adaptation to LLM-QAT, apply fake_quant_observed_rows to $[B,T,Hd_h]$, then reshape into heads, and only then apply RoPE to K. It still is not a reproduction of the original epsilon/backward rule. With row $[0.2,1,2,10]$ split into two heads, per-head and full-projection reductions choose different grids. KIVI’s K-per-channel grouped history, KVQuant’s pre-RoPE calibrated K channels, and QServe’s token/head metadata within pages from M8.6 also require their own QAT state and decoder contracts.

0x04D Residual And Nonlinear Boundaries

The quantizer placement decides where error is injected:

The article’s later export test must preserve these graph edges exactly.

0x05 Observer And Fake-Quant Schedules

0x05A Three Independent Switches

A practical fake-quant module has at least three logically separate controls:

observer_enabled:
    collect statistics and update candidate qparams

fake_quant_enabled:
    apply quantize-dequantize noise in the forward pass

qparam_trainable:
    allow the optimizer to update scale/clipping/zero-point parameters

These controls create different phases. Updating an observer is not the same as learning a scale. Applying fake quantization while an observer rewrites the range every batch is not the same objective as training against frozen deployment qparams.

0x05B A Robust Lifecycle

One useful schedule is:

1. initialize from the pretrained model and a deployment-feasible precision map;
2. run representative batches with observers enabled and fake quantization disabled or gradually introduced;
3. enable fake quantization so the model sees the grid;
4. freeze observer-derived qparams once the ranges stabilize;
5. continue optimization against the fixed deployment grid;
6. optionally learn selected scales or clipping parameters under explicit constraints;
7. disable training-only state, convert, pack, and validate the runtime path.

The ordering is a design choice. Some methods learn step sizes from the beginning. Others initialize from PTQ and then train. Some progressively lower precision. The invariant is that the final training phase must expose the model to the qparams and operator placements that conversion will actually retain.

0x05C Moving Statistics And Freeze Drift

An exponential moving maximum might update:

The derived scale is $s_t=m_t/q_{max}$. While the observer is enabled, the objective itself changes because the quantization grid moves with every batch. After freezing at step $t_f$:

The post-freeze saturation rate is therefore a required metric:

If $r_{sat}$ rises sharply after freeze, either the observer saw the wrong distribution, training moved the activation distribution, or the chosen static scale is incompatible with the target workload.

0x05D Progressive Bit Schedules

A progressive schedule can move from an easier grid to the target grid:

BF16 initialization
  -> W8/A8 fake quant
  -> W4/A8 fake quant
  -> target W4/A6 or W4/A8/KV4

This can reduce the shock of changing many code assignments at once, but it also changes the objective multiple times and consumes more training. Direct-to-target QAT is simpler and can work when initialization is close enough. The decision should be an ablation, not folklore.

0x05E Checkpoint State Is Larger Than Model Weights

A resumable QAT checkpoint must identify:

trainable/master weights
optimizer and scheduler state
observer buffers and sample counts
fake-quant enable flags
frozen or learned qparams
bit width, code range, symmetry, granularity, and axes
clipping / rounding / STE policy
excluded modules and insertion graph
teacher identity and distillation temperature
data-generation and data-mix manifest
framework and quantizer implementation version

Saving only state_dict() is insufficient if graph transformation or quantizer configuration lives outside the module state.

0x06 Objectives: Task Labels, Teacher Distributions, And Quantization Noise

0x06A Ordinary Next-Token Loss

For student logits $z^S_{t}\in\mathbb R^{V}$ and next token $y_t$:

This trains the quantized student toward one observed label per position. It discards the teacher’s probability mass over alternative tokens.

0x06B Logit Distillation

M6.10 owns generic teacher/student objectives, temperature, teacher identity, and synthetic/online/cached supervision. M8.7 consumes that contract for a student whose W/A/KV forward contains fake quantization and whose final state must pass conversion and runtime parity.

At temperature $\tau$:

A common distillation term is:

The teacher distribution is detached: there are not two differentiated teacher/student paths. For one token, the derivative of the unscaled KL with respect to student logit $z^S_i$ is $(p^S_i-p^T_i)/\tau$. At high temperature and fixed logits, the probability difference is itself approximately proportional to $1/\tau$, giving the familiar approximate $1/\tau^2$ shrinkage. The $\tau^2$ multiplier compensates that asymptotic effect; it is not an exact temperature-invariance guarantee.

The combined objective can be:

where $\mathcal R_q$ can regularize clipping bounds, scale drift, or another method-specific quantizer parameter. There is no universal coefficient choice.

0x06C Why Logits Can Be Better Than Sampled Labels

If a teacher samples token $y_t$ from its distribution, that token is one noisy draw. Training only on $y_t$ treats it as probability one. Logit distillation instead preserves the teacher’s ranking and relative uncertainty over all vocabulary entries.

This matters for self-generated data: the sampled continuation creates a plausible context, while the full teacher distribution provides a denser target at each position.

Hidden-state or attention-map losses are alternatives, but layerwise alignment can overconstrain a student whose internal representation must move to become robust to quantization. LLM-QAT’s ablation reported logit distillation as preferable in its evaluated setup and did not benefit from adding hidden or attention distillation. That is source-reported evidence, not a theorem against intermediate losses.

0x07 LLM-QAT: Data-Free Distillation With W/A/KV Noise

0x07A Why “Data-Free” Still Uses Data

LLM-QAT does not train with zero sequences. It avoids the original pretraining corpus by asking the pretrained model to generate a synthetic corpus.

Its generation process is:

1. choose an initial token;
2. run the frozen pretrained teacher autoregressively;
3. use deterministic top-1 for the first few positions in the paper’s hybrid strategy;
4. sample later positions from the teacher distribution for diversity;
5. stop at EOS or maximum length;
6. run teacher and fake-quant student on the generated sequence and distill teacher logits.

The paper reports selecting top-1 for the first 3-5 tokens and sampling afterward. Those values are source-reported settings, not a default for every model.

0x07B Generation And Training Trace

The published LLM-QAT objective, Eq. 4, is teacher-soft-label cross-entropy with ordinary softmax, equivalent to $\tau=1$, not a reported arbitrary-temperature KL recipe. At $\tau=1$, soft cross-entropy and $D_{KL}(p^T\|p^S)$ differ by the fixed teacher entropy and have the same student gradient. The trace below is a generic masked QAT-distillation adaptation: $\tau=1$ preserves that source objective per position; nonunit temperature changes the method. Its valid-token mean, mask, coefficient, and optimizer cadence are explicit local choices, not a claim to reproduce the paper’s full trainer.

@torch.no_grad()
def synthesize(teacher, start_token, max_len, greedy_prefix):
    ids = [start_token]
    for t in range(max_len - 1):
        logits = teacher(ids)[-1]
        probs = softmax(logits)
        if t < greedy_prefix:
            next_id = probs.argmax()
        else:
            next_id = multinomial(probs)
        ids.append(next_id)
        if next_id == EOS:
            break
    return ids

def masked_qat_distillation_loss(teacher_logits, student_logits, causal_valid, tau=1.0):
    # causal_valid: bool [B, T-1], selecting logits that predict a valid next
    # token in the same sequence; exclude padding and packed boundaries.
    if not math.isfinite(tau) or tau <= 0:
        raise ValueError("temperature must be positive and finite")
    if (teacher_logits.shape != student_logits.shape or student_logits.ndim != 3
            or student_logits.shape[-1] == 0
            or causal_valid.shape != student_logits[:, :-1, 0].shape
            or causal_valid.dtype != torch.bool or not causal_valid.any()):
        raise ValueError("expected matching logits and nonempty causal token mask")
    if not torch.isfinite(teacher_logits).all() or not torch.isfinite(student_logits).all():
        raise ValueError("expected finite teacher and student logits")
    tau_squared = tau * tau
    if not math.isfinite(tau_squared) or tau_squared <= 0:
        raise ValueError("temperature square is outside supported arithmetic")
    # Re-read temperature constants in the actual operand dtypes; a positive
    # Python float may underflow to zero or overflow when cast to FP16/FP32.
    teacher_tau = teacher_logits.new_tensor(tau)
    student_tau = student_logits.new_tensor(tau)
    multiplier = student_logits.new_tensor(tau_squared)
    for value in (teacher_tau, student_tau, multiplier):
        if not torch.isfinite(value).all() or not (value > 0).all():
            raise ValueError("unrepresentable dtype temperature or multiplier")
    teacher_scaled = teacher_logits.detach()[:, :-1, :] / teacher_tau
    student_scaled = student_logits[:, :-1, :] / student_tau
    if not torch.isfinite(teacher_scaled).all() or not torch.isfinite(student_scaled).all():
        raise ValueError("temperature-scaled logits exceed arithmetic domain")
    teacher_prob = softmax(teacher_scaled, dim=-1)
    student_log_prob = log_softmax(student_scaled, dim=-1)
    per_token = -(teacher_prob * student_log_prob).sum(dim=-1)
    unscaled_loss = per_token[causal_valid].mean()
    loss = multiplier * unscaled_loss
    if (not torch.isfinite(unscaled_loss).all() or not torch.isfinite(loss).all()
            or ((unscaled_loss > 0) & (loss == 0))):
        raise ValueError("loss overflows or positive loss underflows to zero")
    return loss

def qat_distillation_step(teacher, student_qat, token_batch, causal_valid, optimizer, tau=1.0):
    teacher.eval()
    optimizer.zero_grad(set_to_none=True)  # one non-accumulating optimizer step
    with torch.no_grad():
        teacher_logits = teacher(token_batch)

    # Student linear weights/activations and selected K/V tensors pass
    # through fake quantizers whose axes match the deployment proposal.
    student_logits = student_qat(token_batch)

    loss = masked_qat_distillation_loss(teacher_logits, student_logits, causal_valid, tau)
    loss.backward()
    optimizer.step()
    return loss.detach()

The tensor pseudocode assumes math, torch and last-axis softmax/log-softmax adapters are imported; model calls here return floating logits. Temperature denominators and the squared multiplier must remain positive and finite in their operand dtypes; scaled logits and the loss must stay in the supported arithmetic domain. These guards reject rather than silently clamp, promote precision, or turn a positive loss into zero. The caller configures student training/dropout/observer mode and supplies the causal mask from real token/segment validity, including left padding. An accumulation loop must move zero_grad/step to its declared boundary and rescale losses accordingly. The teacher and student must not share trainable parameter storage. Logits can be generated online or cached. Caching reduces teacher compute during QAT but creates a large storage and provenance artifact. Online generation preserves flexibility but increases wall time and requires deterministic sampling metadata for reproducibility.

0x07C The Paper’s Quantizer Placement

LLM-QAT uses the following central choices:

The full training tensor is an implementation convenience. Each projected token can be quantized without rewriting earlier tokens; this does not imply a per-head scale or that post-RoPE cached K remains on the same scalar grid. A serving format must preserve the selected insertion edge, metadata sharing and decoder/rotation order, or establish a new QAT and conversion contract. The source attribution here is bound to commit 5ef1289730de080a4781f21ef27105b6c01b116a.

0x07D Attention With Fake-Quantized K/V

For one head, let $\widehat K$ mean the keys actually supplied to attention. In the pinned LLM-QAT layout these are RoPE(head_reshape(FQ(K_projected))), not FQ(RoPE(K)); $\widehat V$ is head-reshaped fake-quantized V:

The student gradient passes through both uses of fake quantization (and through the intervening reshape/rotation where present). For a direct K/V FQ placement, the schematic chain rules are:

This lets upstream projection weights adapt to the future cache grid. It does not by itself implement page packing, residual windows, fork/truncate semantics, or fused low-bit attention. M8.6’s cache ABI remains the serving contract.

0x07E Data Distribution Is Still A Risk

Self-generated text has failure modes:

• repetitive or low-diversity generations can narrow the student;
• random initial tokens can produce unnatural contexts;
• one teacher sampling policy can underrepresent code, multilingual text, tools, safety behavior, or long-context structures;
• synthetic maximum length can leave long-context K/V quantization undertrained;
• teacher errors and biases become training targets;
• generated data may not reproduce later instruction/RL alignment distributions.

The correct claim is that LLM-QAT removes dependence on the unavailable original corpus in its method. It does not remove the need to design and audit the QAT data distribution.

0x08 Training-State And Memory Ledgers

0x08A Full-Model QAT State

Let $P$ be the number of trainable model parameters. A common mixed-precision AdamW arrangement can contain:

bytes, before activations, temporary fake-quant reconstructions, qparams, communication buffers, and allocator overhead. The subtotal is:

For $P=7\times10^9$ this illustrative unsharded subtotal is about 112 GB. Frameworks differ: some do not keep a separate FP32 master copy, some use lower-precision optimizer state, and ZeRO/FSDP shards portions across ranks. The formula is an inventory, not a universal measurement.

0x08B QAT Adds Live Tensors Even When Codes Are Conceptual

For a weight matrix, a naive forward may materialize:

master/trainable W
normalized W/s
rounded/clipped code-like float tensor
dequantized W_hat
STE masks or saved inputs
linear activations needed for backward
scale and zero-point tensors

Autograd may retain intermediate matrices the size of $W$. Custom kernels, recomputation, or carefully written autograd functions can avoid some copies. Simply representing fake codes as BF16 does not provide packed INT4 training memory.

0x08C Activation Checkpointing Trades Memory For Recompute

Checkpointing a block stores its inputs and recomputes the forward during backward. For QAT, recomputation must reuse equivalent:

qparams
observer state
fake-quant enable state
rounding randomness
dropout RNG state
teacher or target inputs

If an observer updates during both the original and recomputed forward, statistics advance twice. Observer mutation must be disabled, made idempotent, or excluded from checkpoint recomputation.

0x09 Reducing QAT Cost Without Pretending It Is The Same Algorithm

0x09A Full-Model QAT

The direct method updates every selected high-precision weight under fake-quant noise:

It offers the most trainable degrees of freedom but pays for gradients and optimizer state over the whole model. At LLM scale, that often requires FSDP/ZeRO, activation checkpointing, many GPUs, or a restricted sequence/batch budget.

0x09B Low-Rank QAT Must Put The Update Inside The Grid

Ordinary LoRA computes a separate high-precision path:

If that branch remains at inference, the result is adapter serving, not a single packed low-bit weight matrix. LR-QAT instead places a low-rank perturbation inside the quantization decision. A conceptual symmetric form is:

$\Phi$ maps the frozen pretrained state into the method’s grid-aligned/downcast domain. The exact LR-QAT paper includes variants for how that state is represented and initialized; the reusable invariant is that $BA$ changes the final integer codes before clipping rather than surviving as an independent BF16 branch.

After training, only $Q_{final}$ and deployment qparams are required. The low-rank matrices are optimization state and can disappear.

def lr_qat_weight(base_grid_state, A, B, scale, alpha, qmin, qmax):
    integer_domain_update = (alpha / A.shape[0]) * (B @ A)
    latent_codes = base_grid_state + integer_domain_update
    final_codes = round_ste(latent_codes).clamp(qmin, qmax)
    return scale * final_codes

The matrix product $BA$ can itself be large if materialized. LR-QAT uses checkpointing so backward recomputes this path rather than retaining all intermediate matrices. The paper also studies downcasting the frozen base using fixed-point or double-packed integer representations. Those choices trade recomputation and implementation complexity for lower memory.

0x09C Block-Wise All-Parameter Training

EfficientQAT separates local recovery from cheap global adjustment.

Its Block-AP stage trains a block’s inherent weights and quantization parameters against block reconstruction. For floating block input $X$ and reference output $F(X;W)$:

Only one block needs full trainable state at a time. The method can therefore expose more local degrees of freedom than a scale-only method without holding optimizer state for all model weights simultaneously.

The weakness is structural: independently reconstructed blocks do not optimize all cross-block interactions end to end.

0x09D End-To-End Quantizer-Parameter Training

EfficientQAT’s E2E-QP stage starts from block-trained quantized weights and trains quantization parameters end to end while keeping the low-bit codes/model state frozen. A simplified symmetric decoder is:

with trainable $s_l$ and fixed $Q_l$. Then:

This phase captures cross-block task interactions with far less optimizer state than full-model QAT. It cannot change code assignments when only $s$ is trainable, so its reachable solution set is narrower.

0x09E Method Comparison

M6.4 owns QLoRA’s frozen quantized base, adapter optimization, merge and serving-state choices. It is a negative control here: training with a quantized base does not automatically yield the single packed QAT artifact discussed above.

Paper-reported memory, accuracy, and speed results depend on model, data, bit width, granularity, and implementation. The table compares mechanisms, not benchmark winners.

0x0A Distributed QAT Is A State-Consistency Problem

0x0AA Which State Must Agree Across Ranks?

Data-parallel ranks must agree on model parameters after gradient synchronization. Quantizer state may also need synchronization:

If each rank freezes an activation observer from a different local sample distribution, conversion can depend on which checkpoint shard supplied the buffer.

0x0AB Tensor Parallelism Changes The Quantization Axis

Suppose a weight $W[N,K]$ is column-sharded across $p$ ranks:

A group of width $G$ is local only if group boundaries align with shard boundaries. If a group straddles ranks, its absmax or learned qparam needs a collective reduction. Changing from an unsharded training graph to a tensor-parallel export can silently change groups unless the original global coordinate is retained in the manifest.

For row-sharded output channels, per-output-channel weight scales are naturally local. For per-tensor scales, every rank must agree on one global statistic.

0x0AC Gradient Accumulation And Observer Cadence

With microbatch accumulation, model gradients are accumulated before the optimizer step. Observer statistics may update once per microbatch. Therefore one optimizer step can contain $A$ observer transitions:

Changing accumulation factor changes observer dynamics even when the effective model batch stays constant. A reproducible run logs observer updates in microbatches, not only optimizer steps.

0x0AD Determinism And Recomputation

Stochastic rounding or randomized data generation needs explicit RNG streams. Checkpoint recomputation must not draw a different rounding sample. A robust design derives random decisions from stable identifiers such as:

(global_step, microbatch_id, layer_id, tensor_role, element_block)

or stores the required random state. Otherwise the backward pass differentiates a different fake-quant realization from the original forward.

0x0AE Distributed Training Trace

for step, batch in enumerate(loader):
    set_qat_phase(model, schedule.phase(step))

    with deterministic_qat_rng(step):
        loss = qat_forward_and_loss(model, teacher, batch)
        loss = loss / grad_accum_steps
        loss.backward()

    if end_of_accumulation(step):
        all_reduce_model_and_qparam_gradients(model)
        clip_gradients_separately(model_weights, quantizer_parameters)
        optimizer.step()
        optimizer.zero_grad(set_to_none=True)

    if observer_sync_due(step):
        merge_observer_state(model, process_group)

    assert_same_qat_phase_across_ranks(model)

Separate gradient telemetry for model weights and qparams is useful because their scales and optimal learning rates differ.

0x0B Prepare, Train, Convert, Pack, Lower

0x0BA Prepare Is A Graph Transformation

prepare decides:

which modules are replaced or wrapped
where activation fake quantizers are inserted
weight/activation bit widths and code ranges
granularity and axis
observer or learned-qparam implementation
excluded modules
backward surrogate
state names needed for checkpointing

Two QAT runs with the same optimizer and data but different prepare transforms are different algorithms.

0x0BB Convert Is Not Merely Removing Fake Quantizers

Conversion must:

1. freeze the final qparams;
2. finalize codes from the complete method-specific state, or retain already fixed codes;
3. remove optimizer, observer, STE, and training-only auxiliary state;
4. emit Q/DQ semantics or a framework-specific quantized tensor;
5. preserve scale axes, zero-points, code ranges, and excluded layers;
6. attach enough schema for the target packer and loader;
7. reject configurations without a real runtime representation.

One universal finalize(master_weight) is insufficient. A grid-aware low-rank update belongs inside the final rounding operation; a fixed-code qparam phase must preserve its code tensor and learned effective scales. For example, base grid value $0.49$ plus low-rank update $0.02$ rounds to code 1; finalizing the base alone gives 0. Re-encoding an old master weight with a newly learned scale can likewise change codes that training deliberately froze.

This logical reference handles three explicitly named scalar cases. state is an immutable, detached export snapshot with validated tensor/group axes and effective stored qparams; learned qparams have already been storage-rounded in the final frozen training forward. Method-specific formats outside these three cases need their own finalizer, not a fallback through master weights.

def finalize_logical_qat_weight(state):
    p = state["qparams"]
    scale, zero = p["scale"], p["zero"]
    qmin, qmax = p["qmin"], p["qmax"]
    check_affine_qparams(scale, zero, qmin, qmax)
    if state["method"] == "master_affine":
        latent = state["weight"] / scale
        if not torch.isfinite(latent).all():
            raise ValueError("invalid affine latent state")
        codes = (torch.round(latent) + zero).clamp(qmin, qmax)
    elif state["method"] == "lr_grid_symmetric":
        if not (zero == 0).all():
            raise ValueError("this low-rank teaching finalizer is symmetric")
        A, B = state["A"], state["B"]
        latent = state["base_grid"] + (state["alpha"] / A.shape[0]) * (B @ A)
        if not torch.isfinite(latent).all():
            raise ValueError("invalid low-rank latent state")
        codes = torch.round(latent).clamp(qmin, qmax)
    elif state["method"] == "fixed_codes_affine":
        codes = state["codes"]  # no refit, requantization, or master-weight path
    else:
        raise ValueError("method needs its own logical finalizer")
    if (not torch.isfinite(codes).all() or not (codes == codes.round()).all()
            or not ((codes >= qmin) & (codes <= qmax)).all()):
        raise ValueError("invalid finalized codes")
    return {"codes": codes.detach().clone().to(torch.int64),
            "scale": scale.detach().clone(), "zero": zero.detach().clone(),
            "qmin": qmin, "qmax": qmax}

def decode_logical_qat_weight(logical):
    return logical["scale"] * (logical["codes"] - logical["zero"])

The graph conversion remains orchestration pseudocode. Its snapshot and frozen-forward adapters must retain method identity, low-rank factors/downcast base where applicable, fixed codes, learned qparams, axes and insertion metadata. The frozen-forward reference comes from the actual trained module, not from calling the finalizer a second time. Validate all candidates before replacing modules in a separate converted graph; failure must not leave a partially converted training object.

def convert_qat_model(model, target):
    model.eval()
    freeze_observers_and_qparams(model)

    prepared = []
    for name, module in qat_modules(model):
        spec = module.export_quantization_spec()
        assert target.supports(spec), (name, spec)

        state = module.freeze_logical_export_state()
        trained_reference = module.frozen_forward_weight(state)
        logical = finalize_logical_qat_weight(state)
        assert_close(decode_logical_qat_weight(logical), trained_reference,
                     rtol=0, atol=0)  # same reference arithmetic; no hidden refit
        prepared.append((name, logical, spec,
                         module.activation_quantizer.export_spec()))

    return build_converted_copy(model, prepared, target)

Exact logical codes and stored qparams must survive packing. A different compute accumulator/epilogue can have a separately documented numerical tolerance, but it cannot excuse changed codes, lost auxiliary contributions, or altered fixed-code state.

0x0BC Packing Changes The Physical ABI

Logical INT4 codes may be packed two per byte, tiled for a tensor-core kernel, interleaved with scales, or reordered by output tile. Let logical code tensor be $Q[N,K]$ and packed bytes be:

The required reference invariant is:

QAT cannot compensate for a packer that swaps nibbles, pads groups incorrectly, or attaches scales to the wrong tiles.

0x0BD Explicit Q/DQ And Fused Kernels

An exported graph may retain explicit operations:

float activation
  -> QuantizeLinear(scale, zero)
  -> integer code tensor
  -> DequantizeLinear
  -> floating operator

A compiler can recognize this pattern and lower it to a fused low-precision kernel. The graph’s Q/DQ boundaries preserve intended numerical semantics; the engine still decides whether a supported fusion exists.

The TensorRT 11.1 documentation inspected in the August 12 review is retained as historical evidence. On September 29, the renamed TensorRT quantized-types overview and explicit Q/DQ documentation were inspected in the browser with the documentation navigation displaying 11.3.0; they describe explicit quantization through Q/DQ layers and support for QAT import under type- and operator-specific constraints, not an installed-runtime support test. The failed raw retrieval remains recorded separately and does not negate that browser inspection. The TorchAO workflow page displaying 0.17, rechecked on September 29, describes prepare inserting fake-quant operations and convert producing quantized operations for supported base configurations. These are version-scoped workflow descriptions, not support for every quantizer here or proof of kernel dispatch; stable/latest URLs remain mutable.

0x0BE Export Parity Test Ladder

Q0: quantizer unit
    exact codes, clipping, rounding, qparams, and dequantization

Q1: fake-quant vs converted operator
    same float input, compare output before packing

Q2: logical codes vs packed/unpacked codes
    bit-exact code and metadata round trip

Q3: reference converted operator vs target kernel
    compare accumulator/epilogue under documented tolerance

Q4: block parity
    residual, norm, attention, and MLP boundaries

Q5: teacher-forced model parity
    logits/perplexity on held-out sequences

Q6: autoregressive state parity
    prefill/decode, K/V append, long context, rollback, reuse

Q7: deployment measurement
    verify target kernel, memory, latency, throughput, and energy

M8.9 will own concrete artifact and loader matrices. M8.10 will own promotion and rollback. M8.7 establishes what they must prove about a trained candidate.

0x0C Evaluation During QAT

0x0CA Quantizer Telemetry

For every important quantizer, log:

scale/clip value and relative drift
zero-point and integer-code range
saturation rate
zero-code fraction
bin occupancy and unused bins
mean and tail reconstruction error
weight-to-code transition count per step
gradient norm for model state and qparams
STE pass-mask fraction
observer range before and after freeze

The transition count:

distinguishes a model whose latent weights still cross bins from one whose training updates are too small to change the deployed codes.

0x0CB Loss And Reconstruction Are Diagnostic, Not Sufficient

Track:

• task or next-token loss;
• distillation KL and teacher entropy;
• block reconstruction loss where used;
• fake-quant versus floating logits;
• held-out perplexity and capability evaluations;
• generation stability at multiple context lengths.

A lower layer MSE can worsen task loss because task sensitivity is anisotropic. A lower distillation loss on synthetic data can still overfit its generation distribution. Promotion requires held-out evaluation that was not used for qparam selection or early stopping.

0x0CC Training Cost Ledger

Measure:

GPU-hours and energy
peak allocated/reserved memory
tokens and optimizer steps
teacher-generation and teacher-forward cost
checkpoint storage
communication volume
recompute overhead
conversion and packing time
failed-run and hyperparameter-search budget

QAT may reduce recurring inference cost while spending substantial one-time training energy. Whether that trade is worthwhile depends on served token volume and model lifetime.

0x0CD Runtime Metrics Still Belong To The Real Artifact

The QAT graph cannot establish:

loaded HBM footprint
kernel selection
prefill throughput
decode TPOT
concurrency under KV pressure
energy per generated token
startup / load time
fallback frequency

Those require the converted and loaded artifact. Training metrics predict quality; they do not prove acceleration.

0x0D Choosing A QAT Strategy

QAT is an escalation, not a mandatory badge. Start from the simplest artifact that passes quality and runtime gates.

0x0E Failure Lab

0x0F Boundaries And Handoffs

It consumes M6.10’s generic distillation contract and M6.4’s QLoRA/base-plus-adapter contract. This article owns their bounded QAT comparison/specialization, not a second generic KD or adapter-serving owner.

M8.7 produces:

fake-quant forward and surrogate-backward contract
fixed/observed/learned qparam state machine
STE, LSQ, and clipping parameter equations
weight, activation, Q/K/V, and K/V insertion map
observer warmup/freeze and progressive-bit schedule
task + distillation objective and synthetic-data manifest
source-scoped LLM-QAT placement and generic masked QAT-distillation trace
full-model and efficient-QAT memory ledgers
distributed qparam/observer synchronization rules
prepare/convert/pack/lower parity ladder
quantizer telemetry and failure tests

It does not explain architectures whose binary or ternary arithmetic is part of pretraining from the beginning. M8.8 receives the latent-weight, normalization, optimizer, scaling, architecture, and kernel questions for BitNet, BitNet b1.58, OneBit, and related native low-bit designs.

It also does not select a production package. M8.9 receives final codes, qparams, insertion graph, trained-state provenance, reference decoder, and parity tests. M8.10 receives the converted candidates, quality evidence, phase benchmarks, energy ledger, and rollback requirements.

0x10 Claim Status

0x11 Publish Gate

• Pretrained, fake-quant, converted, and lowered graphs are distinct.
• Fake quantization is derived as quantize then dequantize arithmetic.
• The true rounding derivative and STE substitution are separated.
• Clipped and identity-everywhere STE design choices are explicit.
• Gradient mismatch is treated as the central QAT approximation.
• LSQ’s piecewise step-size gradient is derived.
• LSQ gradient scaling is identified as a heuristic preconditioner.
• PACT’s clipping/step-size tradeoff and signed-tensor caveat are derived.
• Observer, fake-quant, and trainable-qparam controls are separate.
• Weight, activation, Q/K/V, and K/V fake-quant insertion sites carry shapes.
• Full-sequence training K/V is reconciled with token-wise append state.
• Task loss, temperature-scaled logit distillation, and synthetic generation are derived.
• LLM-QAT data generation, W/A/KV choices, and runtime boundary are traced.
• Full-model QAT parameter, gradient, master, and optimizer bytes are counted.
• Checkpoint recomputation and mutable observer state are reconciled.
• LR-QAT grid-aware low-rank state and inference absorption are explained.
• EfficientQAT block-wise all-parameter and end-to-end qparam phases are distinguished.
• Distributed observer/qparam synchronization and shard-axis constraints are explicit.
• Prepare, convert, pack, lower, and runtime parity form one test ladder.
• Training telemetry and real-artifact runtime metrics are separated.
• Native binary/ternary models remain M8.8; artifact/runtime and release remain M8.9/M8.10.
• Affine tie order, inclusive STE endpoints, detached observer scales and zero-row policy are explicit.
• Generic KD, source LLM-QAT loss/axes, and method-specific logical export state are distinct.

These are content-coverage checks, not publication approval. The accompanying source-extracted NumPy/shim tests exercise logical contracts; they do not run PyTorch autograd, real training, packing kernels, or a deployed model. Independent post-edit review and the consolidated build/browser checks remain separate gates.

0xFF References

Local Series And Transformer System Sources

• M6.10 - Teacher-Student Distillation and Model Transfer - generic KD objectives, temperature, target identity, and supervision provenance. [TRANSFORMER SYSTEM OWNER]
• M6.4 - Low-Rank Adaptation Design Space - QLoRA base/adapter/merge and serving-state contract. [TRANSFORMER SYSTEM OWNER]
• M8.1 - Quantized Numbers and the Kernel Contract - affine codes, rounding, clipping, qparams, packing, accumulation, and kernel ABI. [TRANSFORMER SYSTEM]
• M8.2 - Calibration, Sensitivity, and Precision Maps - observers, calibration coverage, clipping selection, sensitivity, and deployment-feasible precision maps. [TRANSFORMER SYSTEM]
• M8.3 - Outlier Geometry and Equivalent Transforms - learned transforms, clipping, and graph-equivalent range migration. [TRANSFORMER SYSTEM]
• M8.4 - Scalar Weight-Only PTQ - RTN/GPTQ/AWQ baselines and packed scalar weight-only contract. [TRANSFORMER SYSTEM]
• M8.5 - Extreme Weight Compression - trained codebooks, post-quantization refinement, and structured artifact decoders. [TRANSFORMER SYSTEM]
• M8.6 - Live-Data Quantization - activation/KV scale lifetime, cache ABI, KIVI/KVQuant/QServe, and runtime state. [TRANSFORMER SYSTEM]
• Transformer Series (34) - Quantitative Fundamentals - local PTQ/QAT taxonomy, fake-quant workflow, training, and export spine. [SOURCE-LOCAL]
• Transformer Series (35) - Fundamentals of Large Model Quantization - local LLM outlier and QAT/PTQ context. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - local LLM-QAT data-free distillation, W/A/KV quantization, and low-bit method lineage. [SOURCE-LOCAL]

Primary Papers

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation - straight-through estimator lineage and the distinction between true and copied gradients. [PAPER]
• PACT: Parameterized Clipping Activation for Quantized Neural Networks - learned activation clipping and low-precision activation training. [PAPER]
• Learned Step Size Quantization - learned quantizer step size, piecewise scale gradient, gradient scaling, full-precision latent weights, and STE training. [PAPER]
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models, ACL 2024 Findings 26 - published Eq. 4 soft-label cross-entropy, teacher-generated corpus, and source-reported W/A/KV choices. [PAPER]
• LLM-QAT: Data-Free Quantization Aware Training for Large Language Models, arXiv preprint - preprint version of the ACL entry above: teacher-generated corpus, logit distillation, per-token activations/KV, per-channel weights, and source-reported LLM QAT results. [PAPER, PREPRINT]
• LLM-QAT attention at 5ef1289730de080a4781f21ef27105b6c01b116a and its quantizer - full projected-token reduction before head reshape/RoPE; source epsilon and backward differ from the teaching reference. [PINNED OFFICIAL CODE]
• Low-Rank Quantization-Aware Training for LLMs - grid-aware low-rank auxiliary state, downcasting, checkpointing, and inference-time absorption into low-bit codes. [PAPER]
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models, ACL 2025 - block-wise all-parameter training followed by end-to-end quantizer-parameter optimization; published source consumed by the historical review. [PAPER]
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models, arXiv preprint - preprint version of the ACL entry above: block-wise all-parameter training followed by end-to-end quantizer-parameter optimization. [PAPER, PREPRINT]

Official Runtime And Framework Documentation

• TorchAO Quantization-Aware Training Workflow - displayed documentation version 0.17, browser access September 29, 2026; mutable stable URL, prepare/convert workflow only. [OFFICIAL DOCS, DATED OBSERVATION]
• PyTorch 2.8 FakeQuantize API - quantize-dequantize equation and separate observer/fake-quant controls; no claim that this article duplicates its autograd implementation. [VERSIONED OFFICIAL DOCS]
• TensorRT Quantized Types - TensorRT documentation displaying 11.3.0, browser-inspected September 29, 2026 (renamed overview and explicit Q/DQ pages); mutable URL. The August 12 TensorRT 11.1 observation is historical, and the September 29 raw-retrieval failure is preserved separately. No installed-runtime support test. [OFFICIAL DOCS, DATED OBSERVATION]
• TensorRT Explicit Quantization - explicit Q/DQ engine logic, strong typing, low-bit weight packing, ONNX import boundaries, and QAT import under type- and operator-specific constraints; documentation navigation displaying 11.3.0, browser-inspected September 29, 2026; mutable URL. No installed-runtime support test. [OFFICIAL DOCS, DATED OBSERVATION]

The September 29 repair records attempted retrieval URLs/revisions and statuses alongside article/test hashes in docs/numinens-evidence/m8-7-qat-contracts/. Failed direct retrievals are retained, not promoted into source-byte receipts. The earlier source review remains historical evidence where a fresh fetch is unavailable. Local tests execute extracted teaching listings with NumPy/shims; neither a source citation nor those tests establish a trained model’s quality, a production pack ABI, or real-kernel parity.