M8.3 - Outlier Geometry and Equivalent Transforms

M8.3 - Outlier Geometry and Equivalent Transforms

0x00 Why Large Transformers Create A Different Quantization Problem

The move from ordinary neural-network quantization to large-Transformer quantization was not only a move to larger matrices. The numerical distributions changed in a structured way.

The A35 article devotes substantial space to this transition because a small number of activation dimensions can become much larger than the surrounding values, recur across tokens, and appear through many layers. LLM.int8() reported that these systematic emergent features dominate the failure of naive INT8 matrix multiplication at scale and introduced a mixed-precision decomposition that isolates exceptional dimensions. SmoothQuant began from the related observation that LLM weights are generally easier to quantize than activations and moved range between the two through an equivalent transform.

Those papers changed the design question. Instead of asking only for a better clipping threshold, they asked whether the coordinate system presented to the quantizer is itself the problem.

0x00A “Outlier” Must Name A Population And An Axis

An outlier is not an intrinsic label attached to a scalar. It is a claim relative to a population, statistic, axis, and operational consequence.

Consider a hidden-state tensor with token and channel axes. One large element may be a value outlier but occur only once. A channel may contain moderately large values on nearly every token and become a persistent channel outlier. A token may be exceptional across many channels. A weight can be large but harmless under the observed activations, while a modest weight can be highly sensitive because it lies in a direction strongly excited by $X$.

The sharing domain makes these distinctions operational:

Calling all of these “outliers” and applying one remedy loses the mechanism. M8.3 keeps the axes visible before choosing decomposition, scaling, shifting, rotation, or learned transforms.

0x00B Observation, Function, And Cause Are Different Claims

The original source usefully collects several hypotheses for why Transformer outliers emerge: attention heads may need an approximate no-op path, softmax and causal visibility can concentrate behavior on delimiter or sink-like tokens, normalization and residual updates can amplify persistent directions, and RoPE or gated MLP structure can change where large values appear.

These explanations must not be collapsed into one established cause. Three evidence levels should remain separate:

1. observation: large values or persistent dimensions are measured at specified tensors and model scales;
2. functional evidence: ablating, clipping, or isolating them changes attention behavior, perplexity, or downstream quality;
3. causal account: a training or architectural mechanism explains why they formed.

The first two can motivate a quantizer even when the third remains debated. A deployment method needs to know where difficult values occur, how stable their support is, and what happens when they are perturbed. It does not need to settle one universal theory of emergence.

0x00C Why Flatness And Coordinate Choice Matter

A uniform quantizer spends its levels across the observed range. If most values lie near zero while a few coordinates set a much larger maximum, the step size grows and the central mass collapses onto fewer codes. Clipping narrows the range but risks removing a functionally important direction. Finer granularity gives the difficult coordinate its own metadata but increases overhead and may break a preferred kernel layout.

Equivalent transforms introduce another option. For a linear map, an invertible change of basis can preserve the floating-point function while changing the component-wise distribution seen by the quantizer. Diagonal scaling moves range between activations and weight columns. Orthogonal rotations preserve Euclidean norm while spreading coordinate energy. Learned affine transforms can fit both scale and basis to calibration data.

This creates a progression of design freedom:

clip the same coordinates
    -> isolate exceptional coordinates
    -> rescale coordinates
    -> shift asymmetric coordinates
    -> rotate into a flatter basis
    -> learn a structured invertible transform

Each step can reduce quantization error, but each also changes calibration state, graph rewriting, fusion requirements, distributed ownership, and runtime cost. The point of M8.3 is not that rotations are always better than scales. It is to derive the invariants that let these alternatives be compared without confusing floating-point equivalence with quantized equivalence.

0x00D Preserve The Function, Change The Numerical Difficulty

M8.1 derived the object that a low-precision kernel consumes. M8.2 then built the observers, clipping objectives, calibration manifests, and precision maps that choose its numerical parameters.

Those mechanisms still face a structural problem. A tensor can place most of its energy in a small number of coordinates. One exceptional activation then determines the scale shared by many ordinary values, or one persistent channel remains difficult across almost every token. More calibration does not remove that geometry. Clipping can reduce mean error while destroying a rare but functionally important direction.

The owner question is:

Why do structured outliers damage low-precision arithmetic, and how can a deployment isolate, move, spread, or learn a better coordinate system without first changing the floating-point function?

The governing invariant is:

before quantization:
    transformed graph must implement the same function

after quantization:
    transformed coordinates may produce a different error

at deployment:
    every online transform, conversion, fallback, and metadata object
    must enter the runtime ledger

This article follows five progressively more expressive design choices:

The ordering is conceptual, not a leaderboard. A more expressive transform introduces more calibration state, inverse/fusion constraints, and runtime work. A simple diagonal scaling that fuses away can be preferable to a better reconstruction score that requires an unsupported online kernel.

This article owns the geometry and graph transformation. It does not derive GPTQ/AWQ scalar weight updates, extreme weight codebooks, live K/V quantization policies, QAT, native low-bit architecture, artifact integration, or release policy. Those remain M8.4-M8.10.

0x01 An Outlier Is A Geometric And Operational Claim

Use the M8 row-major convention:

$M$ is the local token-row count, $K$ is the reduction/input-feature dimension, and $N$ is the output-feature dimension. An activation outlier can refer to several different objects:

These are not synonyms. A large value need not be sensitive. A modest value can be sensitive because downstream weights amplify it. A channel can be a stable activation outlier while the corresponding weight column is small. A per-token observer and a per-channel observer see different geometries.

0x01A Statistics That Keep The Axes Visible

For channel $k$, useful calibration statistics include:

$a_k$ records the maximum. $e_k$ records accumulated energy. $\rho_k(\tau)$ records occupancy above a threshold. The three disagree when one rare spike and one persistent moderate channel have the same maximum.

A concentration ratio makes the geometry explicit. Let $e_{(1)}\ge\cdots\ge e_{(K)}$ be sorted channel energies:

Large $C_r$ for small $r$ means a small set of channels owns much of the observed energy. It does not prove those channels may be clipped. It says the coordinate basis is concentrated.

For one row vector $x\in\mathbb{R}^{K}$, an incoherence-style score is:

These diagnostics assume finite data, positive $M,K$, and $1\le r\le K$. The ratio $C_r$ requires positive total energy, and $\mu$ requires a nonzero row. Record a separate zero-energy case for an all-zero tensor or row; neither undefined ratio should be silently assigned the nonzero-vector lower bound.

A nonzero perfectly even-magnitude vector has $\mu=1$. A one-coordinate vector has $\mu=\sqrt{K}$. Orthogonal rotations preserve $\lVert x\rVert_2$ but can change $\lVert x\rVert_\infty$, which is why they can change quantization difficulty without changing vector energy.

Renderer-safe definitions:

a_k        = max_m abs(X[m, k])
e_k        = sum_m X[m, k]^2
rho_k(tau) = count_m(abs(X[m, k]) > tau) / M
C_r        = energy in top-r channels / total channel energy
mu(x)      = sqrt(K) * ||x||_inf / ||x||_2

0x01B Empirical Explanations Are Not Universal Laws

The local A35 source discusses delimiter-like tokens, attention concentration, normalization, RoPE, and gated FFNs as possible contexts in which extreme activations appear. Primary papers also use different definitions: LLM.int8() studies systematic feature dimensions, SmoothQuant studies activation channels that are harder to quantize than weights, and later rotation papers study coordinate concentration across several graph sites.

Treat causal explanations as model-, layer-, and paper-specific until measured. A production audit should store:

module and tensor edge
token/request identifiers
layer and phase: prefill or decode
channel maximum, energy, occupancy, and sign asymmetry
corresponding weight-column statistics
output reconstruction sensitivity
observer and model revision

Outliers exist is externally supported. This architectural feature always causes them is usually an interpretation or an unverified generalization.

0x01C Three Rare Structures That Must Not Be Collapsed

Later papers sharpen the vocabulary used loosely in A35. Three structures may all damage a low-bit grid, but they have different tensor support and different evidence:

The distinctions matter operationally. Let an event set be

A persistent feature dimension has a projection onto the channel axis with high token occupancy:

A massive activation can instead have tiny global support,

while its amplitude is orders of magnitude above the ordinary population. A super weight is not defined by either occupancy formula. Its defining claim is intervention-based: perturbing a particular parameter produces a disproportionate functional change and a corresponding activation event in the paper’s studied models.

This gives a minimum diagnostic trace:

def classify_rare_structure(X, W, threshold, intervention=None):
    mask = abs(X) > threshold
    channel_occupancy = mask.float().mean(dim=0)
    token_occupancy = mask.float().mean(dim=1)

    report = {
        "global_density": mask.float().mean(),
        "persistent_channels": nonzero(channel_occupancy > CHANNEL_RATE),
        "rare_tokens": nonzero(token_occupancy > TOKEN_RATE),
        "max_to_median_ratio": abs(X).max() / median(abs(X)),
    }

    if intervention is not None:
        report["causal_delta"] = evaluate_parameter_intervention(
            W, intervention
        )
    return report

The thresholds are model- and tensor-specific. LLM.int8() used a paper-specific magnitude rule in its mixed-precision decomposition and reported a scale-dependent emergence pattern. That observation is valuable evidence, not a universal parameter-count phase transition. Massive Activations and Super Weight use still different definitions and interventions. A production exporter must store the exact detection rule rather than serialize one undifferentiated outlier_mask.

0x01D Attention No-Op Pressure And Implicit Bias Are Related, Not Identical

Quantizable Transformers provides one architectural account for activation outliers. Standard softmax satisfies

For a head update

there is no explicit do nothing code in the attention probabilities. If the desired residual update is zero, the head must synthesize cancellation or place nearly all mass on a token whose value contribution is near zero. For two logits, making a sink probability at least $1-\epsilon$ requires

As $\epsilon\to0$, the required gap grows. The paper connects this training pressure to strong activation outliers in its studied Transformer models and tests two architecture/training alternatives: clipped softmax and gated attention. Those alternatives create a more direct zero or partial-update path. They are not post-training equivalent transforms and cannot be retrofitted by merely changing scales.

Massive Activations reports a different but related functional observation: a few large, largely input-invariant activations can attract attention to their tokens and contribute an implicit bias-like term through the value path. That does not prove every sink token is generated by the no-op mechanism, or that every massive activation is safe to remove.

The deployment consequence is conservative:

observe:
    tensor location, token role, channel support, amplitude, recurrence

intervene:
    zero, mean-replace, preserve, isolate, or transform one mechanism at a time

validate:
    full-precision function, held-out quality, low-bit error, runtime path

do not infer:
    one causal theory from the presence of a large scalar

0x02 Why One Extreme Coordinate Can Erase Ordinary Values

Consider one activation row quantized symmetrically to signed $b$-bit integers. Let

An ordinary coordinate $u$ receives code

It becomes zero whenever

The result follows directly from round-to-nearest: all real values between $-1/2$ and $1/2$ map to integer zero. The outlier does not only receive a coarse code. It enlarges the step applied to every coordinate that shares the scale.

0x02A INT4 And INT8 Trace

Let the row maximum be $A=64$ and an ordinary value be $u=1$.

At INT4, every value with magnitude below $64/(2\cdot7)\approx4.57$ maps to zero. Increasing bit width helps because $Q$ grows, but it does not change the basis mismatch.

If the exceptional values recur in one fixed feature channel, per-token scaling remains awkward: every token row that contains that channel pays for its range. Per-channel activation scaling can isolate it numerically, but high-throughput matrix multiplication reduces along $K$. Applying a different activation scale for every $K$ coordinate inside the dot product can interrupt the regular low-precision MMA path. SmoothQuant was designed around this mathematical/hardware mismatch.

0x02B Error Is Seen Through The Weight Matrix

Let $\widehat X=X+E_X$. With exact weights,

For output row $m$ and channel $n$:

Deleting a large activation coordinate is harmless only if its corresponding weight column contributes little to every relevant output. Magnitude statistics identify range pressure; reconstruction or held-out loss decides functional importance. This is why M8.2 hands M8.3 both outlier statistics and sensitivity evidence.

0x03 The Master Equivalent-Transform Derivation

Let $P\in\mathbb{R}^{K\times K}$ be invertible. Define

Then

The full-precision linear map is unchanged. The coordinate distributions presented to the activation and weight quantizers are not.

Renderer-safe invariant:

X_prime = X * P
W_prime = W * inverse(P).T

X_prime * W_prime.T
    = X * P * inverse(P) * W.T
    = X * W.T

This one equation contains several method families:

0x03A Affine Recentering

For a bias-bearing layer

let $c\in\mathbb{R}^{K}$ and define

Then

The shift can reduce channel asymmetry, but it is not free: $b'$ or an equivalent upstream parameter must exist, every consumer must receive a consistent transform, and nonlinear graph edges can block the movement.

0x03B Equivalence Before Quantization Does Not Imply Equal Quantized Error

Quantize the transformed tensors:

The output error is

$P$ changes $X'$, $W'$, both quantization errors, and how those errors are amplified. The optimization target is therefore not make activation maxima small in isolation. It is a joint activation/weight/runtime objective.

An ill-conditioned transform is especially dangerous. If

is large, one side of the pair can amplify perturbations or finite-precision inversion error. Orthogonal matrices have $\kappa_2(P)=1$; learned nonorthogonal transforms gain expressivity but need invertibility and conditioning controls.

0x03C Graph Equivalence Checklist

The algebra applies to one isolated linear operation. A Transformer graph adds constraints:

every consumer of X must agree on the transformed basis
residual branches must meet in the same basis
normalization parameters may need absorption or rewriting
nonlinearities prevent arbitrary transform movement
RoPE does not commute with an arbitrary hidden/head rotation
tied weights and tensor-parallel shards must transform consistently
online transforms need real kernels and layouts

A correct conversion should test floating-point equivalence before introducing quantization:

def assert_equivalent(reference, transformed, samples, atol, rtol):
    for x in samples:
        y_ref = reference(x)
        y_new = transformed(x)
        assert_close(y_new, y_ref, atol=atol, rtol=rtol)

Failure here is a graph rewrite bug, not quantization error.

0x04 LLM.int8(): Decompose The Reduction Axis

LLM.int8() takes the most direct response to systematic activation dimensions: do not force them through the same INT8 path. Split the reduction dimension into a normal set $\mathcal N$ and an outlier set $\mathcal O$:

Partition activation columns and weight columns on the same axis:

No approximation has happened yet. This is distributivity of the dot product over a partition of its reduction terms. The method then computes the large normal partition through vector-wise INT8 and the small outlier partition in higher precision.

0x04A Vector-Wise Quantization

For the normal partition, a simplified symmetric form assigns one scale to each activation row and one scale to each weight output row:

With integer codes $Q^X,Q^W$:

The matrix of output dequantization factors is the outer product $s^X(s^W)^\top$. This is finer than one tensor-wide scale while remaining compatible with a regular matrix product over the normal columns.

0x04B Tensor Trace

Let the third coordinate be the outlier dimension:

Using zero-based indices, $\mathcal O=\{2\}$ and $\mathcal N=\{0,1,3\}$. The exact partitions are:

Therefore

In the mixed-precision implementation, only the first matrix is approximated by INT8 arithmetic; the second remains in FP16/BF16 or another higher-precision path.

0x04C Detection, Packing, And Combine Trace

def mixed_int8_linear(x, weight, threshold):
    # x: [M, K], weight: [N, K]
    # The real implementation may cache or stabilize the outlier map.
    outlier_mask = abs(x).amax(dim=0) > threshold   # [K]
    normal_idx = nonzero(~outlier_mask)
    outlier_idx = nonzero(outlier_mask)

    x_normal = gather_columns(x, normal_idx)
    w_normal = gather_columns(weight, normal_idx)
    x_outlier = gather_columns(x, outlier_idx)
    w_outlier = gather_columns(weight, outlier_idx)

    qx, sx = quantize_rows_int8(x_normal)       # sx: [M]
    qw, sw = quantize_rows_int8(w_normal)       # sw: [N]

    accum_i32 = int8_gemm(qx, qw.T)             # [M, N]
    y_normal = accum_i32 * sx[:, None] * sw[None, :]
    y_outlier = fp_gemm(x_outlier, w_outlier.T)
    return y_normal + y_outlier, outlier_idx

The LLM.int8() paper used a threshold-based definition and reported that systematic outlier dimensions can be handled in higher precision while more than 99.9% of values remain on the INT8 path. That percentage is source-reported, not a universal deployment constant. The set and threshold depend on model, layer, input, and implementation.

0x04D The Exception Path Is A Runtime Design

The decomposition adds work that nominal bit width does not show:

detect or retrieve outlier dimensions
gather activation and weight columns
launch or execute an irregular high-precision product
run the normal INT8 product
dequantize and combine two outputs
preserve deterministic column maps across sharding

For a very small $|\mathcal O|$, the FP arithmetic count is small, but kernel launch, gather, synchronization, and shape inefficiency can dominate. Dynamic outlier detection adapts to each input but adds runtime work. A static exception map is cheaper but can miss request-dependent dimensions.

LLM.int8() therefore establishes an important baseline: isolation preserves difficult coordinates explicitly. SmoothQuant asks whether the exception path can be removed by moving the range instead.

0x05 SmoothQuant: Move Range Along Matched Channels

Let $D=\operatorname{diag}(s_1,\ldots,s_K)$ with every $s_k>0$. Choose the master transform $P=D^{-1}$:

Then

Each activation channel is divided by $s_k$ and the matching weight column is multiplied by $s_k$. Activation range has not disappeared. It has moved across an equivalent pair.

0x05A Deriving The Migration Rule

Collect channel maxima:

SmoothQuant uses a migration-strength parameter $\alpha\in[0,1]$:

In practice, maxima are clamped by a small $\epsilon$ before exponentiation/division. The transformed maxima are:

The derivation explains the endpoints:

$\alpha=0.5$ equalizes the two channel maxima under this max-based proxy. It is not automatically optimal for output loss or a specific kernel. The paper searches $\alpha$ on calibration data because activation and weight quantizers, modules, and models differ.

Renderer-safe scaling:

s[k] = max(a[k], eps)^alpha / max(w[k], eps)^(1 - alpha)
X_s  = X / s
W_s  = W * s[None, :]

X_s * W_s.T = X * W.T

0x05B Four-Channel Numeric Trace

Suppose:

Then

The activation maximum in channel 2 falls from 64 to 4. The corresponding weight-column maximum rises from 0.25 to 4. The exact full-precision product is unchanged; the two quantizers now receive more balanced ranges.

0x05C Offline Fusion Through Normalization

Suppose a LayerNorm-like affine output is

and the next linear layer computes $ZW^\top$. To produce $ZD^{-1}$ without a separate runtime multiply, change

Then the transformed norm output and weight reconstruct the same linear result. The exact fusion location depends on graph topology. If the same normalized tensor feeds Q, K, and V projections, all three weight matrices must receive the matching column scaling. If the tensor also enters a residual addition or another consumer, the basis/range movement must remain consistent.

def smooth_linear(norm, linears, activation_max, alpha, eps=1e-5):
    # linears share the same normalized activation input.
    weight_max = max_over_consumers_and_output_rows(linears)
    s = activation_max.clamp_min(eps).pow(alpha)
    s /= weight_max.clamp_min(eps).pow(1.0 - alpha)

    norm.weight.div_(s)
    if norm.bias is not None:
        norm.bias.div_(s)

    for linear in linears:
        linear.weight.mul_(s[None, :])

    return s

The transformation should be applied once. Re-running conversion on an already transformed checkpoint compounds $D$ and breaks the intended state unless a manifest records and reverses the previous transform.

0x05D Static And Dynamic Variants

SmoothQuant’s design space includes different choices for activation granularity and when scales are obtained. The paper’s O1/O2/O3 variants trade fine dynamic scaling against coarser static execution. The stable lesson is:

M8.3 owns the offline equivalent scaling. M8.6 will own the live per-request activation quantizer and its prefill/decode cost.

0x05E Outlier Suppression: Migrate LayerNorm Gamma Before Clipping

The original Outlier Suppression paper predates Outlier Suppression+ and makes a narrower, instructive claim. In its studied BERT, RoBERTa, and BART models, the learned LayerNorm scale $\gamma$ aligns with and amplifies difficult activation channels. The proposed Gamma Migration removes that multiplication from the tensor observed by the activation quantizer and folds it into downstream consumers.

For one token, write LayerNorm as:

where $n(x)$ is the normalized vector before learned affine parameters, including LayerNorm’s variance-stabilizing epsilon. Assuming every $\gamma_k\ne0$, define a non-scaling output:

Then:

For a following linear operator $y=W\operatorname{LN}(x)+b$, use:

so:

The full-precision function is unchanged, while the quantized activation edge sees $x'$ rather than $\gamma\odot x'$. This is not the same transform as SmoothQuant. SmoothQuant chooses a channel scale from activation and weight ranges; Gamma Migration moves a learned normalization parameter already present in the graph.

Residual topology matters. If the LayerNorm output feeds both a linear branch and a shortcut branch, each consumer must receive an algebraically matching rewrite. Folding $\gamma$ only into $W$ while leaving the residual basis unchanged is not equivalent.

def migrate_layernorm_gamma(layernorm, linear_consumers, residual_consumers):
    # Snapshot before resetting the producer; an alias would become all ones.
    gamma = layernorm.weight.detach().clone()
    assert all_finite(gamma) and all_nonzero(gamma)

    # LN now emits n(x) + beta/gamma.
    if layernorm.bias is not None:
        layernorm.bias.div_(gamma)
    layernorm.weight.fill_(1.0)

    for linear in linear_consumers:
        linear.weight.mul_(gamma[None, :])

    for residual in residual_consumers:
        residual.insert_channel_scale(gamma)

The pseudocode is a graph-rewrite contract, not a generic PyTorch recipe. It assumes a learned scale, distinct untied consumer parameters, and a conversion context with gradient recording disabled. The saved scale must remain independent of the mutated producer; a missing bias means zero bias. Zero or non-finite scale entries fail this rewrite rather than being divided through. Architectures differ in pre-norm/post-norm placement, residual ownership, tied parameters, and whether normalization has a learned scale at all. The source evidence is LayerNorm-centric; it must not be silently generalized to every RMSNorm decoder LLM.

0x05F Token-Wise Clipping Searches Importance, Not Only Magnitude

Gamma Migration attacks one source of range amplification. Outlier Suppression then observes that a long activation tail can be carried by only a few token rows, and that clipping impact is not monotonic in magnitude. Its Token-Wise Clipping procedure has two stages:

1. coarse token-wise search: use quantiles of per-token upper and lower extrema to propose clipping bounds, so the search jumps over tail regions and evaluates candidate clipping ratios efficiently;
2. fine search: refine the selected scale by gradient descent on quantized reconstruction loss.

These are the source method’s two stages. Its affine interval uses $s=(c_u-c_l)/(2^b-1)$. The following symmetric teaching variant instead searches a single half-range $c$ and uses discrete candidates in both stages; it is not a reproduction of the paper’s optimizer.

For $b\ge2$, use the narrow symmetric integer alphabet $q\in[-Q,Q]$, where $Q=2^{b-1}-1$. For a positive half-range $c$, the step is:

Thus the endpoints reconstruct exactly to $-c$ and $c$ in real arithmetic; one of the $2^b$ possible bit patterns is unused. Round ties to even, saturate the codes, and reconstruct with $\widehat x=sq$. The teaching search uses the same general output-reconstruction principle, rather than minimizing only local activation MSE:

import numpy as np


def fake_quant_symmetric(activation, clip, bit_width):
    # Float64 reference, not a storage format or an optimized kernel.
    if isinstance(bit_width, (bool, np.bool_)) or not isinstance(
        bit_width, (int, np.integer)
    ) or not 2 <= bit_width <= 16:
        raise ValueError("teaching helper requires integer bits in [2, 16]")
    x = np.asarray(activation, dtype=np.float64)
    c = np.asarray(clip, dtype=np.float64)
    if x.size == 0 or not np.isfinite(x).all():
        raise ValueError("activation must be nonempty and finite")
    if c.ndim != 0 or not np.isfinite(c) or c < 0:
        raise ValueError("clip must be a finite nonnegative scalar")
    if c == 0:
        if np.any(x != 0):
            raise ValueError("zero clip is reserved for an all-zero input")
        # Sentinel grid: scale=1, every code=0, reconstruction=0.
        return np.zeros_like(x)
    Q = 2 ** (int(bit_width) - 1) - 1
    scale = float(c) / Q
    if not np.isfinite(scale) or scale <= 0:
        raise ValueError("scale is not representable in the reference dtype")
    if not np.isfinite(scale * Q):
        raise ValueError("decoded grid endpoint overflows the reference dtype")
    # Clip first so a very large finite x cannot overflow x/scale.
    codes = np.clip(np.rint(np.clip(x, -c, c) / scale), -Q, Q)
    return scale * codes


def token_wise_clipping(reference_op, activation, bit_width, ratios):
    activation = np.asarray(activation, dtype=np.float64)
    if activation.ndim != 2 or activation.size == 0:
        raise ValueError("expected a nonempty token-by-channel matrix")
    token_max = np.max(np.abs(activation), axis=-1)
    global_max = token_max.max()
    reference_output = reference_op(activation)

    best = None
    for ratio in coarse_candidates_from_token_maxima(token_max, ratios):
        clip = ratio * global_max
        x_q = fake_quant_symmetric(activation, clip, bit_width)
        loss = output_mse(reference_output, reference_op(x_q))
        best = keep_lower_loss(best, clip, loss)

    if best is None:
        raise ValueError("coarse search must provide at least one candidate")
    for clip in fine_candidates_around(best.clip):
        x_q = fake_quant_symmetric(activation, clip, bit_width)
        loss = output_mse(reference_output, reference_op(x_q))
        best = keep_lower_loss(best, clip, loss)

    return best

The quantizer is executable NumPy; the search remains a contract with explicit adapters for candidate generation, output MSE, and retaining the lower-loss result. Candidate clips must be finite and nonnegative, the coarse list must be nonempty, and reference_op must be deterministic and leave its inputs unchanged. Both loops use the same grid. The all-zero sentinel and float64 scale/decoded-endpoint checks are numerical boundary policies of this teaching helper, not claims about the source implementation. An unrepresentable grid fails rather than silently changing its decoder.

The source paper’s fine stage learns a small clipping adjustment on calibration data. That is fitted quantizer state, with an optimizer and overfitting risk, even though the base model is not conventionally fine-tuned. Preserve the ablations:

This lineage matters because a quality result from the full Outlier Suppression pipeline cannot be attributed to Gamma Migration alone, and a later LLM method cannot be described as merely the same BERT-era recipe at a larger scale.

0x06 From Closed-Form Scaling To Learned Shifts And Scales

SmoothQuant chooses diagonal scales from channel maxima and a searched $\alpha$. Two limitations motivate learned affine variants:

1. a channel distribution can be strongly asymmetric, so scaling around zero wastes codes;
2. max-based balancing does not directly minimize block output error under the actual quantizers.

0x06A Channel-Wise Shifting

Outlier Suppression+ adds channel-wise shifting before scaling. In the affine derivation from 0x03A, $c_k$ recenters each activation channel and the next bias absorbs $Wc$.

For an asymmetric channel with observed interval $[l_k,u_k]$, a simple midpoint candidate is

After $x_k\leftarrow x_k-c_k$, the interval becomes symmetric around zero with half-range $(u_k-l_k)/2$. This is only an initialization or analytic choice. A downstream reconstruction objective may prefer a different shift because channel values are not uniformly important.

0x06B OmniQuant’s Learnable Equivalent Transformation

OmniQuant keeps the original model weights frozen and learns a small set of quantization parameters from calibration data. Its two central components are:

For block $l$, a schematic reconstruction objective is:

$F_l$ is the reference block. $\widehat F_l$ applies the equivalent parameter rewrite and fake quantization. The base $W_l$ remains frozen; only transform/clipping parameters receive optimizer updates.

for block in model.blocks:
    reference_in = cached_reference_inputs(block)
    reference_out = block.fp_forward(reference_in).detach()

    transform = initialize_scale_shift(block)
    clipping = initialize_weight_clipping(block)
    optimizer = AdamW([transform.params, clipping.params])

    for step in range(num_calibration_steps):
        # Quantizers use fake-quant/STE for fitted parameters.
        y_hat = block.quantized_forward(
            reference_in,
            equivalent_transform=transform,
            clipping=clipping,
            frozen_base_weights=True,
        )
        loss = mse(y_hat, reference_out)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    materialize_or_fuse_transform(block, transform)
    freeze_quant_params(block, clipping)

This is gradient-based PTQ, not ordinary model fine-tuning and not QAT over the original model parameters. It still has training details that must be recorded:

calibration corpus and tokenization
block order and cached-input policy
optimizer, learning rates, epochs/steps, batch size
fake-quantizer and STE definition
scale/shift initialization and parameter constraints
clipping parameterization
whether quantized outputs feed the next block during calibration
stopping and held-out reconstruction criteria

OmniQuant reports block-wise calibration with a small calibration set and frozen original weights. Its exact sample counts and schedules are source-reported recipes, not universal defaults.

0x06C What Can Be Fused

Channel scales and shifts can often be folded into an upstream normalization/linear parameter and a downstream weight/bias. Fusion fails or becomes coupled when:

the transformed activation has multiple consumers with incompatible rewrites
an intervening nonlinearity changes before/after-transform equivalence
the module has no compatible bias path for the shift
shared/tied parameters would require two transformed versions
tensor-parallel sharding splits an axis differently from the transform
the implementation's learned inverse is numerically unstable

The paper reports an exception around one FFN transformation due to optimization stability. A production converter must follow the paper/code graph, not apply one generic fold every scale and shift rule.

0x07 Orthogonal Rotation: Spread Energy Without Changing Norm

Let $R\in\mathbb{R}^{K\times K}$ be orthogonal:

Use

Because $R^{-\top}=R$:

For every row $x$:

The total Euclidean energy is fixed, but coordinate maxima can change. A randomized orthogonal basis makes alignment with one coordinate unlikely; high-probability bounds motivate the word incoherent. It is not correct to claim that every rotation improves every tensor.

0x07A Hadamard Trace

The normalized order-four Hadamard matrix is

For a concentrated row

the rotated row is

The maximum falls from $8$ to $4$ while the squared norm stays $64$. With random sign flips before the Hadamard transform, the method reduces deterministic alignment between model coordinates and Hadamard structure.

0x07B Fast Walsh-Hadamard Transform

A dense $K\times K$ rotation costs $O(K^2)$ operations per row. The Walsh-Hadamard transform uses butterfly stages and costs $O(K\log K)$:

def fwht(x):
    # Last dimension K is a supported Hadamard size.
    y = x.clone()
    width = 1
    while width < y.shape[-1]:
        blocks = y.reshape(*y.shape[:-1], -1, 2 * width)
        left = blocks[..., :width].clone()
        right = blocks[..., width:].clone()
        blocks[..., :width] = left + right
        blocks[..., width:] = left - right
        y = blocks.reshape_as(y)
        width *= 2
    return y / sqrt(y.shape[-1])

Classic Sylvester Hadamard sizes are powers of two. Real hidden/head dimensions may require block decompositions or Kronecker products with known Hadamard orders. That choice changes layout, fusion, and mixing quality and must be part of the artifact manifest.

0x08 QuaRot: Rotation Is A Graph Topology, Not One Matrix

QuaRot applies randomized Hadamard rotations at several Transformer locations so weights, activations, and K/V can be presented in less concentrated coordinates for low-bit quantization. The important implementation lesson is that some rotations are absorbed into weights offline and others remain online because graph operations block fusion.

0x08A Residual-Stream Rotation And RMSNorm

For pure RMS normalization without learned affine scaling:

Orthogonality preserves $\lVert xR\rVert_2$, so the scalar RMS denominator is unchanged. Learned RMSNorm scale parameters do not generally commute with a dense rotation because a diagonal multiplication and a dense $R$ do not commute. QuaRot first absorbs the learned scaling into adjacent weights, then reasons about the remaining norm-preserving operation.

A global residual rotation must be consistent across every residual branch. Rotating one branch but not the skip path makes the addition combine different bases.

0x08B Attention Value/Output Pair

Within one attention head, the value path is linear before the weighted sum. A head-space rotation can be inserted on $V$ and paired with an inverse/absorbed transform in the output projection. The attention probabilities are unchanged because they depend on Q/K scores, not the value basis.

The graph trace is:

V projection
  -> value rotation in head coordinates
  -> attention weighted sum in rotated value basis
  -> matching inverse absorbed into output projection
  -> original residual-space result

0x08C Q/K Rotation Meets RoPE

For scores without positional transformation, applying the same orthogonal head rotation to $q$ and $k$ preserves dot products:

RoPE inserts position-dependent rotations between projection and score computation. An arbitrary Hadamard rotation does not commute with those position-dependent block rotations, so it cannot simply be fused into Q/K projection weights. QuaRot therefore retains an online rotation at the compatible post-RoPE boundary. Its cache placement choice is also operational: rotating one new query row per decode is different from repeatedly rotating a growing key history.

M8.3 owns why the transform must sit at this graph edge. M8.6 owns the exact K/V code, group, cache layout, append, and attention-kernel contract.

0x08D FFN Down-Projection Boundary

The input to the down projection is produced by a nonlinearity and elementwise gate. A dense rotation cannot move backward through that nonlinear operation. QuaRot applies an online Hadamard transform after the gated FFN activation and absorbs the paired inverse into the down-projection weight.

def rotated_down_projection(gated_hidden, down_weight_rotated):
    # Online because the gate/nonlinearity blocks fusion into prior weights.
    h_rot = fwht_with_random_signs(gated_hidden)
    qh, sh = quantize_per_token(h_rot)
    return lowbit_linear(qh, down_weight_rotated, sh)

0x08E QuaRot Conversion And Runtime Ledger

offline:
    absorb RMSNorm scales where required
    sample/store deterministic sign and Hadamard choices
    rotate compatible weights
    pair value/output and FFN-down transforms
    quantize transformed weights (RTN or reconstruction method)

online:
    transform graph sites blocked by nonlinearity or RoPE
    quantize request-dependent transformed activations
    execute low-bit GEMMs and attention path

The paper evaluates an end-to-end W4A4/KV4 system and reports quality/throughput results, but those numbers are source-reported for its models, kernels, and hardware. The reusable mechanism is computational invariance plus explicit online-transform placement.

0x08F DuQuant: Rotation And Permutation Solve Different Block Problems

QuaRot asks how a structured orthogonal basis can reduce coordinate concentration across the Transformer graph. DuQuant v3 adds a different concern: efficient block-wise rotations mix only within their blocks, so concentrating difficult channels in a few rotation blocks limits what the next local mixing stage can achieve. These are rotation blocks, not necessarily quantizer sharing groups; the paper’s main setup uses per-token activation and per-output-channel weight quantization.

The paper distinguishes normal outliers, moderately large activation dimensions recurring across tokens, from massive outliers, far larger values concentrated at very few token-coordinate positions. Its dual transform has three conceptual stages:

1. construct a block-wise rotation using observed outlier locations as a prior, spreading difficult coordinates locally;
2. rank channels by their calibrated absolute maxima and use a zigzag permutation to redistribute them across rotation blocks;
3. apply a second rotation to smooth the resulting within-block distributions.

Let $R_1$ and $R_2$ be block-orthogonal transforms and $\Pi$ a permutation matrix. The activation-side map is

Because each factor is orthogonal,

and the matching weight transform remains

under the orthogonal case of the row-major master identity. The ordering is important. A permutation does not reduce a scalar magnitude; it changes which coordinates the next block rotation mixes together. A rotation changes magnitudes and energy distribution inside the mixed coordinates. Their objectives are therefore complementary. Under a permutation-invariant per-token range rule, permutation alone preserves that token’s extrema and scalar quantization error up to the same reordering. A groupwise quantizer could additionally change scale-sharing membership, but that is a different quantizer contract, not the reason to describe DuQuant’s blocks as scale groups.

A teaching conversion trace is:

def build_dual_transform(X_cal, block_size):
    outlier_prior = locate_normal_and_massive_outliers(X_cal)
    R1 = make_outlier_aware_block_rotation(outlier_prior, block_size)

    X1 = X_cal @ R1
    channel_maxima = X1.abs().amax(dim=0)  # one statistic per channel
    permutation = zigzag_balance_channels(channel_maxima, block_size)

    X2 = permute_channels(X1, permutation)
    R2 = make_second_block_rotation(X2, block_size)

    P = compose(R1, permutation, R2)
    assert_close(P.T @ P, identity(P.shape[0]))
    return P, outlier_prior

Here X_cal is a token-by-channel matrix; the named rotation and zigzag routines are conceptual adapters, not a reproduction of the full converter. The real implementation must define one consistent channel permutation across all token rows, block boundaries, tensor-parallel ownership, and which factors fuse into adjacent weights. A channel permutation crossing tensor-parallel shards is not metadata-only; it can require checkpoint reshaping or communication. As with QuaRot, any transform left online enters the kernel and latency ledger. DuQuant’s benchmark improvements are source-reported, not evidence that its prior or block size transfers unchanged to another model.

0x09 SpinQuant: Learn The Orthogonal Basis

Random orthogonal rotations reduce coordinate alignment on average, but different random matrices can produce noticeably different quantized quality. SpinQuant treats the rotation matrices as calibration parameters and optimizes them under a quantized model objective while keeping the base model frozen.

The constraints matter. An unconstrained learned matrix can become singular or ill-conditioned and destroy the easy inverse. SpinQuant keeps rotations on the Stiefel/orthogonal manifold.

0x09A Cayley Update Derivation

Let $R$ be orthogonal and $G=\partial L/\partial R$ its Euclidean gradient. A schematic skew-symmetric tangent generator is

Define the Cayley factor

Because $A^\top=-A$:

Therefore

remains orthogonal if $R$ is orthogonal. The sign/order convention can be reversed depending on the optimizer’s descent convention; the invariant is that a skew-symmetric generator produces an orthogonal Cayley factor.

Renderer-safe schematic:

A     = G * R.T - R * G.T        # A.T = -A
C     = inverse(I + eta*A/2) * (I - eta*A/2)
R_new = C * R

R_new.T * R_new = I

0x09B Calibration Loop

In SpinQuant v4, sections 3.1–3.2, learned $R_1,R_2$ are mergeable; optional online $R_3,R_4$ remain fixed Hadamard transforms. SpinQuant_no_had uses only the mergeable pair; SpinQuant_had adds the online pair. The following sketch follows the main experimental recipe in section 4.1: learn rotations with activation quantization and 16-bit weights, then apply GPTQ to the rotated weights. It does not optimize every inserted transform.

freeze(base_model.weights)
rotations = initialize_spinquant_rotations(model, variant)
learned_rotations = select_R1_R2(rotations)

for step, batch in calibration_steps:
    logits = quantized_forward(
        model,
        batch,
        rotations=rotations,
        fake_quant_activations=True,
        quantized_or_reconstructed_weights=False,  # main recipe: 16-bit here
    )
    loss = cross_entropy(logits[:, :-1], batch.tokens[:, 1:])
    gradients = backward(loss, learned_rotations)

    for R, G in zip(learned_rotations, gradients):
        R.copy_(cayley_step(R, G, learning_rate))
        assert_close(R.T @ R, identity(R.shape[0]))

# Only after rotation learning: absorb R1/R2 and reconstruct/quantize weights.
apply_gptq_to_rotated_weights(model, rotations)

Training details are part of the method:

The paper reports one concrete iteration count, sample set, learned-parameter fraction, and speed/quality result. Those are source-reported recipe/results, not architecture-independent guarantees.

0x09C QuaRot Versus SpinQuant

Learning the basis can improve quantization error. Record the variant: an online transform, when selected, is not free, while the no_had path must not be charged for transforms it does not execute. Neither variant replaces a kernel compatibility check.

0x0A FlatQuant: Learn A Broader Structured Invertible Map

Diagonal transforms cannot mix channels. Orthogonal transforms mix channels but preserve singular values and cannot rescale directions. A general invertible $P$ can do both, but a dense $K\times K$ matrix requires $K^2$ parameters and $O(MK^2)$ online work for $XP$ if it cannot be absorbed.

FlatQuant learns a layer-specific invertible transformation while constraining it to a fast Kronecker structure.

0x0A1 Kronecker Structure

Let $K=k_1k_2$ and

A dense map stores $K^2=k_1^2k_2^2$ parameters. The two factors store

When $k_1\approx k_2\approx\sqrt K$, factor storage is $O(K)$ rather than $O(K^2)$.

Using column-major vectorization, the identity

replaces one large multiplication with two small matrix multiplications on a reshaped activation. Exact transpose order depends on row/column and reshape conventions; an implementation must unit-test its convention rather than copy the symbolic order blindly.

For the row-major article convention, suppose a row is transformed as

Reshape $x$ in C order into $Z\in\mathbb{R}^{k_1\times k_2}$. Transposing the row product and applying the vectorization identity gives:

That is the convention implemented below. A different reshape convention requires a different transpose order.

def kronecker_transform_row(x, P1, P2):
    # Row-major contract: y = x @ kron(P1, P2).
    assert P1.ndim == P2.ndim == 2
    k1, k2 = P1.shape[0], P2.shape[0]
    assert k1 > 0 and k2 > 0
    assert P1.shape == (k1, k1) and P2.shape == (k2, k2)
    assert x.ndim == 1 and x.numel() == k1 * k2
    z = x.reshape(k1, k2)
    y = P1.T @ z @ P2
    return y.reshape(-1)


def test_kronecker_row_convention(x, P1, P2):
    y_fast = kronecker_transform_row(x, P1, P2)
    y_dense = x @ kron(P1, P2)
    assert_close(y_fast, y_dense)

0x0A2 Layer-Specific Training Objective

FlatQuant jointly learns transformation factors, per-channel scaling, and clipping thresholds for each Transformer block using calibration data. A schematic objective is:

where $\Theta_l$ includes the invertible transform factors, scales, and clipping parameters for block $l$. The paper trains blocks sequentially with original model weights fixed.

Its reported recipe uses 128 sequences of 2048 tokens, 15 epochs, AdamW, and separate learning rates for transform and clipping state. Those values are source-reported implementation details. A new deployment should validate convergence and held-out reconstruction rather than inherit them as constants.

0x0A3 Inversion Is A Numerical Problem

The paired weight rewrite requires $P^{-\top}$. Direct low-precision inversion can produce a matrix for which $PP^{-1}$ is not close enough to identity. FlatQuant discusses two training recipes:

For

with orthogonal $U,V$ and nonzero diagonal $\Sigma$:

The converter should monitor at least:

||P * P_inverse - I||
minimum singular value
maximum singular value
condition number
full-precision graph equivalence error
quantized block reconstruction error

0x0A4 Transformer Integration And Kernel Fusion

FlatQuant’s logical transform sites include the Q/K/V input, attention output-projection input, per-head K/V coordinates, FFN input, and down-projection input. Compatible scales are merged into neighboring weights or normalization parameters. The paper reports five transformations remaining online after its pairwise fusion choices.

The online transform is memory-bound and quantization is also memory-bound. Materializing the intermediate transformed activation in HBM would add a read/write pass and a kernel launch. FlatQuant’s fused Triton path keeps transform intermediates in on-chip SRAM and quantizes before writing the result when the selected factor/tile working set fits. Appendix B.2 of v2 also describes a split path for capacity-limited large transformations; the one-kernel sketch is not an unconditional SRAM-residency guarantee:

def fused_transform_quantize_tile(x_tile, P1, P2, qspec):
    # Conceptual single-row tile, with the same C-order contract as above.
    assert P1.ndim == P2.ndim == 2
    k1, k2 = P1.shape[0], P2.shape[0]
    assert k1 > 0 and k2 > 0
    assert P1.shape == (k1, k1) and P2.shape == (k2, k2)
    assert x_tile.ndim == 1 and x_tile.numel() == k1 * k2
    z = x_tile.reshape(k1, k2)
    z = small_matmul_left(P1.T, z)
    z = small_matmul_right(z, P2)
    # Quantizer groups refer to the flattened row, not the temporary axes.
    q, scale = quantize_in_sram(z.reshape(-1), qspec)
    store_packed_codes_and_scale(q, scale)

For several token rows, apply this contract independently per row; the temporary factor axes do not redefine the quantization sharing domain. The matrix-multiply, quantizer, and store calls are adapters, not executable Triton. Test the pre-quantization result against x_tile @ kron(P1, P2) with unequal factor sizes and nonsymmetric factors before testing packed output.

This is a deployment mechanism, not merely a PTQ score. If the fused kernel is absent, the transform may erase the latency saved by the low-bit GEMM. Factor sizes also affect memory access regularity; the paper reports that balanced factor shapes are favorable and that some larger factors create irregular access penalties.

0x0A5 FlatQuant Versus Earlier Learned Transforms

AffineQuant v1, section 3.3 describes merging supported transforms into neighboring parameters; its weight/activation path restricts post-LayerNorm transforms to diagonal form for that fusion. A generic unabsorbed dense map has the cost derived above, but that is not a universal description of AffineQuant inference. FlatQuant’s paper reports W4A4 quality and speed improvements over several baselines. Those are source-reported measurements. The general lesson is that transform expressivity, inverse stability, factor shape, fusion, and kernel memory traffic must be optimized together.

0x0B Unified Design Comparison

0x0B1 These Methods Are Not Strict Supersets

It is tempting to order the table from old to new and choose the last row. That is not a valid deployment rule.

• LLM.int8() preserves exceptional dimensions explicitly and can be robust when a small mixed-precision path is supported.
• SmoothQuant may remove a mixed path with a transform that largely fuses away.
• Rotations can help both activation and weight coordinates without selecting an exception threshold, but online graph barriers remain.
• Learned transforms can fit a particular calibration objective more closely while adding calibration, artifact, inversion, and kernel state.
• A weight-only deployment may choose AWQ/QuIP-style preconditioning instead; M8.4/M8.5 own those weight representation decisions.

The correct comparison is a Pareto surface over held-out quality, loaded memory, prefill latency, decode inter-token latency, transform overhead, conversion complexity, and runtime support.

0x0C End-To-End Transform Selection And Conversion Trace

def build_outlier_transform_plan(model, D_cal, D_select, runtime):
    # 1. Measure geometry on exact tensor edges.
    stats = collect_outlier_stats(
        model,
        D_cal,
        metrics=["max", "energy", "occupancy", "asymmetry", "incoherence"],
        phases=["prefill", "decode"],
    )

    # 2. Cross range pressure with functional sensitivity.
    candidates = []
    for edge in stats.edges:
        candidates += propose_methods(
            edge,
            stats[edge],
            allowed=["decompose", "smooth", "shift_scale",
                     "fixed_rotate", "learn_rotate", "structured_affine"],
        )

    # 3. Remove candidates whose graph or kernels are unsupported.
    candidates = [
        c for c in candidates
        if graph_rewrite_is_legal(model.graph, c)
        and runtime.supports(c.online_ops, c.layouts, c.dtypes)
    ]

    artifacts = []
    for candidate in candidates:
        transformed = fit_or_construct(candidate, model, D_cal)

        # 4. Equivalence is tested before quantization.
        assert_fp_equivalent(model, transformed, D_select.fp_probe)

        # 5. Quantize under the exact intended representation/kernel contract.
        quantized = quantize_transformed_graph(transformed, candidate.qspec)
        metrics = evaluate_candidate(
            quantized,
            D_select,
            runtime,
            include=["quality", "prefill", "decode", "memory", "fallbacks"],
        )
        artifacts.append((candidate, quantized, metrics))

    # 6. Freeze the choice before held-out release evaluation.
    return choose_pareto_candidate(artifacts)

0x0C1 Transform Manifest

transform:
  method: smoothquant
  source_model_revision: ...
  graph_edge: blocks.12.input_layernorm:output
  consumers: [q_proj, k_proj, v_proj]
  input_axis: -1
  alpha: 0.55
  scale_state_hash: sha256:...
  calibration_manifest: sha256:...
  fused_into:
    producer: blocks.12.input_layernorm
    consumers: [q_proj, k_proj, v_proj]
  fp_equivalence:
    max_abs: ...
    max_rel: ...

quantization:
  activation: int8_per_token_symmetric
  weight: int8_per_output_channel_symmetric
  packed_layout: ...

runtime:
  engine_revision: ...
  required_kernels: [...]
  online_transforms: []
  fallback_forbidden: true

A rotation/FlatQuant manifest additionally needs matrices or reproducible seeds, factor shapes, reshape convention, inverse hashes, fusion locations, and the online transform kernel ABI.

0x0D Runtime Ledger

For one transformed operation, deployment time is approximately:

The transform can still be worthwhile if it enables a faster low-bit operator or protects quality at a lower bit width. The ledger must be measured separately for prefill and decode:

Tensor parallelism adds another constraint. A transform over the full $K$ dimension may mix coordinates owned by different ranks. Options include:

restrict transform to rank-local blocks
apply a distributed transform with communication
change sharding after conversion
choose a factorization aligned with shards
reject the candidate for that runtime

The first option changes mixing quality; the second can erase speedup; the third changes checkpoint/runtime compatibility. There is no cost-free just rotate the tensor operation in a sharded graph.

0x0E Failure Lab

0x0F Boundaries And Handoffs

M8.3 produces:

outlier geometry statistics with explicit axes
candidate exception maps or equivalent transforms
full-precision equivalence evidence
fused and online transform graph plan
learned transform/clipping state where applicable
conditioning and inverse diagnostics
runtime kernel and phase-cost requirements

It does not claim that the resulting model is a released artifact.

0x10 Claim Status

0x11 Publish Gate

• Activation value, token, channel, persistent-dimension, weight, and sensitivity outliers are distinguished.
• Systematic feature dimensions, massive activations, and super-weight-induced activations are separated by tensor support and intervention evidence.
• Attention no-op pressure and massive-activation implicit bias are derived and kept as distinct empirical accounts.
• Quantization resolution loss is derived from the shared scale and code range.
• The master invertible transform and affine shift identities are derived.
• Full-precision equivalence and post-quantization error are separated.
• LLM.int8() includes a reduction-axis derivation, tensor trace, pseudocode, and runtime costs.
• SmoothQuant includes its scale derivation, $\alpha$ endpoints, numeric trace, and fusion mechanics.
• Original Outlier Suppression includes Gamma Migration derivation, residual-consumer constraints, Token-Wise Clipping stages, training state, and architecture scope.
• Shift/scale methods include bias absorption, learned PTQ state, and calibration details.
• Orthogonal rotations include norm preservation, a Hadamard trace, and FWHT pseudocode.
• QuaRot includes residual, attention, RoPE, FFN, offline, and online graph boundaries.
• DuQuant includes normal/massive outlier distinctions, rotation-permutation-rotation mechanics, block granularity, and sharding constraints.
• SpinQuant includes learned-rotation state, Cayley mechanics, and comparison with fixed rotations.
• FlatQuant includes Kronecker derivation, inversion, training, Transformer integration, and kernel fusion.
• The Kronecker row/reshape convention is explicit and tested against the equivalent dense transform.
• Paper variants are compared by mechanism and runtime contract rather than benchmark rank.
• Tensor parallelism, phase shape, conversion idempotence, and kernel fallbacks are explicit.
• M8.4-M8.10 retain clear ownership.

0xFF References

Local Series Sources

• M8.1 - Quantized Numbers and the Kernel Contract - affine representation, quantization error, scale shape, packing, and kernel contract. [TRANSFORMER SYSTEM]
• M8.2 - Calibration, Sensitivity, and Precision Maps - observer state, clipping, reconstruction, calibration corpora, sensitivity, and runtime-feasible precision maps. [TRANSFORMER SYSTEM]
• Transformer Series (35) - Fundamentals of Large Model Quantization - LLM activation/weight behavior and local outlier discussion. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - LLM.int8(), SmoothQuant, ZeroQuant, OmniQuant, rotation, and FlatQuant source material. [SOURCE-LOCAL]

Primary Papers

• LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale - vector-wise INT8, mixed-precision decomposition, and systematic outlier dimensions. [PAPER]
• Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing - attention no-op pressure, clipped softmax, gated attention, and activation-outlier evidence. [PAPER]
• Massive Activations in Large Language Models - rare massive activations, location/intervention evidence, and implicit bias-like function. [PAPER]
• The Super Weight in Large Language Models - super-weight intervention evidence, induced super activations, and high-precision exception experiments. [PAPER]
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - equivalent channel scaling, migration strength, calibration, and W8A8 execution variants. [PAPER]
• Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models - LayerNorm Gamma Migration, token-wise coarse-to-fine clipping, and BERT/RoBERTa/BART quantization evidence. [PAPER, SCOPE-LIMITED]
• Outlier Suppression+: Accurate Quantization of Large Language Models by Equivalent and Optimal Shifting and Scaling - equivalent channel-wise shifting and scaling for activation outliers. [PAPER]
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models - learnable weight clipping, learnable equivalent transformation, and block-wise PTQ calibration. [PAPER]
• QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs - randomized Hadamard rotations, computational invariance, graph integration, and online/offline rotation boundaries. [PAPER]
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs - outlier-aware block rotation, zigzag permutation, second rotation, and normal/massive outlier treatment. [PAPER]
• Official DuQuant Repository - conversion and implementation context. [OFFICIAL CODE]
• SpinQuant: LLM Quantization with Learned Rotations - learned orthogonal rotations, Cayley/Stiefel optimization, and quantized calibration loss. [PAPER]
• FlatQuant: Flatness Matters for LLM Quantization - layer-specific affine transformations, Kronecker factors, block-wise calibration, inverse parameterization, and fused transform/quantize kernels. [PAPER]
• AffineQuant: Affine Transformation Quantization for Large Language Models, v1 - eligible transform sites, gradual fitting and fusion restrictions; section 3.3 qualifies inference cost. [PAPER, REVISION-BOUND]