M8.9 - Quantized Artifacts and Runtime Integration

M8.9 - Quantized Artifacts and Runtime Integration

0x00 From A Numerical Method To A Software Artifact

Quantization papers often end after reporting reconstructed weights, perplexity, and a kernel benchmark. Deployment begins after that point. The chosen numerical representation must cross a software supply chain without changing meaning:

trained or calibrated state
    -> converted logical tensors
    -> packed physical layout
    -> serialized container and shards
    -> loader validation and placement
    -> graph lowering and kernel dispatch
    -> stateful serving execution

Each arrow can introduce a new error class. A converter can use the wrong scale axis. A serializer can truncate packed tails. A loader can accept inconsistent shapes. A runtime can repack into a different nibble order. A graph compiler can place dequantization on the wrong side of a transform. A dispatch system can silently select a high-precision fallback and make the model correct but slow.

The original A34-A36 articles motivate quantization through model size, arithmetic, and method design. They necessarily treat formats and runtimes as the final implementation detail. M8.9 makes that detail a full article because, for sub-byte and structured representations, the artifact is part of the algorithm.

0x00A A Checkpoint Has Several Historical Roles

Traditional framework checkpoints often store named floating tensors and enough framework state to reconstruct a Python model. Large-model deployment separated those concerns:

• tensor-only containers such as safetensors emphasize bounded parsing, direct tensor regions, and lazy access;
• inference containers such as GGUF combine model metadata with typed tensor payloads expected by a specific executor family;
• graph formats such as ONNX encode operator semantics, including quantize/dequantize boundaries, while tensor data may live inline or externally;
• compiled runtimes can introduce another stage where a source checkpoint is converted into a topology- and hardware-specific engine.

No format is “the quantized format.” A container may know only that a byte region is U8, while companion tensors and configuration define two logical INT4 codes per byte. Another format may assign a versioned tensor type whose decoder is built into the runtime. A graph may express explicit Q/DQ semantics but leave physical packing to an execution provider.

0x00B Serialization Is Not The Same As Representation

Four layers must agree:

The safetensors specification is intentionally a simple safe tensor container and notes that sub-byte dtypes create alignment and API complications. GGUF is designed as an extensible inference format with typed tensor payloads and model metadata for GGML-based executors. Those different goals explain why comparing filename extensions by latency is a category error: execution speed belongs to the representation, loader, placement, and kernel path together.

0x00C Export And Loading Are Compiler Phases

A robust converter does more than save tensors. It freezes the quantizer’s semantic state, canonicalizes shapes and axes, packs codes, emits provenance, writes transactionally, reopens the production artifact, and proves parity against a reference decoder. A robust loader parses bounded metadata before large allocation, validates representation invariants, maps logical tensors to topology shards, admits a compatible kernel, and makes fallback observable.

This compiler framing also clarifies versioning. Container version, logical representation version, packing version, graph schema, runtime build, and kernel ABI can evolve independently. A hash proves which bytes were loaded. It does not prove that a new decoder interprets those bytes the same way.

0x00D A Checkpoint Is Not Yet An Executable Model

M8.1 defined a quantized tensor as codes plus reconstruction metadata and a consuming operator. M8.2-M8.8 then produced scalar, sparse, codebook, transformed, activation/KV, QAT, binary, and ternary candidates. None of those mechanisms answers the final integration question:

How are trained or reconstructed codes converted, serialized, loaded, validated, and matched to an executable kernel and serving ABI?

The answer is not a filename extension. A deployable low-bit model is a chain of four contracts:

1. the container contract says where bytes live and how bounds, alignment, sharding, and metadata are encoded;
2. the logical quantization contract says what the codes, scales, zero-points, groups, transforms, exceptions, and original shapes mean;
3. the graph contract says which tensor belongs to which operator and where quantize, dequantize, rotation, normalization, and cache transitions occur;
4. the kernel contract says which physical layout, dtypes, tile multiples, device capability, epilogue, and parallel placement a concrete implementation accepts.

Two files can hold the same reconstructed matrix while exposing incompatible kernel layouts. Conversely, two files can contain identical bytes while metadata causes loaders to interpret them as different tensors. M8.9 therefore treats export as a small compiler pipeline and loading as validation plus dispatch, not as generic save/load plumbing.

This article owns:

The local A34-A36 posts provide the quantization and deployment motivation. Format/runtime examples below are revision-bound snapshots, not a promise of present support on arbitrary installs. M8.2 supplies calibration/precision-map identity; M8.3 supplies transform order and fusion receipts. A student from M6.10 or pruned target from M6.11 enters this owner only when it also has a low-bit numerical representation. M6 retains its generic student/structural identity and independent loadability proof; no generic nonquantized production owner is assigned here.

0x01 The Four-Layer Artifact Contract

0x01A Layer 1: Container Bytes

A container answers questions such as:

How is the format identified and versioned?
Where does metadata end and tensor data begin?
What integer width stores offsets and lengths?
Are offsets absolute or relative to a data region?
What alignment and byte order are required?
Can tensor payloads live in external or sharded files?
How are duplicate names, overlap, truncation, and overflow rejected?

It does not necessarily define how an INT4 tensor is grouped, whether zero-points are packed, or which GEMM consumes it. Safetensors deliberately provides a simple tensor container. Quantization semantics often live in tensor names, companion tensors, and configuration. GGUF carries more model- and tensor-type semantics, but its ggml_type still names a specific representation whose implementation is versioned with the ecosystem. ONNX encodes quantization in graph operators or Q/DQ nodes, while large tensor bytes may be external.

0x01B Layer 2: Logical Quantized Tensor

For one logical weight matrix $W\in\mathbb{R}^{N\times K}$, a scalar affine artifact may need:

original_shape: [N, K]
code_dtype: uint4
signedness: unsigned code + zero point
group_axis: input_features
group_size: G
scale_dtype: fp16
scale_shape: [N, ceil(K/G)]
zero_dtype and zero_shape
clipping convention
tail-group convention
optional permutation and inverse permutation

An AQLM, SpQR, QuIP#, VPTQ, OneBit, or BitNet artifact needs different fields. A useful canonical descriptor therefore describes the representation family first and only then its members:

@dataclass(frozen=True)
class QuantizedTensorDescriptor:
    name: str
    original_shape: tuple[int, ...]
    representation: str          # affine, lut_sparse, additive, e8p, ternary...
    code_contract: "CodeContract"   # tagged scalar/vector/ternary/mixed streams
    rate_claims: tuple["RateClaim", ...]  # accounting, never a dispatch key
    group_axis: int | None
    group_size: int | None
    code_tensor: TensorRef
    qparam_tensors: tuple[TensorRef, ...]
    exception_tensors: tuple[TensorRef, ...]
    transform_tensors: tuple[TensorRef, ...]
    permutation: TensorRef | None
    decoder_version: str
    packing: "PackingDescriptor"
    operator: "OperatorRequirement"

The descriptor separates logical reconstruction from physical packing. That matters because a converter may keep the same codes and scales while preshuffling nibbles into a kernel tile order. CodeContract identifies the alphabet or codebook, selection unit, block/vector shape, axes, stream roles and decoder revision. A scalar width is valid within a scalar subtype, not a universal field for all families. A RateClaim names its numerator bits, denominator count/unit, scope, included metadata and coding scheme: alphabet information, fixed-slot payload, whole artifact and loaded memory are different rates. For example, ternary information $\log_2 3$, a two-bit physical slot and mixed-stream effective $2.91$ bits/original-weight cannot substitute for one another. A loader rejects a rate-only descriptor; the decoder and required streams remain mandatory.

0x01C Layer 3: Graph Semantics

A tensor cannot define the entire execution graph. These models are numerically different even if their packed weight bytes match:

dequantize(W4) -> X @ W^T
rotate(X) -> quantize(X) -> W4A4 GEMM -> inverse rotate(Y)
X @ dequantize(W4)^T -> add sparse correction
dequantize pre-RoPE K -> apply RoPE -> append/attend
append quantized post-RoPE K -> attend with fused dequantization

The graph contract must retain operator order, shared parameters, tied weights, normalization placement, cache format, and transform fusion. ONNX makes this explicit with nodes. PyTorch ecosystems may reconstruct it from model code plus quantization configuration. GGUF couples architecture metadata and tensor naming conventions to loader code. None is automatically safer than the others at the semantic layer; the loader must prove that the expected architecture and inventory match.

0x01D Layer 4: Executable Layout

A logical row-major uint4 array is not necessarily the array read by a fast kernel. The operator may require:

• K-major or N-major interleaving;
• 32-, 64-, or 128-element code blocks;
• qparams interleaved with code blocks;
• lanes preshuffled for warp/thread ownership;
• tile padding beyond the original shape;
• a scale type tied to a specific accumulator or epilogue;
• an activation dtype and group policy;
• minimum compute capability or CPU instruction set;
• tensor-parallel slicing before or after packing.

The physical object is better written as

where $\ell$ is a layout family, $v$ its version, and $t$ the target tile/topology. A generic bits=4 field does not invert this function.

0x02 Three Containers, Three Different Boundaries

0x02A Safetensors: A Bounded Tensor Byte Region

The official safetensors format begins with an unsigned 64-bit little-endian header length $N_h$, followed by $N_h$ bytes of UTF-8 JSON, followed by one byte buffer containing tensor payloads. Each tensor entry records dtype, shape, and data_offsets relative to the start of that buffer. Metadata is string-to-string. The specification rejects duplicate keys and requires the data buffer to be completely indexed without holes.

byte 0                         byte 8                    byte 8 + Nh
+----------------------------+-------------------------+--------------------+
| Nh: uint64 little endian   | JSON header: Nh bytes   | tensor byte buffer |
+----------------------------+-------------------------+--------------------+
                                                       ^ offsets are relative

For tensor $i$ with relative interval $[a_i,b_i)$, basic validation is

The expected byte count for a byte-addressable dense tensor is

That equation needs care for sub-byte formats. A logical uint4[N,K] has $NK/2$ payload bytes only after a nibble order, tail rule, and physical shape are chosen. The safetensors README explicitly notes that sub-byte dtypes make alignment difficult. In practice, many quantized ecosystems store packed data in byte or integer tensors and supply the missing quantization semantics through config and companion tensors.

Admission requires stored and expected extents to agree. This small validator checks one byte-addressable tensor; its signed-63-bit ceiling is a local resource limit, not a safetensors format version. Whole-buffer overlap/hole checks, header limits and inventory validation are additional gates. Rank-zero scalars and zero-sized tensors remain valid.

def validate_dense_extent(begin, end, data_bytes, shape, dtype_bits,
                          limit=(1 << 63) - 1):
    if (type(limit) is not int or limit < 0
            or any(type(v) is not int or v < 0 or v > limit
                   for v in (begin, end, data_bytes))
            or not begin <= end <= data_bytes):
        raise ValueError("invalid byte interval")
    if type(dtype_bits) is not int or dtype_bits <= 0 or dtype_bits % 8:
        raise ValueError("use physical byte-addressable dtype, not nominal bits")
    if not isinstance(shape, (tuple, list)) or any(type(d) is not int or d < 0 or d > limit for d in shape):
        raise ValueError("invalid physical shape")
    expected = 0 if 0 in shape else dtype_bits // 8
    if expected > limit:
        raise ValueError("dtype exceeds resource bound")
    for d in shape:
        if d and expected > limit // d:
            raise ValueError("byte count exceeds resource bound")
        expected *= d
    if end - begin != expected:
        raise ValueError("stored and expected extents disagree")
    return expected

Safetensors protects the parser from several pickle-style hazards and enables lazy tensor access. It does not prove that a scale axis is correct, that values are finite, that a packed nibble order matches a kernel, or that a tensor name belongs to the declared architecture.

0x02B GGUF: Model Metadata Plus Typed Tensor Payloads

GGUF version 3 starts with a header containing magic, version, tensor count, and metadata key/value count. Tensor-info records provide a name, dimensions, ggml_type, and an offset relative to the aligned tensor_data region. Metadata such as general.architecture, general.quantization_version, and general.alignment participates in interpretation.

For current offset $o$ and required alignment $A$, define

The double modulo behavior is important. If $o$ is already aligned, padding is zero rather than $A$. For payload size $B_i$, the recurrence is

The physical file includes padding

ggml_type values such as Q4, K-quant, IQ, ternary, or MXFP variants name concrete block representations, not merely scalar bit widths. The list evolves. A loader must gate on GGUF version, quantization version, tensor type, architecture, and its own decoder/kernel support rather than assuming that all .gguf files are interchangeable.

0x02C ONNX: Quantization As Graph Semantics

ONNX Runtime documents two major quantized graph forms:

QDQ is especially useful for QAT and for expressing where quantization boundaries occur. It is not a performance guarantee. A provider may fuse the pattern into a low-bit kernel, partially lower it, or execute explicit conversions around a higher-precision operator.

Large ONNX tensors can use external data. The tensor stores a relative location and may store offset, length, and checksum; data_location marks it external. This separates graph structure from payload transport but introduces path traversal, symlink, truncation, and lifecycle concerns. Page-aligned external offsets can improve memory mapping, yet page alignment is a transport optimization, not the low-bit kernel layout.

0x02D Comparison: Do Not Ask Which Extension Is Fastest

The right question is: which format preserves this representation and is consumed directly by the target runtime on the target hardware?

0x03 Deriving A Packed W4 Artifact

0x03A Logical Code Bytes

Let $Q\in\{0,\ldots,15\}^{N\times K}$ be four-bit codes. If every row is packed independently, K is even, and there is no tile padding, two codes share one byte:

The ceiling belongs inside the row count. Packing the entire matrix as one bitstream would let one row’s tail share a byte with the next row, which many row-addressable kernels cannot accept.

For group size $G$ along K,

With one FP16 scale and one FP16 zero value per group,

The simple representation size is

before tensor headers, padding, permutations, high-precision tensors, and shard metadata.

0x03B A 4096 x 4096 Example

Take $N=K=4096$, $G=128$, W4 codes, FP16 scales, and no zero-point:

There are

groups, so FP16 scales occupy

The affine-symmetric payload is therefore 8.25 MiB. FP16 zero values would add another 256 KiB, giving 8.5 MiB. Those numbers remain logical packed payload. A real tile layout can be larger.

0x03C Physical Tiles And Preshuffling

Suppose a kernel requires $N$ padded to $T_N=128$ and $K$ padded to $T_K=64$:

Its code payload is then

Within each tile, the byte index may be a nontrivial permutation:

That permutation aligns contiguous memory transactions with thread ownership. It is why a generic nibble pack and a Marlin-, ggml-, or custom-kernel pack cannot be substituted merely because both say W4.

0x03D Total Artifact Ledger

For tensors $i=1,\ldots,m$,

Effective bits per original scalar over quantized tensors with $P_q$ weights are

The attribution rule must state whether embeddings, output heads, norms, tokenizer files, graph metadata, codebooks, sparse coordinates, and duplicated shard headers are included. Comparing a payload-only number with a whole-model file size is invalid.

0x04 Export Is A Compiler Pipeline

0x04A Freeze Semantics Before Packing

The exporter must start from an immutable quantization specification, not infer a new one while saving:

def export_quantized_model(model, qspec, target):
    graph = freeze_eval_graph(model)
    assert_graph_matches_qspec(graph, qspec)

    logical = []
    for op in graph.quantized_operators:
        codes, qparams, extras = extract_frozen_representation(op)
        assert_reference_decode(codes, qparams, extras, op.reference_weight)
        logical.append(canonical_descriptor(op, codes, qparams, extras))

    placed = plan_parallel_placement(logical, target.topology)
    packed = [target.packer.pack(t) for t in placed]
    shards = assign_files(packed, target.file_policy)

    manifest = build_manifest(graph, qspec, target, packed, shards)
    write_transactionally(shards, manifest)
    return reopen_and_validate(manifest.path, target)

Re-quantizing a dequantized tensor through another library’s default min/max observer is not export. It changes codes, clipping, groups, and potentially quality. If a target packer needs only a permutation, prove

before checking floating reconstruction.

0x04B Transactional Writing

A robust writer does not expose a partially written final artifact:

1. write shards under temporary names;
2. flush and, where required, synchronize file contents;
3. compute byte lengths and cryptographic hashes;
4. write the manifest/index referencing those exact objects;
5. reopen every object through the production parser;
6. run inventory, byte, decode, and operator parity checks;
7. atomically publish the completed manifest or directory.

The manifest is committed last because it is the root of reachability. A manifest that points to only three of four completed shards is invalid; four shards without a published manifest are merely unreachable temporary state.

0x04C Export-Reopen Is A Mandatory Round Trip

In-memory converter tests miss offset, dtype, endianness, alignment, truncation, and name-mapping failures. The acceptance object is the reopened artifact:

def reopen_and_validate(path, target):
    raw = target.container.open(path)
    validate_container(raw)
    model = target.loader.load(raw, fallback="forbid")

    for case in golden_operator_cases():
        y_ref = reference_quantized_operator(case.x, case.logical_descriptor)
        y_run = model.run_operator(case.name, case.x)
        assert_close(y_run, y_ref, case.tolerance)

    assert_dispatch_trace_uses_expected_kernels(model, target.kernel_ids)
    return model

0x05 Loading Is An Untrusted Validation Pipeline

0x05A Parse Before Allocation

Container metadata controls lengths, shapes, offsets, and allocations. Treat it as untrusted even when the checkpoint came from an internal pipeline. The parser should use checked integer arithmetic:

def checked_tensor_bytes(shape, bits, limit):
    count = 1
    for dim in shape:
        require(0 <= dim <= MAX_DIM)
        count = checked_mul(count, dim)
    total_bits = checked_mul(count, bits)
    total_bytes = checked_ceil_div(total_bits, 8)
    require(total_bytes <= limit)
    return total_bytes

Validation order matters. Do not allocate prod(shape) before proving the multiplication cannot overflow and the byte interval is inside the file.

0x05B Container Gates

At minimum, reject:

• unknown mandatory container versions;
• unreasonable tensor/metadata counts;
• invalid UTF-8 or duplicate names where forbidden;
• negative, overflowing, out-of-bounds, overlapping, or misaligned intervals;
• a payload length inconsistent with physical dtype/shape;
• missing shards, path traversal, unexpected absolute paths, or escaping symlinks;
• checksum mismatch where the manifest declares a checksum;
• trailing or unindexed regions when the format forbids them.

Container success means only that the byte object is structurally admissible.

0x05C Semantic Gates

The next layer validates:

architecture id and revision
required tensor inventory
unexpected tensors under strict mode
tied tensor aliases
logical and physical shapes
qparam rank, axis, and group count
decoder/layout version
permutation bijection and inverse
sparse coordinate bounds and ordering
codebook dimensions
transform dimensions and graph placement
activation/KV schema and scale lifetime
tokenizer and vocabulary compatibility

For W4 groups along K, the loader should prove

or the explicitly declared broadcast-equivalent form. A scale tensor that broadcasts in Python but groups along N instead of K is a semantic corruption, not a valid convenience.

0x05D Kernel Admission Before Materialization

Only after structural and semantic validation should the loader select placement and kernels. This avoids filling HBM with an object that will later be rejected or silently dequantized.

def load_quantized_artifact(manifest, runtime):
    meta = parse_bounded_metadata(manifest)
    validate_container_graph(meta)
    validate_tensor_inventory(meta, runtime.model_schema)

    placement = runtime.topology.plan(meta.logical_tensors)
    plans = []
    for tensor in placement.local_tensors:
        capability = runtime.dispatch.resolve(tensor.operator_signature())
        require(capability.supported)
        require(capability.fallback_kind != "full_dequant")
        plans.append(materialization_plan(tensor, capability))

    storage = map_or_read_required_ranges(plans)
    model = materialize_exact_layouts(storage, plans)
    run_loader_parity_suite(model, meta)
    return model

0x06 File Shards Are Not Tensor-Parallel Shards

0x06A Transport Sharding

Hugging Face sharded checkpoints commonly use an index file with metadata.total_size and a weight_map from tensor name to shard filename. This solves file-size, download, and loading-management problems. It does not say how a tensor is split over GPUs.

{
  "metadata": {"total_size": 123456789},
  "weight_map": {
    "model.layers.0.mlp.up_proj.qweight": "model-00001-of-00004.safetensors",
    "model.layers.0.mlp.up_proj.scales": "model-00001-of-00004.safetensors"
  }
}

0x06B Semantic Parallel Placement

For $W\in\mathbb{R}^{N\times K}$:

• row/column naming differs by framework, so declare the actual sliced dimension;
• slicing N gives each rank a subset of output channels and corresponding qparam rows;
• slicing K gives each rank input-channel ranges and may require reduction of partial outputs;
• expert parallelism assigns whole or sliced experts;
• pipeline parallelism assigns layers;
• tied embeddings/output heads need consistent ownership.

If groups run along K, a K-slice boundary that cuts a group creates a choice:

1. align partitions to $G$;
2. duplicate the boundary qparam and retain partial-group semantics;
3. repack per rank using a new group partition, which changes reconstruction and requires parity revalidation.

The first is simplest but can constrain partition shapes. The third is not a byte-layout conversion; it can become a numerical conversion.

0x06C Read Amplification

Suppose a rank needs $B_{assigned}$ bytes but transport shards force it to read or map $B_{touched}$ bytes. Define

Whole-tensor file shards can yield high $R_{read}$ when every rank reads a full packed tensor before slicing. Rank-native export reduces startup traffic but creates topology-specific artifacts. An intermediate design stores tile-indexed ranges that can be read selectively while retaining a portable logical manifest.

0x07 Kernel Admission Is A Predicate, Not A Hope

0x07A Capability Signature

Define a requested operator signature

where:

A kernel $k$ is admissible only if

This is stricter than weight_bits == 4. Here layout_id is the exact (family, version, encoding) tuple, code_contract is a hashable tagged identity, and qparam_contract includes zero-point, grouping/axis and qparam dtypes. These fields are not effective-rate claims. The illustration assumes validated positive tile sizes/dimensions and exact topology identities; a production predicate also checks strides, alignment, workspace and backend-specific resources.

def supports_marlin_like(kernel, op):
    return all([
        op.representation in kernel.representations,
        op.layout_id in kernel.layout_ids,
        op.code_contract in kernel.code_contracts,
        op.qparam_contract in kernel.qparam_contracts,
        op.activation_dtype in kernel.activation_dtypes,
        op.group_size in kernel.group_sizes,
        op.scale_axis == kernel.scale_axis,
        op.accumulator_dtype == kernel.accumulator_dtype,
        op.output_dtype in kernel.output_dtypes,
        op.topology in kernel.topologies,
        op.local_n % kernel.tile_n == 0,
        op.local_k % kernel.tile_k == 0,
        op.device_capability >= kernel.min_capability,
        op.epilogue in kernel.epilogues,
    ])

0x07B Dispatch Lattice

Runtime selection should produce a trace such as:

layer=17.down_proj
requested=affine_w4a16:g128:layout_marlin_v2
shape=[N=4096,K=14336,T=1]
device=sm_90
candidates:
  marlin_v2_decode -> accepted
  marlin_v2_prefill -> rejected: T below threshold
  generic_w4 -> lower priority
  fp16_linear -> forbidden by policy
selected=marlin_v2_decode
repack=none

Without this trace, a successful load can hide a generic fallback. vLLM’s quantization integrations, TorchAO’s derived tensor/layout stack, the revision-specific TensorRT-LLM backends discussed below, and llama.cpp/GGUF type dispatch expose different APIs, but all must solve the same lattice: representation, layer type, dtype, shape, hardware, layout, and fallback.

0x07C Prefill And Decode May Select Different Kernels

M4 showed that phase shape changes arithmetic intensity. A W4 operator with $T=1$ decode can be dominated by weight traffic and unpack overhead; a large-$T$ prefill can amortize weights and favor a different tile. Dispatch policy may therefore be

The artifact must either support both kernels directly, permit a bounded secondary layout, or accept a phase-specific fallback. Duplicating two packed layouts increases HBM. Repacking per request is generally unacceptable. One universal layout may leave performance on the table.

0x07D Mixed 2/4/16-Bit Execution: The Schedule Is Part Of The Format

The paper Fast and Efficient 2-bit LLM Inference on GPU is a useful A36 source because it makes three artifact/runtime contracts concrete inside one weight matrix:

1. low-sensitivity regions use two-bit codes;
2. more sensitive regions use four-bit codes;
3. selected outliers remain in a sparse 16-bit stream.

This is intra-weight mixed precision, not merely a layer-level precision map. The source’s v4 section 3.3 first assigns precision from range/Hessian sensitivity, then performs QAT-like fine-tuning with groupwise quantize/dequantize in the forward pass. Final conversion uses that map, hierarchical scale quantization and sparse outliers reserved from the two-bit regions. Record the fine-tuned checkpoint, training data/objective, map and converter identity; these are source-specific provenance, not universal requirements for mixed precision. A loader must recover enough metadata to partition one logical matrix into kernel-visible streams:

@dataclass
class Mixed2416Weight:
    shape: tuple[int, int]
    packed_w2: Tensor
    packed_w4: Tensor
    precision_map: Tensor       # group/tile -> W2 or W4
    first_scale_codes: Tensor          # q_s^(1), not reconstructed scales
    first_weight_zero_points: Tensor  # z^(1)
    second_scales: Tensor              # numerical s^(2)
    second_zero_points: Tensor         # z^(2)
    first_group_axis: int             # source: input-channel direction
    first_group_size: int
    second_group_axis: int            # source: output-channel direction
    second_group_size: int
    precision_map_granularity: str
    qparam_packing: PackingDescriptor
    sparse_semantics: str             # replacement or correction, explicit
    sparse_fp16_values: Tensor
    sparse_column_indices: Tensor
    sparse_row_pointers: Tensor
    layout_id: tuple[str, str, str]    # family, version, encoding
    required_kernel: str

If first-level group scale $s_g^{(1)}$ is itself quantized by a second-level affine quantizer, reconstruction is nested:

then an ordinary weight code reconstructs as:

with a sparse high-precision replacement or correction applied under the artifact’s declared exception semantics. The qparam hierarchy saves metadata bytes but adds a dependency chain: second-level state must be decoded before first-level weights become numerical values.

For example, $q_s^{(1)}=5$, $z^{(2)}=1$, $s^{(2)}=0.25$ decode to $\widehat s^{(1)}=1$; with $q_w=3,z^{(1)}=2$, the weight is $1$. Swapping code and scale levels changes that decoder. The source groups weights along input channels and first-level scales along output channels; both maps must survive transposition, packing and sharding. Its Algorithm 1 names first-order scale codes and second-order numerical scales/zeros. The dataclass is a logical schema, not a byte-compatible reproduction of the source’s scale clipping and vector-aligned packing.

The paper’s asynchronous kernel overlaps parts of that chain. While one path loads packed first-level weights, another reconstructs first-level scales from second-level metadata; the input vector is loaded into shared/register state in parallel. The results meet before first-level dequantization and matrix-vector accumulation.

Source: “Fast and Efficient 2-bit LLM Inference on GPU,” Figure 8, arXiv:2311.16442v4, CC BY 4.0. The register/shared-memory schedule is source-reported for the paper’s kernel.

serial implementation:
    decode second-level qparams
    decode all first-level weights
    load input
    matrix-vector multiply

co-designed implementation:
    [decode second-level qparams] || [load first-level codes] || [load input]
    join -> first-level reconstruction -> fused matrix-vector accumulation

The format also aligns qparam fields and mixed streams to vector-load boundaries. A mathematically equivalent serializer that reorders two-bit/four-bit groups, changes sparse exception semantics, or drops second-level alignment can invalidate the kernel even when a slow reference decoder reconstructs the same dense matrix.

The paper reports an effective $2.91$ bits/weight including its studied scale/zero metadata and source-reported end-to-end speedups, including $1.74\times$ for one Llama2-7B setup. These are not generic W2 claims. They depend on the realized two/four/sixteen-bit proportions, source model, accuracy target, kernel, GPU, software, and phase shape. A local manifest records:

realized W2 and W4 group counts
sparse FP16 value/index count and row distribution
first- and second-level qparam bytes
alignment and padding bytes
selected async kernel and device capability
prefill/decode operator coverage and fallback rate
actual artifact bits/weight and loaded HBM

Kernel admission for this representation is stricter than supports_int2:

def supports_mixed_2416(kernel, artifact, op):
    return all([
        kernel.layout_id == artifact.layout_id,
        kernel.supports_precision_map(artifact.precision_map),
        kernel.supports_nested_qparams,
        kernel.supports_sparse_exception_semantics(artifact),
        op.local_k % kernel.vector_k == 0,
        op.device.shared_memory_per_block >= kernel.required_smem,
        op.device.registers_per_block >= kernel.required_registers_per_block,
    ])

This case study is why executable layout belongs in the capability signature. Representation quality, serialized rate, and the instruction schedule were optimized together.

0x08 Four Runtime Integration Strategies

0x08A Direct Prepacked Execution

The artifact stores exactly the kernel layout. Load maps or copies packed bytes and qparams to their final device buffers.

Advantages: low startup transformation cost, no extra pack workspace, byte identity can be checked.

Costs: kernel and layout version coupling, topology constraints, artifact proliferation across hardware generations.

0x08B Load-Time Repacking

The artifact stores canonical codes and qparams. Loader selects a kernel and repacks once.

Advantages: one logical artifact can target several kernels; source codes remain auditable.

Costs: startup latency, peak host/HBM workspace, packer version dependence, potentially expensive multi-rank repetition.

The required invariant is code preservation:

If group order, qparams, or clipping changes, the operation is conversion, not repacking.

0x08C Graph-Level QDQ Lowering

The artifact expresses quantization boundaries in a graph. An execution provider fuses recognized patterns.

Advantages: explicit graph semantics, provider portability, suitable QAT handoff.

Costs: fusion support is provider/version dependent; unrecognized QDQ can materialize larger tensors or conversions; graph success does not prove low-bit execution.

0x08D Full Dequantization Fallback

Loader reconstructs FP16/BF16 weights and invokes ordinary GEMM. This is useful for correctness debugging and unsupported-layer continuity. It invalidates claims of low-bit resident memory and often low-bit speed. Production policy should classify fallback:

If fraction $f$ of weighted operator work uses fallback, a simple latency ledger is

This is only a first-order model; fallbacks can also split fusion, allocate workspace, synchronize streams, and change batching.

0x09 Parity Must Climb A Ladder

0x09A Level 0: Byte And Inventory Parity

Check filenames, tensor names, physical dtypes/shapes, offsets, lengths, alignment, hashes, and manifest versions. This catches serialization corruption, not numerical meaning.

0x09B Level 1: Pack/Unpack Parity

For exact integer codes,

should satisfy elementwise $Q'=Q$. Test odd lengths, tail groups, minimum/maximum codes, every nibble/bit position, tile boundaries, and endian conversion.

0x09C Level 2: Reference Reconstruction

Construct

using a slow, explicit decoder from M8.1/M8.5/M8.8. Compare reopened artifact reconstruction with the pre-export reference. This catches qparam-axis, permutation, sparse-coordinate, codebook, and transform errors.

0x09D Level 3: Operator Parity

For an explicitly weight-only, bias-free linear operator with matching accumulator/epilogue conventions and test activations $X$,

The error

measures implementation differences against that declared numerical reference, separately from weight quantization error

By the triangle inequality,

If only end-to-end error is measured, a kernel bug can be mistaken for expected quantization loss.

The general oracle is the complete operator, not just reconstructed weights: $Y_{ref}=F_{quant,ref}(X;\mathcal{C})$ and $Y_{fp}=F_{fp}(X)$ at the same declared input/output boundary. Contract $\mathcal{C}$ includes activation Q/DQ, transforms, normalization placement, bias/epilogue and accumulation semantics. Define $E_{kernel}=\lVert Y_k-Y_{ref}\rVert$ and $E_{quant}=\lVert Y_{ref}-Y_{fp}\rVert$; the same triangle bound holds. With $X=0.6,W=1$ and unit-step activation rounding, the correct quantized operator returns $1$. Comparing it with $X\widehat W^\top=0.6$ would falsely label $0.4$ of intended activation error a kernel defect.

import numpy as np

def operator_parity(x, full_precision, quantized_reference, kernel):
    # Adapters must represent the same boundary and the declared full contract.
    yfp, yref, yk = [np.asarray(f(x), dtype=np.float64)
                     for f in (full_precision, quantized_reference, kernel)]
    if yfp.shape != yref.shape or yref.shape != yk.shape:
        raise ValueError("operator boundary shape mismatch")
    if not all(np.isfinite(y).all() for y in (yfp, yref, yk)):
        raise ValueError("nonfinite operator output")
    def distance(a, b):
        # Float64 Euclidean/Frobenius norm without squaring huge entries.
        with np.errstate(over="ignore", invalid="ignore"):
            delta = a - b
            value = np.hypot.reduce(delta.ravel(), initial=0.0)
        if not np.isfinite(delta).all() or not np.isfinite(value):
            raise ValueError("error metric is not representable in float64")
        return value
    return {"kernel": distance(yk, yref),
            "quantization": distance(yref, yfp),
            "total": distance(yk, yfp)}

This helper uses a float64 Euclidean/Frobenius norm and rejects unrepresentable error metrics even if each operator output is finite. Tolerance is numerical and backend-specific, not a proof of bit-exact execution. Synthetic adapters verify this separation; actual kernels still require reopened-artifact tests, dispatch traces and the stateful parity level below.

0x09E Level 4: Graph And Stateful Runtime Parity

Compare layer outputs, logits, generated token traces under deterministic policy, and state transitions. Activation/KV quantization also needs append, fork, reorder, truncate, prefix-reuse, and page-boundary tests from M8.6. Graph parity catches wrong tied weights, transform placement, normalization topology, and cache ABI even when every isolated linear operator passes.

0x09F Level 5: Dispatch And Resource Parity

Finally prove:

expected kernel ids were selected
fallback count and reason match policy
no persistent full-weight dequant buffer exists
resident HBM matches the component ledger
startup repack occurred only where declared
prefill/decode select supported phase kernels

0x0A Startup Time And Peak Memory

0x0AA Disk, Page Cache, Host, And HBM Are Different Ledgers

A memory-mappable file can avoid one host copy while still requiring GPU transfer and possible repacking. Track:

artifact bytes on disk
filesystem page-cache residency
mapped virtual address range
resident host pages
pinned transfer staging
canonical device codes
kernel-packed device codes
qparams and high-precision tensors
temporary repack workspace
full-dequant fallback buffers
runtime workspace and KV cache

For load-time repacking, an illustrative peak host ledger is

Peak HBM can be

if canonical and packed copies coexist. Streaming one layer at a time can reduce overlap:

at the cost of more synchronization and a more complex failure/retry protocol.

0x0AB Startup Timeline

Measure stages separately:

• discover: locate manifest, shards, tokenizer, and dependencies;
• parse: read bounded metadata;
• verify: hashes, schema, inventory, compatibility;
• read: fault/map/read required ranges;
• transfer: host-to-device movement;
• repack: canonical to kernel layout;
• engine: graph compilation or TensorRT engine deserialization/build;
• warmup: allocator, autotuning, graph capture, and representative calls.

A format with fast parsing can still have slow readiness because engine build or repacking dominates.

0x0B Runtime Ecosystems Implement Different Compilation Boundaries

0x0BA Safetensors Plus Quantization Configuration

Hugging Face-style checkpoints commonly combine safetensors with model configuration, quantization configuration, and implementation-specific tensor naming. The compressed-tensors project explicitly extends safetensors-based storage with sparse/quantized formats and quantization schemes. This illustrates the central point: the generic container remains one layer, while quantization groups, formats, and module targets live in a higher schema.

A Transformers flow that loads GGUF into ordinary PyTorch may dequantize it to FP32 according to the documented integration. That flow is useful for conversion or fine-tuning, but it does not preserve the packed GGUF execution path. The same file can therefore participate in two materially different runtimes.

0x0BB TorchAO: Derived Tensor Dtype And Layout

TorchAO describes quantization as a stack from algorithm/flow through quantized tensor/derived dtype, packing layout, and primitive/kernel. Quantized tensors can be tensor subclasses stored in a state_dict; official serialization guidance includes creating a meta-device model and loading with assign=True to avoid first materializing the full floating model.

The reference below pins TorchAO v0.17.0, commit 02105d46c61dc80a8c9d39d5836e827ba3af8439. The relocated serialization page displayed documentation version 0.17 when checked on 2026-09-29; that page label is not by itself a package-release identification. The historical /stable/serialization.html URL returned 404.

This design keeps quantized state inside PyTorch’s object model. It also means class/version availability, safe loading policy, and the specific layout subclass are part of compatibility. A plain integer tensor with identical bytes need not reconstruct the same dispatch behavior.

0x0BC vLLM: Runtime Method Dispatch

vLLM exposes multiple quantization integrations and a hardware support matrix that its documentation explicitly marks as subject to change. Quantization configuration participates in module replacement and method selection. Supported activation dtypes, minimum hardware capability, layer type, and quantization method affect admission.

The reusable lesson is not the current list of supported formats. It is that server startup must resolve artifact method + layer type + activation dtype + hardware + layout. Pin the runtime version and capture the selected method per layer.

For the inspected vLLM GGUF snapshot ced6857afa0ea7b2e3f0846a62e1394e90f15607, the GGUF instructions require the out-of-tree vllm-gguf-plugin and mark support experimental. A deployment must additionally pin that plugin’s installed version/commit; no particular plugin build was executed here. Do not reinterpret the snapshot as unconditional built-in GGUF support.

0x0BD TensorRT-LLM: Checkpoint Is Not Engine

This is the TensorRT engine-backend branch, not a timeless TensorRT-LLM requirement. Inspected release v1.2.1, commit 376f7e1bd8ed543f75014309e3fd4b237e9b0e73, still registers trtllm-build and retains its engine builder. Its source/checkpoint-to-engine workflow uses configuration and rank weight files, then hardware/runtime specialization and shape profiles.

That creates at least two versioned artifacts:

portable/source or TRT-LLM checkpoint
    -> converter parity
TRT-LLM rank checkpoint
    -> builder profile + hardware + runtime version
serialized TensorRT engine
    -> load and runtime parity

For that engine branch, publishing only the source checkpoint does not prove an engine can be built; publishing only an engine without its build inputs, profiles and version provenance weakens reproducibility and rollback.

A different inspected snapshot, migration documentation at commit c76f4a856447820aa1f994b053da7580d69b657d, describes removal of the TensorRT engine backend and its conversion/build commands, with PyTorch as the sole backend and direct loading of supported checkpoints. The published documentation advertised 1.3.0rc29 when inspected on 2026-09-29. These two observations do not establish the exact release cutover; in particular, do not infer that v1.2 removed a command present in the inspected v1.2.1 release. Record the actual runtime revision/backend, internal layouts, selected operators and supported model/hardware. TensorRT core itself is a separate product, not removed by this TensorRT-LLM backend change.

0x0BE GGUF And llama.cpp

GGUF’s architecture metadata and ggml_type tensor representations are closely aligned with ggml/llama.cpp execution. That can make a supported model and quantization type highly direct. It also means support is defined by the exact architecture, metadata keys, quantization version, tensor types, and compiled backend. A newer GGUF tensor type can be structurally valid but unsupported by an older loader.

0x0BF Design Comparison

These are alternative compilation boundaries, not a ranking from worst to best.

0x0C Versioning, Provenance, And Security

0x0CA Version Every Interpretation Layer

A useful manifest records:

container:
  kind: safetensors
  container_profile_version: numinens.safetensors.v1  # local, not upstream
  upstream_spec_revision: e246a2560645b7525f5775669ed816eb57c5bcc8
artifact_schema: numinens.quantized.v1
model:
  architecture: example_decoder
  revision: ...
  tokenizer_sha256: ...
quantization:
  family: affine_w4a16
  config_sha256: ...
packing:
  layout: target_layout
  layout_version: 2
  packer_commit: ...
runtime:
  name: ...
  tested_version: ...
  required_kernel_ids: [...]
topology:
  tp: 4
  pp: 1
  ep: 1
provenance:
  source_checkpoint_sha256: ...
  converter_commit: ...
  calibration_manifest_sha256: ...
files:
  - path: model-rank0.safetensors
    bytes: ...
    sha256: ...

format_version=1 cannot stand in for container, quantization schema, packing layout, graph, and runtime versions. They evolve at different rates.

0x0CB Hashes Prove Identity, Not Correctness

A cryptographic hash proves that the bytes match a recorded object. It does not prove that:

• the object was produced from the claimed source;
• the quantization axis is correct;
• the tokenizer belongs to the model;
• the kernel implements the declared decoder;
• the tensors are finite or free of malicious behavior in surrounding model code.

Use hashes together with signed provenance, bounded parsers, allowlisted architectures/operators, parity tests, and reproducible conversion records.

0x0CC External Data Is A Filesystem Trust Boundary

For ONNX external data or sharded manifests:

resolve only relative paths inside an approved root
reject '..' traversal and unexpected absolute paths
open with symlink/hardlink policy appropriate to the threat model
validate identity after open, not only before
check offset + length with overflow-safe arithmetic
verify expected file size and checksum
avoid arbitrary remote fetch during model load

The official ONNX external-data security guidance explicitly treats these as defense-in-depth concerns. A parser-safe container does not make path resolution safe.

0x0D Failure Lab

0x0E Boundaries And Handoffs

M8.9 consumes from M8.1-M8.8:

logical representation family and reference decoder
frozen codes, scales, zero-points, codebooks, sparse exceptions, and transforms
original tensor shapes and quantization axes
graph insertion and normalization/cache topology
high-precision tensor inventory
pack/unpack oracle and numerical test vectors
target operator shapes and phase requirements
training, calibration, PTQ, or native-low-bit provenance

M8.9 produces:

versioned artifact schema and concrete container
tensor inventory, physical shapes, offsets, alignment, and shard map
logical-to-physical packing descriptor
topology placement and qparam ownership
transactional converter and production loader paths
strict kernel capability/fallback policy
byte, code, reconstruction, operator, graph, and dispatch parity evidence
startup time and peak host/HBM ledgers
per-file hashes and source/converter/runtime provenance
runnable candidate artifacts for M8.10

M8.10 receives candidates that are already loadable and numerically attributable. It must not spend evaluation time discovering that one candidate secretly dequantizes to FP16 or uses the wrong scale axis. It owns workload-weighted quality, memory, prefill/decode latency, throughput, energy, operational coverage, promotion thresholds, monitoring, release, and rollback.

0x0F Claim Status

0x10 Publish Gate

• Container, logical quantization, graph, and kernel-layout contracts are separate.
• Safetensors byte structure and validation boundary are explicit.
• GGUF alignment and tensor-offset recurrence are derived.
• ONNX QOperator, QDQ, and external-data roles are distinguished.
• W4 code, qparam, padding, and effective-bit ledgers are derived with a numeric example.
• Logical codes and kernel preshuffling are not conflated.
• Export freezes existing codes/qparams rather than silently requantizing.
• Transactional write and production reopen are part of conversion acceptance.
• Loader validation is ordered before unsafe allocation and HBM materialization.
• File sharding, logical tensors, TP/EP placement, and kernel tiling are separate partitions.
• Group-boundary ownership and read amplification are explicit.
• Kernel admission includes layout, dtype, shape, hardware, topology, epilogue, and phase.
• Intra-weight mixed 2/4/16-bit execution includes hierarchical qparams, precision-map and sparse streams, asynchronous dequantization, admission predicates, and source-scoped performance claims.
• Direct packed, load-time repack, QDQ lowering, and full-dequant fallback are compared.
• The parity ladder separates byte, code, reconstruction, operator, graph, and dispatch evidence.
• Quantization error and kernel error are separated algebraically.
• Disk, page cache, host, transfer, HBM, repack, and runtime memory are separate ledgers.
• Safetensors/config, TorchAO, GGUF, ONNX, vLLM, and TensorRT-LLM integration boundaries are compared.
• Container/schema/layout/runtime versions, hashes, provenance, and external-path security are explicit.
• M8.10 retains evaluation, promotion, release, monitoring, and rollback.

0xFF References

Local Series And Transformer System Sources

• M8.1 - Quantized Numbers and the Kernel Contract - affine reconstruction, code/qparam tensors, packing, accumulation, and operator contracts. [TRANSFORMER SYSTEM]
• M8.2 - Calibration, Sensitivity and Precision Maps - calibration provenance and precision allocation supplied to artifact contracts. [TRANSFORMER SYSTEM]
• M8.3 - Outlier Geometry and Equivalent Transforms - transform identities, placement and eligible fusion sites that the artifact must preserve. [TRANSFORMER SYSTEM]
• M8.4 - Scalar Weight-Only PTQ - GPTQ/AWQ conversion state, group/block distinction, artifact requirements, and W4A16 kernel handoff. [TRANSFORMER SYSTEM]
• M8.5 - Extreme Weight Compression - sparse, codebook, transform, vector, reference-decoder, and effective-bit artifact contracts. [TRANSFORMER SYSTEM]
• M8.6 - Live-Data Quantization - activation and cache format, scale lifetime, page schema, state transitions, and fused consumer requirements. [TRANSFORMER SYSTEM]
• M8.7 - Quantization-Aware Training - fake-quant graph, trained qparams, prepare/convert, and export parity. [TRANSFORMER SYSTEM]
• M8.8 - Native Binary and Ternary Transformers - native code alphabets, graph topology, reference operators, physical-bit caveats, and packed-runtime handoff. [TRANSFORMER SYSTEM]
• Transformer Series (34) - Quantitative Fundamentals - local quantization arithmetic, mapping, and deployment motivation. [SOURCE-LOCAL]
• Transformer Series (35) - Fundamentals of Large Model Quantization - local PTQ/QAT, low-bit, and runtime context. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - local method and implementation survey used as the M8 source spine. [SOURCE-LOCAL]

Primary Papers And Source Implementations

• Fast and Efficient 2-bit LLM Inference on GPU: 2/4/16-bit in a Weight Matrix with Asynchronous Dequantization - intra-weight mixed precision, hierarchical qparams, exclusive sparse outliers, vector-aligned storage, and asynchronous GPU dequantization; Figure 8 is reused under CC BY 4.0. [PAPER, CC BY 4.0 FIGURE, SOURCE-REPORTED RESULTS]

Official Format Specifications

• Safetensors Format README - header length, JSON tensor metadata, relative data offsets, byte order, validation constraints, lazy loading, and sub-byte alignment caveat. [OFFICIAL SPECIFICATION]
• GGUF Specification - versioned header, metadata, tensor info, ggml_type, tensor-data alignment, offsets, and quantization-version metadata. [OFFICIAL SPECIFICATION, VERSION-SPECIFIC]
• ONNX External Data - external tensor locations, offsets, lengths, checksums, and memory-mapping guidance. [OFFICIAL SPECIFICATION]
• ONNX External Data Security - filesystem threat model and defense-in-depth guidance for external tensor data. [OFFICIAL SECURITY DOCUMENTATION]

Official Runtime And Framework Documentation

• ONNX Runtime Quantization - QOperator and QDQ representations, dynamic/static quantization, INT4 MatMulNBits, and provider-dependent execution. [OFFICIAL DOCUMENTATION]
• Hugging Face Big Model Loading - safetensors checkpoint shards, index JSON, total-size metadata, tensor-to-shard weight_map, and low-memory loading. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• Hugging Face Quantization Overview - quantizer integration and the inspected version’s integration/support boundary. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• Hugging Face GGUF Integration - documented GGUF-to-PyTorch flow that dequantizes weights for framework use. [OFFICIAL DOCUMENTATION, PINNED VERSION]
• compressed-tensors - sparse/quantized safetensors extension, quantization schemes, formats, targets, and model serialization integration. [OFFICIAL CODE, VERSION-SPECIFIC]
• TorchAO Quantization Overview - algorithm, derived quantized tensor, layout/packing, primitive, and kernel stack. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• TorchAO Serialization - tensor-subclass state dictionaries, meta-device construction, and assign=True loading. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• vLLM Quantization - revision-specific quantization integrations, hardware matrix and runtime support boundary. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• vLLM GGUF - the inspected GGUF plugin requirement and experimental support boundary. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• TensorRT-LLM Workflow - source model, TensorRT-LLM checkpoint, engine build, and runtime workflow. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• TensorRT-LLM Checkpoint - checkpoint configuration and rank weight-file organization. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• TensorRT-LLM Quantization - inspected quantization recipes and hardware/runtime constraints. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• llama.cpp - GGUF loader, model conversion, quantized tensor types, and CPU/GPU backend implementations. [OFFICIAL CODE, VERSION-SPECIFIC]