M8.10 - Deployment Evaluation, Release, and Rollback

M8.10 - Deployment Evaluation, Release, and Rollback

0x00 Compression Is Not A Release Decision

The first question in model compression is usually “how much smaller is it?” The last question must be “should this exact system receive user traffic?” Those questions are separated by quality statistics, workload behavior, resource limits, operational state, and reversibility.

The A34 introduction motivates quantization through memory pressure, latency, throughput, power, edge deployment, and long-context constraints. A35 and A36 then compare methods by quality and acceleration evidence. That background is necessary, but a production decision needs a stricter denominator. A method-level result does not identify the artifact bytes, runtime build, admitted kernel, topology, scheduler, cache policy, prompt processing, or workload under test.

Quantization is especially vulnerable to this gap because its gains are conditional. A smaller code payload may trigger load-time repacking. A supported shape may use a fast low-bit kernel while a rare shape falls back. Lower weight memory may increase batching until K/V or queueing becomes the new bottleneck. A quality average may remain stable while structured output, one language, or long-context retrieval regresses.

0x00A Evaluation Must Follow The Chain Of Claims

A release candidate makes several different claims:

Passing a later claim cannot repair a missing earlier one. An end-to-end benchmark can hide a fallback. A kernel unit test cannot prove model quality. A mean quality score cannot prove the absence of severe slice regressions. A throughput maximum cannot prove that requests satisfy latency objectives.

0x00B Offline Scores And Online Systems Answer Different Questions

Offline evaluation asks whether behavior changed under fixed cases and scoring. Serving evaluation asks what happens when requests arrive over time, contend for memory and compute, create variable-length state, and experience queueing. Production evaluation asks whether the traffic distribution and failure modes match either controlled experiment.

The evidence ladder should therefore move in one direction:

deterministic representation and operator parity
    -> frozen offline quality suite
    -> controlled latency and saturation curves
    -> shadow traffic
    -> sticky canary with a concurrent control
    -> staged production ramp
    -> continuous monitoring and rollback readiness

Skipping stages saves calendar time by spending uncertainty on users. Repeating stages without immutable candidate identity wastes evidence because results cannot be attributed to one system.

0x00C Release Engineering Is Part Of Numerical Correctness

For a stateless web binary, rollback can sometimes mean restoring the previous image. An LLM server carries state: prefix caches, K/V pages, adapter selection, scheduler queues, in-flight generations, compiled engines, and topology-specific allocations. A quantized release may change the schema or interpretation of that state.

The Google SRE canary guidance emphasizes representative, attributable metrics, concurrent control comparison, staged exposure, and rollback. M8.10 applies that discipline to model serving. Traffic assignment must be sticky enough to preserve a request’s state. Metrics must retain candidate identity. Rollback must define what happens to new traffic, in-flight requests, cache entries, and persistent side effects. The known-good path must remain warm enough to meet a recovery-time objective.

This is not operational decoration around the quantizer. If the runtime cannot identify which kernel executed, detect fallback, separate candidate metrics, or discard incompatible cache state, then the numerical contract cannot be defended in production.

0x00D A Smaller Checkpoint Has Not Earned Production Traffic

M8.9 ended with an artifact that can be reopened by the production loader, reconstructed correctly, dispatched to the intended kernels, and measured without a hidden fallback. That is necessary, but it is not a release decision.

A runnable candidate can still fail in several ways:

• its average benchmark score can hide a severe regression in one critical workload;
• its first-token latency can improve while inter-token latency or queueing gets worse;
• its throughput can rise only because requests violate their latency objectives;
• its disk footprint can shrink while peak HBM, host staging, or workspace memory grows;
• its joules per token can improve only because it emits shorter, incomplete answers;
• its offline behavior can pass while real traffic exposes unsupported shapes, long contexts, rare languages, or structured-output failures;
• its canary can look healthy in fleet-wide averages while the candidate population alone is failing;
• its rollback can restore old containers while leaving incompatible prefix-cache or K/V state in circulation.

The owner question for M8.10 is therefore:

How should quality, memory, latency, throughput, energy, workload coverage, promotion, monitoring, and rollback determine whether one exact quantized deployment candidate ships?

The evaluated object is not a method name such as GPTQ, AWQ, FP8, or W4A16. It is an immutable deployment tuple:

where

A = artifact manifest and hashes
R = runtime build and dependency image
K = admitted kernel set and dispatch policy
T = tensor/data/expert parallel topology and hardware
C = serving configuration, cache policy, and generation policy
W = declared workload and SLO contract

Changing any coordinate creates a new candidate. Repacking weights for another kernel, moving from one GPU to tensor parallelism, enabling prefix caching, changing the chat template, increasing max_model_len, or changing the output cap can alter quality, memory, latency, or energy. Reusing the old candidate ID would erase the causal boundary that evaluation needs.

Here $C$ may contain a frozen, evaluated set of allowed feature modes and switching rules. Switching within that admitted policy need not mint a new candidate, but still requires the mode’s scoped evidence and state-compatibility checks. Adding an untested mode, changing the switching rule, or changing artifact/cache representation changes the tuple. M5/M7 feature receipts are child evidence for a mode, not whole-release promotion authority. No mode switch may bypass M5.3’s compatibility/lifetime gates or the release policy.

This article owns:

The local A34-A36 posts motivate compression through memory, bandwidth, compute, latency, throughput, and deployment constraints. The local Compression Series P2/P3/P13/P14 posts provide the baseline, proxy-suite, registry, and reproducible-package discipline. External runtime examples below are source- and version-scoped, not a claim that every linked moving-head API was reverified together. The measurement-contract examples identify their pinned sources; actual deployment observations remain required.

0x01 Freeze The Candidate And Its Denominator

0x01A Baseline And Candidate Must Share A Measurement Contract

Let $\mathcal{D}_0$ be the accepted floating-point or currently deployed control and $\mathcal{D}_q$ the quantized candidate. The comparison is interpretable only if request construction, task cases, scoring, generation policy, and measurement boundaries are held fixed unless the experiment explicitly owns one of those changes.

For each run, freeze at least:

artifact and manifest hashes
runtime image digest, driver, framework, and kernel versions
hardware SKU, clock/power policy, topology, and placement
tokenizer, processor, chat template, and special-token policy
dataset revision, exact case IDs, media hashes, and prompt hashes
few-shot examples, seeds, sampling, stop strings, and output limits
context length, cache settings, scheduler settings, and parallelism
load-generator mode, arrival distribution, duration, and warmup
scorer implementation, normalizer, judge version, and rubric
energy tool, device scope, synchronization, and measurement window

This list is intentionally broader than model weights. A quantized candidate that uses a different prompt template or output policy is a compound product candidate. It can still be evaluated, but it cannot be credited as a pure quantization effect.

There are two valid comparison modes:

The release gate needs both. Ablations explain why the system moved. The final compound run proves that the exact package users will receive still passes after interactions among packing, runtime, cache, scheduler, prompt policy, and topology.

0x01B The Evaluation Manifest Is Executable State

A useful manifest makes accidental drift fail before expensive benchmarking:

candidate_id: illustrative-w4a8-fp8kv-a100-tp2-r17
parent_id: fp16-a100-tp2-r11
artifact_manifest_sha256: "..."
runtime_image_digest: "sha256:..."
kernel_allowlist_sha256: "..."
hardware:
  accelerator: NVIDIA-A100-SXM4-80GB
  count: 2
  tensor_parallel: 2
workload:
  suite_revision: eval-suite-2026-07-14
  case_manifest_sha256: "..."
  prompt_template_sha256: "..."
  request_length_buckets: [short, medium, long]
  modality_scope: multimodal
  transformation:
    status: required                 # text_only uses explicit not_applicable
    manifest_sha256: "..."           # retain the canonical manifest too
    preparation_lineage_sha256: "..."
generation:
  temperature: 0.0
  max_output_tokens: 512
serving:
  max_model_len: 32768
  prefix_cache: false
  kv_cache_dtype: fp8_e4m3
measurement:
  warmup_requests: 128
  load_seed: 8675309
  energy_scope: [gpu0, gpu1]
  statistical_plan_sha256: "..."     # estimand, unit, sidedness, attempt/look policy

The harness should compute the ID from canonicalized state or verify that a declared ID resolves to the same state. Human-readable names help navigation; content hashes prevent a familiar name from silently pointing at new bytes.

For multimodal workloads, consume X1.3’s transformation identity, not only a media hash. Retain the canonical processor manifest: decoder, resize/crop, frame sampling, normalization, unitizer geometry and flatten order, metadata schema/version. Bind media hashes (or explicitly trusted UUIDs and their trust domain), encoder checkpoint/selected layer, projector or resampler, output dtype/layout, fusion/media-row policy, and instruction/template identity when instruction-conditioned. The preparation lineage must identify the artifact and calibration inputs that used this contract. A transformed-input change is a new comparison unless the experiment explicitly owns it.

This small intake check verifies the retained transformation manifest against its frozen digest. It is not a media decoder, provenance authenticator, or complete schema validator; those are upstream responsibilities. text_only requires explicit non-applicability, not a silently missing multimodal record. Nested processor/media policies remain in the retained manifests referenced here.

def verify_transform_binding(scope, manifest, declared_digest, expected_digest):
    import hashlib
    import json
    if scope == "text_only":
        if manifest != {"status": "not_applicable"} or any(
                x is not None for x in (declared_digest, expected_digest)):
            raise ValueError("text-only transformation must be not_applicable")
        return True
    required = {
        "status", "processor_manifest_sha256", "media_identity_manifest_sha256",
        "encoder_checkpoint_and_layer", "projector_or_resampler",
        "output_dtype_layout", "fusion_media_row_policy",
        "instruction_conditioning", "preparation_lineage_sha256",
    }
    if (scope != "multimodal" or not isinstance(manifest, dict)
            or set(manifest) != required or manifest["status"] != "required"
            or any(not isinstance(v, str) or not v.strip() for v in manifest.values())):
        raise ValueError("incomplete transformation identity")
    payload = json.dumps(manifest, sort_keys=True, separators=(",", ":"),
                         ensure_ascii=True, allow_nan=False).encode("utf-8")
    actual = hashlib.sha256(payload).hexdigest()
    if actual != declared_digest or actual != expected_digest:
        raise ValueError("transformation contract mismatch")
    return True

Digest equality here proves only identity under this declared serializer. It does not grant K/V reuse or physical reclamation permission: M5.3 still owns compatibility and lifetime. The production intake resolves and verifies referenced manifests rather than trusting nonempty strings.

def admit_eval(candidate, baseline, contract):
    assert verify_manifest_hashes(candidate)
    assert candidate.loader_report.parity_passed
    assert candidate.loader_report.forbidden_fallbacks == []
    assert candidate.runtime_image_digest == contract.runtime_image_digest
    assert candidate.hardware == contract.hardware
    assert baseline.task_manifest == candidate.task_manifest
    assert baseline.prompt_manifest == candidate.prompt_manifest
    assert baseline.scorer_manifest == candidate.scorer_manifest
    return FrozenComparison(baseline, candidate, contract)

M8.10 should not rediscover an incorrect qparam axis or an FP16 fallback after three hours of load testing. Those are M8.9 admission failures. Conversely, M8.9 cannot declare a release merely because every expected kernel launched once. Execution correctness and product fitness are different gates.

Admission also preserves evidence class. A synthetic_contract_test can establish a schema, invariant, or failure-path property. A source_reported record describes another study’s result; unmeasured keeps an observation open. Neither becomes a measured release result by passing through a manifest. Quality, serving, memory, energy and recovery gates require observed_exact_tuple evidence for the frozen candidate/control, policy and measurement contract. Structural admission is necessary but cannot substitute for these observations.

Every consumed receipt binds those four identities, its evidence class, the measurement definition, raw-record hash and gate result. A trusted intake verifies the raw record and producer provenance, not just a self-declared label. Missing, malformed, nonfinite, wrong-identity or wrong-class evidence blocks eligibility. The reference in 0x0EB checks the final decision boundary; it does not authenticate an external producer or manufacture measurements.

0x02 Quality Is A Vector Before It Is A Scalar

0x02A Define What Every Score Means

Different tasks expose different model interfaces:

The score contract must say whether larger or smaller is better, how empty/invalid outputs are scored, how text is normalized, and whether the metric consumes generated text or model log-likelihoods. A score without its preprocessing and scorer version is not reproducible evidence.

0x02B Deriving Perplexity, And Its Boundary

For a token sequence $x_{1:M}$, autoregressive factorization gives

Products of many probabilities underflow and are difficult to average, so take the negative logarithm:

Divide by the number of scored tokens to obtain mean token surprise and exponentiate back to probability-space units:

Perplexity is useful because it examines every scored token rather than only one final answer. It is not a universal quality score. It must use identical tokenization, masking, sequence segmentation, context policy, and dataset bytes. Perplexities computed with different tokenizers have different denominators and are not directly comparable. A small PPL movement also does not prove that structured output, retrieval, safety, or tool use survived.

0x02C Pair The Same Cases

Suppose case $i$ has baseline score $s_{0,i}$ and candidate score $s_{q,i}$. Define the paired difference

For a lower-is-better metric such as NLL or latency, reverse the sign or define the margin in the original direction explicitly. Pairing matters because case difficulty is shared. Comparing two unpaired means includes variation from which examples happened to be sampled. Comparing $d_i$ asks how the candidate changed the same case.

For bucket $b$ containing $n_b$ cases,

A weighted headline can be reported as

but it must not replace critical-bucket floors. An easy, high-volume bucket can compensate for a catastrophic regression in code generation, multilingual OCR, safety, or long-context retrieval. The gate is conjunctive:

followed by optional weighted ranking among candidates that survive on selection evidence. This is a point-estimate floor; the final inferential gate uses the strict lower-bound rule in 0x03C. Freeze one candidate before accessing final acceptance data.

0x02D Macro, Micro, And Failure Taxonomy

For additive per-case scores here, a case-weighted mean weights every case equally; a bucket-weighted mean first scores each bucket and then uses equal or declared product weights. Do not generalize this shorthand to every metric: micro F1, for example, pools counts before computing the ratio. If one bucket contains 10,000 easy prompts and another contains 100 safety-critical prompts, a case-weighted mean nearly erases the second bucket. The workload contract declares the actual statistic and denominator.

Every failed case should also receive a failure tag. Useful tags include:

wrong semantic answer
ranking flip among close options
format or schema failure
empty output / premature stop / repetition
rare-token or multilingual corruption
long-context retrieval miss
OCR / number / table transcription error
refusal regression / over-refusal
runtime error / OOM / timeout / unsupported shape
non-deterministic mismatch requiring replay

Quantization can preserve the average score while changing the failure distribution. Per-case outputs, token counts, finish reasons, log probabilities when available, and failure tags should therefore be retained, not only the aggregate JSON.

0x02E Worked Quality Gate

Assume these product weights and non-inferiority margins. Scores and differences below use percentage points; a margin of 1.5 points is 0.015 on a unit-interval score. The last column checks point estimates only, not confidence-bound acceptance:

The weighted baseline is $74.950$ and the weighted candidate is $74.895$: a difference of $-0.055$ percentage points. The multilingual point estimate crosses its floor even though the headline barely moves. No uncertainty observations are supplied here, so the other rows do not establish inferential release passes. A critical product surface cannot be purchased with gains elsewhere.

0x02F Source Evidence: Method, Scale, And Task Difficulty Interact

The large evaluation study Exploring the Trade-Offs tests quantization across model scales from 1B to 405B parameters in the inspected arXiv v6, 13 benchmarks grouped into six task types, and several method families including GPTQ, AWQ, SmoothQuant, and FP8. Its value here is not a universal ranking. It demonstrates that the quality delta is an interaction:

The source reports that a larger quantized model can outperform a smaller full-precision model on many evaluated tasks, but not uniformly. Instruction following and hallucination-related behavior are among the cases where simple parameter-scale intuition can fail. It also reports limited discrimination from some aggregate conversational evaluation settings. Those are source-reported findings from its suite and software versions, not universal laws.

The deployment consequence is a factorial development/selection evaluation matrix, not a larger average benchmark. It must not repeatedly expose an untouched final gate:

for model_scale in candidate_scales:
    for quant_method in supported_methods:
        candidate = build_candidate(model_scale, quant_method)
        for bucket in task_buckets:
            result[candidate.id, bucket.id] = paired_evaluate(
                baseline=declared_baseline(candidate),
                candidate=candidate,
                cases=bucket.frozen_cases,
                scorer=bucket.pinned_scorer,
            )

        enforce_interface_validity(candidate)
        enforce_critical_bucket_floors(result[candidate.id])

Record benchmark difficulty strata where possible. Quantization can leave easy items unchanged while flipping close rankings, long chains, rare-token outputs, or instruction constraints. If a suite has no stable difficulty annotation, use preregistered proxies such as baseline margin, chain length, context length, answer rarity, or deterministic rubric complexity, then keep that stratification fixed before comparing candidates.

The study pins specific evaluation/runtime versions. That is evidence provenance, not a current recommendation. A local release must pin its own harness, tokenizer, templates, scorer, model revision, and runtime exactly as 0x01 requires.

0x03 Uncertainty, Non-Inferiority, And Selection Bias

0x03A A Point Estimate Does Not Define A Gate

The sample mean of paired differences estimates the population difference. Its ordinary standard error is

An approximate two-sided confidence interval is

This form is useful when paired differences are well behaved. Benchmark scores can be discrete, bounded, heavy-tailed, clustered by prompt family, or generated by a judge. A paired bootstrap preserves pairing without assuming Gaussian case observations, but does not guarantee nominal coverage for arbitrary statistics, tiny samples, rare events, degeneracy, or misspecified sampling units.

0x03B Deriving The Paired Bootstrap

The empirical dataset assigns probability $1/n$ to each paired case. A bootstrap sample draws $n$ case indices with replacement from that empirical distribution. For draw $r$:

The bootstrap distribution $\{\bar d^{*(r)}\}_{r=1}^{R}$ approximates repeated sampling of the paired mean. Percentiles of that distribution provide an interval. The important implementation detail is that an index selects both the baseline and candidate result for the same case. Resampling the two result arrays independently destroys covariance and produces the wrong uncertainty.

import numpy as np


def paired_resample_delta(base, cand, indices, statistic, quality_sign):
    # Each row is one independent unit (possibly an aggregated cluster).
    b = float(statistic([base[int(i)] for i in indices]))
    c = float(statistic([cand[int(i)] for i in indices]))
    delta = quality_sign * (c - b)  # normalize so positive means better
    if not np.isfinite([b, c, delta]).all():
        raise ValueError("nonfinite statistic")
    return delta


def paired_statistic_bootstrap(base, cand, statistic, *, statistic_id,
                               quality_sign=1, draws=50_000, seed=7, alpha=0.05):
    ids = [x["unit_id"] for x in base]
    if (len(ids) < 2 or any(not isinstance(i, str) or not i for i in ids)
            or len(set(ids)) != len(ids)
            or ids != [x["unit_id"] for x in cand]):
        raise ValueError("unique, aligned independent units required")
    if (quality_sign not in (-1, 1) or isinstance(quality_sign, bool)
            or type(draws) is not int or draws < 2
            or type(seed) is not int or seed < 0
            or not isinstance(statistic_id, str) or not statistic_id.strip()
            or isinstance(alpha, bool) or not np.isfinite(alpha) or not 0 < alpha < 0.5):
        raise ValueError("invalid declared bootstrap contract")
    rng = np.random.default_rng(seed)
    delta = paired_resample_delta(base, cand, range(len(ids)), statistic, quality_sign)
    boot = np.array([
        paired_resample_delta(base, cand, rng.integers(0, len(ids), len(ids)),
                              statistic, quality_sign)
        for _ in range(draws)
    ])
    return {
        "delta": delta,
        "one_sided_lower": float(np.quantile(boot, alpha, method="linear")),
        "two_sided_interval": np.quantile(boot, [alpha / 2, 1 - alpha / 2],
                                          method="linear").tolist(),
        "alpha": alpha, "draws": draws, "seed": seed,
        "statistic_id": statistic_id, "quality_sign": quality_sign,
        "method": "illustrative_percentile_linear", "unit_count": len(ids),
        "degenerate": bool(np.ptp(boot) == 0),
    }


def paired_mean_bootstrap(base, cand, metric, *, statistic_id, **contract):
    def mean_score(units):
        values = np.asarray([metric(x) for x in units], dtype=float)
        if not np.isfinite(values).all():
            raise ValueError("nonfinite per-unit score")
        return values.mean()
    return paired_statistic_bootstrap(base, cand, mean_score,
                                       statistic_id=statistic_id, **contract)


def corpus_nll(units):
    if not units or any(type(x["tokens"]) is not int or x["tokens"] <= 0
                        or not np.isfinite(x["loss_sum"]) or x["loss_sum"] < 0
                        for x in units):
        raise ValueError("positive scored-token counts and finite NLL sums required")
    return sum(x["loss_sum"] for x in units) / sum(x["tokens"] for x in units)

This executable teaching adapter uses NumPy arrays, not Torch tensors or device work. paired_mean_bootstrap estimates the mean of paired per-unit scores. For corpus NLL use corpus_nll with quality_sign=-1; for corpus perplexity recompute exp(corpus_nll(units)) on every resample, also with negative direction. Never average per-sequence perplexities as a substitute for corpus perplexity. Ratios, macro scores and fixed stratum weights similarly require their declared statistic to be recomputed, with a resampling design that preserves the intended strata.

For example, baseline (tokens, loss_sum) pairs (1, 0), (9, 18) and candidate pairs (1, 1), (9, 9) have equal mean per-sequence NLL, but corpus NLL changes from 1.8 to 1.0. This two-unit example proves arithmetic, not interval calibration. If cases share a document or conversation, aggregate/resample those independent units with the appropriate within-unit totals and weights; twenty correlated questions are not twenty independent observations.

At alpha=0.05, the illustrative lower bound uses the 5th percentile; the two-sided 95% interval uses the 2.5th and 97.5th. They are not interchangeable. SciPy 1.16.2’s bootstrap documentation distinguishes statistic recomputation, paired indices and interval sidedness. The percentile method above is a transparent example, not a generally calibrated release test; upstream method validation must address coverage, clustering, discreteness and degeneracy. A zero-width empirical interval is not proof of certainty. More bootstrap draws reduce simulation noise, not the uncertainty caused by too few independent observations.

0x03C Non-Inferiority Matches The Release Question

The release question is usually not “is the candidate exactly equal?” It is “can we rule out a regression larger than the tolerated margin?”

For a higher-is-better score with margin $\epsilon_b>0$, define

Let $L_b$ be the lower bound of a one-sided $(1-\alpha)$ interval. Bucket $b$ passes only when

Observe the difference between these outcomes:

point estimate above margin, lower CI below margin -> inconclusive under this plan
point estimate and lower CI above margin          -> non-inferior under declared margin
point estimate below margin                       -> reject or diagnose

“Not statistically significant” is not evidence of equivalence. A small, noisy test can fail to detect a large regression. Non-inferiority requires a predeclared margin and enough evidence to exclude that regression.

For the all-required-buckets claim, the global null is the union of the bucket nulls and acceptance is the intersection of their rejections. If any bucket null is true, accepting all buckets entails rejecting that true null, so a valid level-$\alpha$ test in each bucket bounds false global acceptance by $\alpha$ without assuming independence. There is no automatic Bonferroni/Holm penalty merely because every required bucket must pass. This is the intersection–union argument; the all-required endpoint distinction is also described in FDA’s October 2022 guidance, section III.C.1, cited for statistics, not as a deployment regulation.

This argument does not provide simultaneous interval coverage, validate a poorly calibrated bootstrap, or protect repeated candidate selection, method search, adaptive sample sizes, or repeated looks. Bind the required bucket set, estimands/directions, margins, independent units, interval method/sidedness, candidate/control identities, attempt ID and sampling/look policy before opening final evidence. A fixed-sample gate stops at its declared sample size even when inconclusive. Additional sampling requires a prespecified valid sequential procedure or a separately authorized new confirmation design that accounts for prior exposure; ordinary intervals do not become sequentially valid by being logged. Time-uniform confidence sequences provide one family of explicitly sequential methods; none is implemented by the percentile example.

0x03D Do Not Tune On The Final Gate

If twenty quantization variants are selected because they scored best on one suite and the winner is reported on the same suite, the reported score includes selection luck. Separate these evidence roles:

1. development diagnostics are visible and may guide clipping, mixed precision, recovery, or runtime work;
2. selection evidence compares candidates and Pareto frontiers, then freezes one candidate and its rule;
3. final acceptance evidence is untouched until that candidate and the decision plan are frozen;
4. production monitoring observes live distributions and delayed quality signals after release.

The ordinary final gate is a single frozen-candidate attempt with fixed cases, metrics, margins, weights and sample/look policy, consistent with M8.2’s evidence roles. It accepts or rejects that candidate; it does not choose the best of a frontier. Any multi-candidate or sequential alternative needs its own prespecified error-control design. Record all attempts and accesses, but recording reuse alone does not control error. Exposed final data becomes development/selection evidence rather than a fresh holdout for the next candidate.

For judge-based generation metrics, pin judge model/version, prompt, decoding settings, and rubric; randomize or blind candidate identity; retain raw judgments; and manually inspect disagreements near the gate. A judge is a measurement instrument with its own variance and bias, not ground truth.

0x04 Serving Benchmarks Must Reproduce A Workload

0x04A Latency Is A Timeline, Not One Number

For request $i$, let

a_i = arrival time at the service boundary
s_i = first scheduled execution time
f_i = first output-token delivery time
z_i = final response completion time
t_i,j = delivery time of output token j
O_i = number of output tokens

Then common metrics are

and for $O_i>1$,

Inter-token latency retains each gap:

TPOT is a request-level average after the first token; ITL is a distribution of individual streaming gaps. They answer different questions. The first token dominates perceived startup, while large ITL tails cause visible streaming stalls. A metric implementation may measure TTFT from arrival or from scheduling. Current vLLM documentation exposes TTFT, queue, prefill, decode, ITL, and E2E metrics, but the harness still needs to state the exact timestamp boundary and runtime version.

0x04B Offline, Closed-Loop, And Open-Loop Tests Answer Different Questions

A closed-loop benchmark can hide overload because response slowdown reduces the rate at which clients issue new work. An open-loop generator continues at the declared arrival rate $\lambda$, allowing the queue to reveal that service demand exceeds capacity. Current vLLM benchmark code supports request rate, Poisson-like and gamma arrival intervals, and linear or exponential ramp-up. That is a measurement control, not a default that automatically matches production.

0x04C Workload Shape Is A Joint Distribution

Do not publish one latency number without the workload that generated it. A useful request descriptor is

where $I$ is input length, $O$ output length, $M$ modality or media-token budget, $C$ concurrency/arrival context, $H$ cache-hit state, and $P$ product route or priority. Quantization changes can affect each region differently:

• weight-only compression often helps decode at small batch because weight reads dominate;
• W8A8/FP8 kernels may help compute-heavy prefill or larger batches;
• K/V compression mainly changes long-context capacity and attention traffic;
• packing tails and unsupported dimensions may create shape-specific fallback;
• online dynamic activation scaling adds work whose relative cost depends on tile size;
• multimodal encoders and projectors may remain high precision and dominate short outputs.

Construct a matrix rather than a single synthetic prompt:

input length:   128, 1k, 8k, 32k
output length:  16, 128, 512
arrival:        steady, Poisson, bursty, trace replay
cache:          cold, warm, prefix hit, no hit
mode:           text, multimodal, structured output, tool route
load:           isolated, moderate, SLO knee, overload

Use the production distribution for headline weighting and preserve the buckets. A candidate that wins the median short prompt but loses the 32k-context route needs a routing decision or a rejection, not an averaged victory.

0x04D Measurement Loop

def benchmark_candidate(candidate, workload, rates, contract):
    server = launch_exact_tuple(candidate)
    assert server.startup_report.kernel_ids == contract.kernel_allowlist
    assert server.startup_report.forbidden_fallbacks == []

    warm(server, workload.warmup_cases)
    reset_runtime_metrics(server)

    results = []
    for rate in rates:
        for repeat in range(contract.repeats):
            requests = workload.materialize(seed=(contract.seed, rate, repeat))
            trace = open_loop_run(
                server=server,
                requests=requests,
                request_rate=rate,
                arrival_process=workload.arrival_process,
                duration=contract.duration,
            )
            results.append(summarize_trace(trace, server.metrics_snapshot()))
    return results

Warmup requests should be excluded from measured windows. Run order should be randomized or interleaved across baseline and candidate so thermal drift, neighboring load, or clock policy does not align with only one system. Record failures and timeouts in the denominator; dropping them makes an overloaded candidate look fast. metrics_snapshot() must return this run’s window or be differenced against its start snapshot; a cumulative counter is not a per-run result.

0x05 Throughput Without SLOs Is Not Serving Capacity

0x05A Deriving Goodput

At offered rate $\lambda$, let request $i$ be acceptable only when it succeeds and satisfies every declared SLO:

The fraction of acceptable requests is

Multiplying offered work by its accepted fraction gives request goodput:

If the product requires at least $\rho$ of requests to meet the contract, the SLO-constrained operating capacity is

This derivation explains why raw throughput can keep rising after the service is no longer useful. More requests complete eventually, but an increasing fraction waits too long or streams too slowly.

For a finite observed window of duration $T$, use the realized offered rate $N_{offered}/T$, not the configured arrival-process parameter: $G_{req}=N_{good}/T=(N_{offered}/T)(N_{good}/N_{offered})$. Preserve the same request cohort and time basis in every factor, and retain requested rate, observed rate, offered/completed/good counts, failures and pending/censored outcomes. In the following window-cutoff example, all offered requests remain in the denominator; unresolved requests are bad at cutoff and later drain completions are not retroactively credited. A drained-cohort estimator needs its own explicit window policy and must not mix its counts with this denominator.

def cohort_goodput(records, duration_seconds, *, ttft_limit, tpot_limit,
                   e2e_limit, single_token_policy):
    import math
    def finite_nonnegative(x):
        return (isinstance(x, (int, float)) and not isinstance(x, bool)
                and math.isfinite(x) and x >= 0)
    if (not records or not finite_nonnegative(duration_seconds)
            or duration_seconds == 0 or single_token_policy != "not_applicable"
            or not all(finite_nonnegative(x) for x in (ttft_limit, tpot_limit, e2e_limit))):
        raise ValueError("invalid window or SLO policy")
    ids = [r["request_id"] for r in records]
    if any(not isinstance(i, str) or not i for i in ids) or len(set(ids)) != len(ids):
        raise ValueError("missing or duplicate offered request identity")
    completed = good = 0
    for r in records:
        if any(type(r[k]) is not bool for k in ("completed", "success", "interface_valid")):
            raise ValueError("invalid outcome flags")
        if r["success"] and not r["completed"]:
            raise ValueError("success before completion")
        completed += int(r["completed"])
        if not (r["completed"] and r["success"] and r["interface_valid"]):
            continue  # failures/timeouts/pending remain in offered denominator
        tokens = r["output_tokens"]
        if type(tokens) is not int or tokens < 0:
            raise ValueError("invalid token count")
        if tokens == 0:
            continue  # unexpected empty output is not accepted work
        if (not all(finite_nonnegative(r[k]) for k in ("ttft", "e2e"))
                or r["ttft"] > r["e2e"]):
            raise ValueError("invalid request timeline")
        if tokens > 1 and not finite_nonnegative(r["tpot"]):
            raise ValueError("missing streaming measurement")
        tpot_ok = tokens == 1 or r["tpot"] <= tpot_limit
        good += int(r["ttft"] <= ttft_limit and r["e2e"] <= e2e_limit and tpot_ok)
    offered = len(records)
    return {"offered": offered, "completed": completed, "good": good,
            "pending": offered - completed, "good_fraction": good / offered,
            "offered_rate": offered / duration_seconds,
            "completed_rate": completed / duration_seconds,
            "goodput": good / duration_seconds}

The single-token policy above treats TPOT as not applicable while still requiring valid success, TTFT and E2E; another policy must be declared before evaluation. Specify whether TPOT uses last-token delivery or response completion: the earlier $(z_i-f_i)/(O_i-1)$ includes any completion overhead in $z_i$. The pinned vLLM benchmark is one implementation example of per-request SLO conjunction and an explicit one-token convention, not a universal definition. The helper consumes normalized traces; it does not collect or authenticate timestamps.

0x05B Numeric Load Curve

Assume the contract requires at least 99% of offered requests jointly to succeed and meet TTFT at most 1.5 s, TPOT at most 45 ms when applicable, and E2E at most 20 s. Separate success and marginal percentile gates may also be required, but cannot substitute for this conjunction. With 10,000 requests and three disjoint sets of 90 violations, each marginal compliance is 99.1% while joint compliance is only 97.3%. The illustrative table uses consistent rate precision; its good fractions include E2E and failure outcomes not recoverable from the displayed marginal p99s alone:

The raw completion rate peaks near 15.1 req/s. The largest eligible tested offered rate is 12 req/s under these empirical gates, not a proven continuous optimum or confidence guarantee. Reporting only the larger number would overstate this tested usable rate by roughly 26%.

0x05C Tail Aggregation Must Preserve The Distribution

Report at least p50, p95, and p99 for user-visible latency, with confidence or repeated-run spread near the release gate. Do not average per-replica p99 values. Quantiles are nonlinear: the p99 of the union is not the mean of the p99s. Prometheus documentation explicitly distinguishes aggregatable histogram observations from precomputed summary quantiles that cannot be aggregated across replicated workers.

For classic histograms, sum bucket counts across the intended population and then apply histogram_quantile. Bucket boundaries determine quantile resolution. Native histograms change storage and aggregation details, so pin the monitoring stack version and schema. Labels must include candidate/control identity but avoid request IDs, prompt text, or other unbounded values.

0x06 Memory Savings Must Become Capacity

0x06A Six Memory Ledgers

A complete memory report separates:

M8.9 measured conversion and startup. M8.10 asks whether the complete service can sustain the workload with enough headroom. A candidate that saves 20 GiB of weights but consumes 18 GiB in repack/workspace at peak may not increase useful concurrency. During a rolling release, old and new replicas can coexist; temporary fleet and node memory must also fit.

0x06B From HBM To A First Capacity Bound

Let

M_HBM       = usable device memory under the deployment policy
M_fixed     = loaded weights + qparams + runtime + graphs
M_workspace = reserved kernel and communication workspace
M_headroom  = fragmentation, allocator, observability, and safety reserve
m_i         = dynamic activation + KV demand for request i (disjoint allocation model)

For disjoint per-request allocations, the admission constraint is

If every request had equal dynamic cost $\bar m$, a rough memory-only concurrency bound would be

Real requests do not have equal K/V state. For a full-KV cache with decoder layer count $L$, K/V head count $H_{kv}$, head dimension $d_h$, bytes per cache element $b_{kv}$ (not bits), and sequence length $S_i$, the payload approximation is

The actual admission policy must use request-specific lengths, page allocation, forks, shared prefixes, quantization metadata, and reserve policy from M5/M8.6. The simple bound is still valuable because it forces fixed, workspace, headroom, and per-request state into separate terms.

Here each additive term must denote disjoint storage. If a high-precision residual window replaces low-bit rows rather than duplicating them, split the sequence into its actual representation segments instead of charging the same rows once in the full low-bit term and again as full residual payload.

For heterogeneous or shared representations, replace the per-request sum by a ledger over unique physical allocations within each rank and memory tier:

Admission reserves a conservative bound on additional allocations not already present in that union, plus workspace and safety reserve. A shared prefix page is counted once on a rank; a replica on another rank is another allocation. Keep host and HBM ledgers separate. Full KV, MLA latent/RoPE streams, residual-precision windows, qparams, page rounding and padding must follow the actual cache schema, not a universal bytes-per-token guess. If a packed row/page already includes scales or padding, do not add them a second time.

def allocated_union_bytes(references, *, rank, tier):
    # Allocator IDs are unique in this rank/tier and retain their generation.
    unique = {}
    for ref in references:
        if ref["rank"] != rank or ref["tier"] != tier:
            raise ValueError("mixed allocation scope")
        key, size = ref["allocation_id"], ref["allocated_bytes"]
        if (not isinstance(key, str) or not key or type(size) is not int or size < 0
                or (key in unique and unique[key] != size)):
            raise ValueError("invalid or inconsistent allocation record")
        unique[key] = size  # repeated shared references do not allocate again
    return sum(unique.values())


def paged_payload_bytes(rows, rows_per_page, packed_row_bytes):
    if (any(type(x) is not int for x in (rows, rows_per_page, packed_row_bytes))
            or rows < 0 or rows_per_page <= 0 or packed_row_bytes <= 0):
        raise ValueError("invalid page geometry")
    return ((rows + rows_per_page - 1) // rows_per_page
            * rows_per_page * packed_row_bytes)

As one pinned representation example, FlashMLA’s FP8 sparse-cache interface describes 512 E4M3 bytes, four FP32 scales, and 64 BF16 RoPE elements: $512+4\times4+64\times2=656$ bytes per row. This is not a formula for all MLA caches. With an illustrative 16-row page, 17 rows occupy 20,992 payload bytes; two independent requests occupy 41,984, while sharing their first page leaves three unique pages, 31,488 bytes. The four scales are already inside 656. Allocator headers, separate metadata, replication and transient conversion buffers still need their own non-overlapping entries. These arithmetic helpers do not discover allocations, prove safe admission under concurrency, or free memory.

0x06C Memory Capacity Is Also An SLO Question

Fitting 64 long requests does not mean serving 64 long requests meets TTFT or TPOT. Define effective capacity as

Sweep context length and concurrency together. Record OOM, admission rejection, preemption, swap/offload, K/V utilization, queue depth, and fragmentation. Test cold and warm cache states. A cache hit changes prefill work; it should never silently enter one candidate’s benchmark but not the other’s.

0x07 Energy Needs A Boundary, A Window, And The Same Work

0x07A Deriving Joules From Power

Power is energy per unit time:

Therefore energy over a measurement window $[t_0,t_1]$ is

For sampled power $P_k$ at intervals $\Delta t_k$, a simple numerical approximation is

or trapezoidal integration when adjacent samples are used. A board telemetry counter may provide accumulated energy directly. The measurement report must state which path was used, its sampling behavior, and the device boundary.

Zeus documentation notes that GPU frameworks dispatch asynchronously and provides synchronized begin/end windows so work launched inside the interval is captured. Without synchronization, the CPU can close the window while GPU work is still executing. For distributed runs, aggregate the intended GPUs and nodes using synchronized windows. MLCommons power reporting is a useful contrast: its validated measurements cover full-system wall power for the benchmark scenario. GPU-only joules and wall-system joules answer different questions and must be labeled accordingly.

0x07B Do Not Normalize Away A Product Regression

Useful quantities include

and for a fixed accepted suite,

But output length is part of service behavior. A candidate that truncates answers can show excellent joules per request and poor quality. A verbose candidate can show similar joules per token while spending more total energy per completed task. Report total suite joules, joules/request, prompt and output token counts, and quality eligibility together.

A request accounting model is

This is not a claim that components are perfectly separable. It identifies the controls that make energy comparisons interpretable.

0x07C Idle Subtraction And Energy-Delay Products

If the server has substantial idle draw, an incremental estimate can be reported:

Idle subtraction is sensitive to how $P_{idle}$ is measured, power management, clocks, and neighboring processes. Keep raw window energy and incremental energy together rather than publishing only the adjusted value.

An energy-delay product

penalizes a candidate that saves energy by running much longer. Other exponents such as $J\cdot T^2$ encode stronger latency preference. These are policy utilities, not physical truths. Hard quality and SLO gates should run before any scalar energy-delay ranking.

0x07D Repetition And Variance

For repeated suite energies $J_1,\ldots,J_r$, report

High variation can indicate thermal drift, dynamic clocks, unrelated activity, inconsistent output lengths, or an insufficiently long window. Use pilot runs to choose repetitions/window duration for the required precision, then freeze the confirmatory plan. Do not keep extending an ordinary fixed-sample interval until it becomes narrow enough or passes; a sequential decision requires a valid prespecified method. Interleave baseline and candidate order and preserve each raw run. More data is useful, but its stopping rule is part of the evidence contract.

0x08 Select With Hard Gates, Then A Pareto Frontier

The frontier and worked rankings in this section use development/selection evidence. They choose the candidate and deployment policy to freeze. The untouched final gate then accepts or rejects that one candidate; passing final results must not be recycled to pick a new winner without a new valid design (0x03D). The diagram is a selection overview, not the exhaustive receipt schema of 0x0EB.

0x08A Why One Weighted Score Is Dangerous

A scalar objective such as

can rank candidates, but its units and weights encode policy. Worse, it allows a large energy gain to compensate for a safety or quality failure if applied too early. The safer order is:

1. artifact, loader, graph, and kernel parity
2. interface validity and critical quality floors
3. runtime compatibility and forbidden-fallback gate
4. memory/OOM/headroom gate
5. TTFT, TPOT/ITL, E2E, success, and goodput SLOs
6. declared hard energy/cost limits, if any
7. telemetry, isolation, retained-control capacity and rehearsed recovery gates
8. Pareto ranking of eligible candidates under the remaining soft objectives

Let $x$ and $y$ be eligible candidates and orient every objective so larger is better. Candidate $x$ Pareto-dominates $y$ when

Dominated candidates have no reason to ship under the measured objectives. Non-dominated candidates expose the real tradeoff: perhaps one has lower joules while another has higher goodput or wider hardware coverage.

Declare whether energy is a hard constraint or Pareto-only objective before opening results. In hard mode a measured violation rejects the candidate. In Pareto-only mode a valid energy observation affects ranking, not a fictitious passed energy limit. Operational readiness remains mandatory in both modes. A candidate with attractive measurements but missing rollback capacity, telemetry, isolation or a successful recovery rehearsal is not eligible. The reference policy below models energy explicitly; any additional hard cost limit needs its own typed field, validated measurement and predicate.

0x08B Worked Candidate Decision

Assume all percentages are relative to B0 and the quality gate is evaluated separately by bucket:

C3 is not rescued by best energy. C4 has a small artifact but does not execute the intended efficient path. C1 and C2 are both non-dominated if broad coverage matters. The release policy might ship C2 on the target fleet and retain C1 as the portable fallback, with distinct candidate IDs and monitoring populations.

0x08C Variants Need Deployment-Specific Comparisons

Paper variants should be compared at three levels:

Claims from papers cannot be copied across levels. A lower reconstruction error does not guarantee a supported kernel. A faster kernel microbenchmark does not guarantee higher service goodput. A smaller checkpoint does not guarantee lower loaded or peak memory. A source-reported speedup remains source-reported until reproduced on the exact deployment tuple.

0x08D Source Evidence: The Format Winner Can Change Along One Load Curve

The paper “Give Me BF16 or Give Me Death?” provides a useful empirical example of why M8.10 refuses a context-free format ranking. Its Figure 2 measures inter-token latency against queries per second for Llama-3.1-8B-Instruct on one A6000 during docstring generation. In that setup, W4A16-INT is the lower-latency path at light load, while W8A8-INT continues to higher throughput before its latency curve rises.

Source: “Give Me BF16 or Give Me Death?”, Figure 2, arXiv:2411.02355v4, CC BY 4.0. Setup shown by the source: Llama-3.1-8B-Instruct, 1x A6000, docstring generation. The curves are source-reported, not a universal ranking.

The crossover has a systems explanation. At low concurrency, a weight-only W4A16 path can reduce weight movement and produce a strong single-request latency point. Under continuous batching, W8A8 can use low-precision activation and weight matrix instructions across larger work units and sustain more aggregate queries before queueing and inter-token latency rise. The balance depends on prompt/decode lengths, batch formation, kernel support, model size, GPU architecture, and memory headroom.

The same paper evaluated vLLM 0.6.4.post1 across A6000, A100, and H100 systems. That pinned version is evidence provenance, not a recommended current runtime. A release candidate must rerun the experiment on its own runtime and kernel revisions.

Store the whole curve, not only each candidate’s best point:

def compare_formats_under_load(candidates, request_rates, workload):
    curves = {}
    for candidate in candidates:
        points = []
        for rate in request_rates:
            run = benchmark_open_loop(
                candidate=candidate,
                request_rate=rate,
                workload=workload,
            )
            points.append({
                "qps": run.successful_queries_per_second,
                "p50_itl_ms": run.p50_inter_token_latency_ms,
                "p99_itl_ms": run.p99_inter_token_latency_ms,
                "p99_ttft_ms": run.p99_ttft_ms,
                "goodput": run.goodput_under_slo,
                "fallback_rate": run.fallback_rate,
            })
        curves[candidate.id] = points
    return curves

A candidate wins the deployment only over the load region and SLO boundary the service actually needs. Even within one paper, tuned algorithm parameters can change the quality ordering inside a format. Therefore the release comparison has two nested questions:

within-format:
    which calibration, clipping, grouping, transform, and reconstruction recipe?

between-format:
    which resulting artifact/runtime path wins over the target workload curve?

Neither answer can be imported from a single perplexity table or peak-throughput number.

0x08E Fixed Model Bits Create A Different Scaling Question

The correctly linked source for The case for 4-bit precision: k-bit Inference Scaling Laws is arXiv 2212.09720, not 2305.13048. The latter identifier points to an unrelated RWKV paper. This matters because the source question is specific:

Given a fixed total number of parameter bits, should the budget buy a larger low-precision model or a smaller high-precision model?

Ignoring metadata for the moment, a model with $P$ parameters at $b$ bits has:

At fixed budget $B^*$:

Reducing $b$ buys more parameters, but increases quantization error. The source runs more than 35,000 zero-shot experiments across 19M—176B parameter models, several model families, and 3—8-bit weights with 16-bit inputs. In those studied settings, performance at fixed total model bits generally improved from 16 toward 4 bits and reversed at 3 bits, making four-bit weights the source’s empirical optimum. It also reports that small block sizes and datatype choice matter materially at four bits.

This is not a derivation that W4 always wins. Its boundary is:

studied:
    weight quantization with 16-bit inputs
    fixed total parameter-bit budget
    specified model families and zero-shot tasks
    studied block sizes and datatypes

not established by that source alone:
    W4A4 or K/V-cache deployment
    continuous-batching goodput
    codebook/sparse/transform metadata-free artifacts
    instruction-following, multimodal, safety, or long-context release fitness
    current hardware kernel support

M8 therefore uses the paper as a candidate-generation prior, not a release rule. The real budget is:

and the real decision includes loaded HBM, K/V state, kernel coverage, workload quality, and SLO goodput. A larger W4 model should be evaluated against a smaller W8/BF16 model at equal deployable resource budget, with the task-stratified gates from 0x02F.

0x09 The Release Evidence Bundle

Promotion should consume an immutable evidence bundle rather than a dashboard screenshot. At minimum it contains:

identity/
  candidate manifest, parent, artifact and runtime hashes
  hardware/topology, kernel allowlist, dispatch and fallback report

quality/
  suite/task/scorer revisions and exact case manifest
  per-case baseline and candidate outputs/scores
  bucket aggregates, margins, bootstrap seed/draws/intervals
  failure tags and signed decision

performance/
  workload manifest, arrival traces, warmup and run order
  per-request timestamps/tokens/finish reason
  rate-by-rate TTFT/ITL/TPOT/E2E/queue/goodput tables
  server metric snapshots and saturation plots

resources/
  disk/startup/loaded/peak/workspace/KV ledgers
  energy tool, scope, sync mode, raw windows, repetitions, variance

release/
  promotion policy version and gate outputs
  control candidate and rollback target
  cache/session compatibility declaration
  shadow/canary/ramp plan, owners, expiry, and approval record

The promotion record should state the policy version and the evidence hashes it evaluated. Changing a threshold after looking at results requires a new policy revision and decision record, but those records do not restore untouched data or statistical validity. Treat exposed final data as development/selection evidence; require a fresh authorized confirmation design or a prespecified procedure that accounts for the reuse. Otherwise release criteria become post-hoc explanations rather than gates.

@dataclass(frozen=True)
class PromotionRecord:
    candidate_id: str
    control_id: str
    policy_sha256: str
    evidence_sha256: str
    gate_results: tuple[GateResult, ...]
    rollback_target: str
    approved_stage: str
    approver: str
    timestamp_utc: str

The evidence bundle is also the rollback diagnosis package. If the canary fails, operators need to know exactly what changed, which workloads were covered offline, and which runtime/kernel path is active without reconstructing the experiment from shell history.

0x0A Release Is A State Machine, Not A Percentage Knob

0x0AA Offline Acceptance Comes Before Production Exposure

A canary is not a cheaper substitute for offline quality evaluation. Rare semantic regressions, safety failures, and long-tail workload gaps may require too much traffic or delayed labels to detect safely. The production sequence begins only after artifact parity, quality, resource, and load gates pass:

ASSEMBLED
  -> OFFLINE_ACCEPTED
  -> SHADOW
  -> CANARY_1
  -> CANARY_2
  -> RAMP
  -> FULL

any production stage
  -> PAUSE
  -> ROLLBACK
  -> QUARANTINE / DIAGNOSE

Google’s SRE canary guidance defines canarying as a partial, time-limited deployment evaluated against a control and recommends gradual stages, attributable metrics, representative duration/traffic, and per-version monitoring. Kubernetes Deployments provide one concrete mechanism for controlled rollout, status, revision history, pause/resume, and rollback, but M8.10’s release state machine is runtime-agnostic.

0x0AB Shadow Traffic

Shadowing copies representative requests to the candidate while the control still serves the user response. It can reveal parser, shape, memory, latency, fallback, and output-distribution problems without exposing candidate answers.

Shadowing has constraints:

• user data must be authorized for duplicate processing and logging;
• the candidate must not execute side effects, tools, purchases, writes, or externally visible callbacks;
• non-deterministic sampling makes response equality an invalid sole metric;
• duplicate work consumes compute and can interfere with the control if resources are shared;
• session state and cache keys must be isolated by candidate identity;
• shadow response bodies should follow retention and redaction policy.

For deterministic cases, compare exact outputs and log-probability drift. For stochastic generation, compare validity, length, finish reason, policy outcomes, structured fields, and distributional diagnostics under frozen seeds where possible. Shadowing validates production-shaped inputs; it does not prove causal quality if candidate and control do not process the same request state.

0x0AC Canary Assignment Must Preserve Stateful Semantics

For independent requests, random assignment can work. Multi-turn sessions, prefix caches, adapters, tools, and K/V-affecting state make request-level randomization unsafe. Route by a stable key:

then map a fixed range to the canary. Here $u$ can be a tenant, user, or session identifier chosen according to privacy and product semantics. Stable assignment keeps all turns in a conversation on one model version and makes exposure reproducible.

Prefix and K/V reuse must consume M5.3’s compatibility manifest, including semantic identity, physical layout, namespace and publication generation. The compact projection below is illustrative, not the complete compatibility gate:

cache_key = hash(
    model_artifact_id,
    runtime_cache_schema,
    exact_model_input_identity,
    adapter_id,
    kv_dtype_and_scale_policy,
    rope_and_position_config,
    prefix_tokens,
)

Do not share cache entries merely because model architecture and token IDs look similar. Quantized K/V format, scale ownership, RoPE stage, model weights, or adapter state can change the cached representation or the computation that consumes it.

Tokenizer/template revisions remain recorded provenance, but are not automatically equality-bearing when the exact model inputs are unchanged. Conversely, multimodal side inputs, transforms, positions or adapters can change cache meaning while placeholder token IDs stay equal. M8.6 contributes the quantized representation facts; M5.3 remains the reuse/lifetime owner. An M8 release namespace can impose stricter isolation, never weaken that gate.

0x0AD Canary Size, Duration, And Rare Failures

Canary stages need both an exposure cap and an evidence target. Five minutes at 1% traffic may be enough for crash loops and useless for a rare language route.

For a Bernoulli failure probability $p$, observing zero failures in $n$ independent requests has probability

The one-sided 95% upper bound after zero events solves

so

This “rule of three” means zero failures in 3,000 roughly independent requests only bounds the rate near 0.1%, under the sampling assumptions. Correlated traffic, delayed labels, and nonrepresentative routes weaken the inference. Do not claim that a tiny canary validates rare safety or quality events. Use offline targeted suites and stratified live exposure.

This bound also presumes the declared fixed-$n$ observation rule. Repeatedly inspecting it or restarting the count until a quiet interval appears is not covered by the calculation. Bind stage/look rules and delayed-label completeness before promotion. Conservative live rollback triggers may stop exposure immediately without claiming a fixed false-alarm rate; promotion claims need the valid inference policy, not just a minimum duration.

0x0AE Promotion Controller

def advance_release(release, policy, metrics):
    if (release.candidate_id != metrics.candidate_id
            or release.control_id != metrics.control_id):
        return hold(release, reason="measurement_identity_mismatch")

    absolute = evaluate_absolute_slos(metrics, policy.absolute_slos)
    relative = compare_canary_to_control(metrics, policy.relative_limits)
    coverage = evaluate_exposure(metrics, policy.required_buckets)

    if absolute.hard_failure or relative.hard_failure:
        return request_rollback(release, reason=first_failure(absolute, relative))

    if not coverage.complete or not metrics.minimum_duration_elapsed:
        return hold(release)

    if absolute.pass_all and relative.pass_all:
        return promote_to_next_stage(release)

    return pause_for_review(release)

Both absolute and relative gates are required. If a shared dependency fails, canary and control may degrade together; the relative comparison can look healthy while users suffer. If the canary lands on noisier machines, an absolute threshold alone can falsely blame the candidate. Attribution requires concurrent control, deployment labels, hardware labels, and known shared failure domains.

Only one candidate should normally progress through the same service population at a time. Overlapping canaries contaminate attribution and make rollback state harder to reason about.

This is decision pseudocode, not a direct router writer. Its promotion, pause and request_rollback helpers submit the transactional commands from 0x0ED; they cannot bypass an already-latched rollback. Live evidence must also be complete, fresh under the stage’s measurement-window policy, and valid for the current tuple. A previous offline pass is not permission to ignore a missing live telemetry or readiness receipt.

0x0B Production Observability Is Part Of The Artifact Contract

0x0BA Histograms, Counters, Gauges, And Traces

The runtime should expose enough information to prove that the deployed tuple still matches the evaluated one.

Current vLLM production documentation exposes histograms for TTFT, ITL, E2E, queue, prefill/decode time, and token counts, plus request counters and running/waiting/K/V gauges. Metric names change across versions, so pin the version and map runtime fields into stable service-level recording rules.

Candidate labels should include bounded identifiers such as:

service
candidate_id / control_id
runtime_image_version
hardware_class and topology
route/workload bucket
finish_reason and error_class
kernel_family or fallback_class

Never label metrics with prompt text, request ID, arbitrary user ID, unbounded model path, or raw sequence length. High-cardinality labels can make the monitoring system itself fail. Put detailed identities in sampled traces or logs and retain bounded buckets in metrics.

0x0BB Monitor The Inputs, Not Only The Outputs

Latency and quality change when the live workload changes. Track distributions for:

prompt and generated lengths
context position and K/V occupancy
media count and visual/audio token budget
language, route, task, adapter, and structured-output mode
prefix-cache hit/miss and shared-prefix depth
arrival rate, burstiness, concurrency, and queue depth

If canary prompts are shorter than control prompts, lower TTFT is not attributable to quantization. Compare stratified populations or reweight to a common distribution. Before/after comparisons are particularly risky because time of day, traffic mix, dependencies, and fleet load change together. Concurrent control traffic gives a better counterfactual.

0x0BC Quality Signals At Different Delays

Online quality arrives on several timescales:

Treat proxy signals as proxies. A shorter response is not automatically better, a higher acceptance rate is not automatically safer, and candidate/control textual disagreement is not automatically a candidate error. Preserve sampled evidence for diagnosis and continuously add confirmed production failures to a versioned regression suite without leaking the confirmation holdout into everyday tuning.

0x0BD Burn Rate Turns Error Budget Into A Release Signal

If the SLO target is $S$, the allowed bad-event fraction is $1-S$. For observed bad fraction $f_{bad}$ over a window, define burn rate

For a 99.9% SLO, the allowed fraction is 0.1%. A canary serving 0.5% bad requests burns at $B=5$. Multi-window alerts combine a fast window for severe regressions and a slow window for sustained smaller regressions. The exact thresholds are service policy. Candidate-relative deltas should accompany absolute burn because a healthy-looking ratio can hide both populations violating the SLO.

0x0BE The Runtime Path Must Remain Observable

M8.9 admitted an expected kernel set and fallback policy. M8.10 monitors that contract in production:

quantized operator count by kernel family
fallback count by reason and tensor/operator class
runtime repack or dequantization events
loaded artifact/manifest hash
K/V dtype and cache-schema version
graph compilation/recompilation and shape misses
workspace and allocator high-water marks
per-rank imbalance and collective time

A driver or runtime upgrade can change dispatch while artifact bytes stay constant. If the expected kernel share drops or a forbidden fallback appears, the deployment tuple changed in behavior and should pause or roll back even before user-level latency crosses a threshold.

0x0C Rollback Is A Forward State Transition To A Known Good Tuple

0x0CA Rollback Must Exist Before Promotion

A release is not rollback-ready merely because an old checkpoint remains in object storage. The previous tuple needs:

retained artifact and manifest
retained runtime image and deployment configuration
capacity to absorb traffic
compatible router and API schema
known cache/session policy
tested credentials and dependencies
an executable command or controller transition
owners, thresholds, and communications path

Kubernetes revision history illustrates the operational point: if old deployment revisions are deleted, the controller cannot undo to them. Model serving adds larger artifacts, warmup, graph compilation, cache state, and accelerator capacity, so retaining metadata without ready bytes or capacity may still produce an unacceptably slow recovery.

0x0CB Separate New Traffic, In-Flight Work, And State

Rollback has at least four tracks:

1. route: stop assigning new sessions/requests to the candidate;
2. drain: allow compatible in-flight requests to finish or cancel them according to the API contract;
3. state: prevent candidate caches, sessions, or artifacts from contaminating the control;
4. capacity: warm or scale the rollback target before candidate capacity disappears.

For stateless single-turn inference, route and drain may be simple. For streaming or multi-turn service, separate logical quarantine from physical reclamation. Quarantine fences new admissions, shared lookups and publication immediately; it does not free bytes held by a legal reader, kernel, transport operation or session lease. Consume M5.1/M5.3’s generation and lease rules rather than replacing them with a timeout.

The following orchestration assumes a durable rollback command and an idempotent, generation-fenced adapter. Its operations reconcile the same intent after restart. A failed step never authorizes later reclamation:

def reconcile_rollback(release, reason, ops):
    # Validates the durable rollback intent and its immutable identities.
    fence = ops.rollback_fence(release)
    ops.stop_candidate_admission(fence)
    ops.quarantine_namespace(fence)  # no new leases/hits/shared publication
    ops.record_incident_once(fence, reason)

    if not ops.control_capacity_ready(fence):
        # Backpressure/retry new traffic; do not reopen the candidate.
        return "PENDING_CONTROL_CAPACITY"
    ops.route_new_work_to_control(fence)
    ops.drain_or_cancel_sessions(fence, release.drain_deadline)

    if not ops.users_quiescent(fence):
        if ops.deadline_elapsed(release.drain_deadline):
            ops.request_cancellation(fence)
        # A cancel request is not an acknowledgement or a device fence.
        return "PENDING_DRAIN"

    # The adapter atomically rechecks quarantine + zero legal users.
    # A crash/retry may repeat this call, but cannot free the bytes twice.
    if not ops.reclaim_if_quiescent(fence):
        return "PENDING_DRAIN"
    if not ops.control_slos_verified(fence):
        return "PENDING_CONTROL_VERIFICATION"
    ops.scale_down_candidate_once(fence)
    return "ROLLED_BACK"

Existing legal readers may finish using their retained bytes. A session may drain only under its pre-existing private ownership contract; it cannot publish a new shared-cache entry or acquire a new quarantined-generation lease. A late producer with an old generation fails publication even if its kernel completed successfully. Reclamation waits for acknowledged closure of every reader, transfer and device user. If cancellation stalls, the state remains quarantined and allocated and the controller escalates; a deadline does not make reuse safe.

Do not copy candidate K/V rows into the control unless an explicit cross-version compatibility proof exists. For a subsequent turn, replay the authorized conversation inputs under the control’s processor/model contract, or return the declared retry/termination outcome. Never relabel foreign K/V. Replaying input is not replaying tools or other persistent side effects. M7’s same-target target-only fallback does not prove portability into a different quantized target. Namespace fencing is part of rollback; physical cleanup may legitimately finish later.

0x0CC Binary, Configuration, And Data Rollback Are Different

A quantized release can change:

Kubernetes documentation notes that a Deployment rollback restores the Pod template revision, not every external configuration or data mutation. The general lesson is broader: list all changed components and give each a recovery path. A model binary rollback cannot undo tool side effects or user-visible writes generated while the candidate was live.

0x0CD Recovery Time Budget

Decompose rollback time as

Parallel work means wall-clock recovery is not always the arithmetic sum, but the decomposition exposes what to measure. Prewarmed control capacity reduces $T_{warm}$ at resource cost. Automated hard triggers reduce $T_{decide}$ but require trustworthy metrics. Long streaming requests increase $T_{drain}$ unless retry or cancellation semantics are explicit.

Set a recovery objective, rehearse rollback, and measure each term. A runbook that has never been executed is a hypothesis.

0x0CE Rollback Does Not Mean Immediate Redeployment

After rollback:

candidate state = quarantined
promotion record = failed with trigger and evidence
traffic exposure = zero except controlled diagnosis
artifacts/logs = retained under incident policy
new attempt = new candidate ID or explicit evidence revision

Do not modify the failed artifact in place and reuse its version. That erases which bytes caused the incident and can make caches, dashboards, and audit records ambiguous.

0x0D Deployment Modes Change The Winner

There is no universal quantized winner because different modes constrain different objectives:

This is where paper-variant comparisons become product decisions. W4A16 may dominate small-batch decode while W8A8 wins compute-heavy prefill. K/V quantization may unlock long-context concurrency while adding dequantization and qparam traffic that does not help short prompts. An extreme codebook representation may minimize storage but lose to a simpler format whose kernel and loader are mature. Native ternary arithmetic may require a dedicated training and hardware/runtime stack rather than replacing a PTQ artifact in an existing service.

The release registry should allow route-specific winners if operational complexity is justified. Each route still has an immutable tuple, quality gates, monitoring population, and rollback target. “Automatic best quantization” without a workload and runtime contract is not a meaningful deployment result.

0x0E End-To-End Implementation Blueprint

0x0EA Registry Schema

@dataclass(frozen=True)
class EvaluationRow:
    candidate_id: str
    parent_id: str
    artifact_hash: str
    runtime_digest: str
    topology_hash: str
    config_hash: str
    workload_hash: str
    quality_evidence_hash: str
    load_evidence_hash: str
    resource_evidence_hash: str
    critical_quality_pass: bool
    interface_pass: bool
    forbidden_fallback_count: int
    peak_hbm_bytes: int
    max_slo_rate: float
    energy_per_accepted_request_j: float
    operational_coverage: tuple[str, ...]
    decision: str       # diagnose, reject, eligible, shadow, canary, full, rollback
    decision_reason: str

The row references raw evidence rather than flattening it into a few numbers. Different task metrics cannot be faithfully reduced to one float, and load curves cannot be reduced to one latency percentile.

0x0EB Evaluation Pipeline

Measurement adapters still run interface/dispatch, paired quality, memory, load, synchronized energy and rollback-rehearsal jobs against the admitted comparison. They may short-circuit expensive work after a failure. The final eligibility decision then consumes their verified receipts. The following small reference is executable independently of a serving runtime:

from dataclasses import asdict, dataclass
from hashlib import sha256
import json
from math import isfinite
from typing import Optional


@dataclass(frozen=True)
class ReleasePolicy:
    critical_margins: tuple  # unique (bucket_name, positive margin) pairs
    energy_mode: str         # "hard" or "pareto_only"
    energy_limit_j: Optional[float]
    recovery_limit_s: float


@dataclass(frozen=True)
class GateReceipt:
    identity: tuple  # candidate, control, policy hash, measurement-contract hash
    evidence_class: str
    value: float
    raw_sha256: str


BOOLEAN_GATES = (
    "interface", "runtime", "memory", "serving", "control_capacity",
    "telemetry", "isolation", "rollback_rehearsal",
)


def finite_number(value):
    try:
        return type(value) in (int, float) and isfinite(value)
    except OverflowError:
        return False


def policy_digest(policy):
    # Canonical local decision-policy projection; not a runtime serializer.
    raw = json.dumps(asdict(policy), sort_keys=True, separators=(",", ":"),
                     allow_nan=False).encode("utf-8")
    return sha256(raw).hexdigest()


def evaluate_for_release(identity, policy, receipts):
    def reject(reason):
        return {"status": "reject", "reason": reason}

    if (type(identity) is not tuple or len(identity) != 4
            or any(type(x) is not str or not x for x in identity)):
        return reject("identity")
    if type(policy) is not ReleasePolicy or type(receipts) is not dict:
        return reject("policy_or_receipts")
    if (policy.energy_mode not in ("hard", "pareto_only")
            or not finite_number(policy.recovery_limit_s)
            or policy.recovery_limit_s <= 0):
        return reject("policy")
    if policy.energy_mode == "hard":
        if (not finite_number(policy.energy_limit_j)
                or policy.energy_limit_j <= 0):
            return reject("energy_policy")
    elif policy.energy_limit_j is not None:
        return reject("ambiguous_energy_policy")
    pairs = policy.critical_margins
    if (type(pairs) is not tuple or not pairs
            or any(type(p) is not tuple or len(p) != 2 for p in pairs)):
        return reject("quality_policy")
    names = [p[0] for p in pairs]
    if (any(type(n) is not str or not n for n in names)
            or len(set(names)) != len(names)
            or any(not finite_number(m) or m <= 0 for _, m in pairs)):
        return reject("quality_policy")
    if policy_digest(policy) != identity[2]:
        return reject("policy_identity")

    required = list(BOOLEAN_GATES) + ["quality:" + n for n in names]
    required += ["recovery_seconds", "energy_j_per_accepted_request"]
    values = {}
    for name in required:
        r = receipts.get(name)
        if (not isinstance(r, GateReceipt) or r.identity != identity
                or r.evidence_class != "observed_exact_tuple"
                or not finite_number(r.value)
                or type(r.raw_sha256) is not str or len(r.raw_sha256) != 64
                or any(c not in "0123456789abcdef" for c in r.raw_sha256)):
            return reject("invalid_receipt:" + name)
        values[name] = r.value
    for name in BOOLEAN_GATES:
        if values[name] != 1:  # exact 0/1 predicate, never generic truthiness
            return reject("hard_gate:" + name)
    for name, margin in pairs:
        if values["quality:" + name] <= -margin:
            return reject("quality_floor:" + name)
    recovery = values["recovery_seconds"]
    if recovery < 0 or recovery > policy.recovery_limit_s:
        return reject("recovery_limit")
    energy = values["energy_j_per_accepted_request"]
    if energy < 0:
        return reject("invalid_energy")
    if policy.energy_mode == "hard" and energy > policy.energy_limit_j:
        return reject("energy_limit")
    return {
        "status": "eligible", "identity": identity,
        "receipt_hashes": tuple((n, receipts[n].raw_sha256) for n in required),
        "energy_role": policy.energy_mode,
        "energy_j_per_accepted_request": energy,
    }

The boolean receipts summarize complete preregistered predicates, not arbitrary success flags: runtime includes forbidden-fallback/dispatch checks; memory includes OOM/headroom; serving includes the required workload and SLO coverage. Each quality receipt is the declared bucket’s lower non-inferiority bound. Rehearsal and recovery receipts identify the retained control and the same rollback scenario, including telemetry, isolation and continuation behavior. The measurement-contract hash binds definitions, windows, units, required cases and freshness rules. Intake rejects incomplete raw evidence before constructing a receipt; a hash-shaped string alone proves nothing.

The policy coordinate is the canonical digest of the actual typed decision policy, checked locally. Changing a margin, energy mode, bound or recovery limit invalidates the old identity and receipts. The separate measurement contract binds the upstream predicates and is verified by intake. Different numeric encodings in this deliberately narrow serializer may produce a new digest; do not treat it as a universal serialization standard.

This closed policy schema does not silently accept unknown hard limits. Extend its typed validation and predicates together when adding one. Pareto energy still needs a valid observation, but has no hard bound to report as passed. Eligibility seals the selected receipt hashes; it neither routes traffic nor replaces the subsequent stage/approval transaction. The local tests use synthetic fixtures, including fixtures exercising the observed class branch; they are not actual workload or recovery measurements.

0x0EC Release Policy As Data

The measurement-contract digest in the typed evaluator binds a separate validated statistical plan. For each quality bucket that plan fixes the estimand/direction, independent unit, one-sided method and alpha; it also binds candidate/control, attempt, final-data access and fixed-sample or valid sequential look policy. Intake checks these facts before issuing a lower-bound receipt. The confidence fields below alone are not that plan, and the percentile teaching helper is not an automatically approved production method.

quality:
  critical_buckets:
    code: {margin: 0.01, confidence: 0.95}
    multilingual: {margin: 0.015, confidence: 0.95}
    long_context: {margin: 0.02, confidence: 0.95}
  invalid_output_rate_max: 0.001
serving:
  ttft_p99_seconds: 1.5
  tpot_p99_seconds: 0.045
  e2e_p99_seconds: 20
  success_rate_min: 0.999
  required_good_fraction: 0.99
memory:
  hbm_headroom_fraction_min: 0.08
  oom_count: 0
runtime:
  forbidden_fallback_count: 0
energy:
  mode: hard                     # or pareto_only, with no hard limit
  joules_per_accepted_request_max: 12.0
readiness:
  retained_control_capacity_required: true
  telemetry_complete_required: true
  cache_session_isolation_required: true
  rollback_rehearsal_required: true
  recovery_seconds_max: 60.0
release:
  stages:
    - {name: shadow, fraction: 0.0, min_requests: 10000}
    - {name: canary_1, fraction: 0.01, min_requests: 5000}
    - {name: canary_2, fraction: 0.05, min_requests: 25000}
    - {name: ramp, fraction: 0.25, min_duration_minutes: 60}
  rollback_target: fp16-a100-tp2-r11

The numbers above are illustrative, not recommended defaults. Real margins and SLOs come from product impact, baseline variability, traffic volume, hardware economics, and error-budget policy. A strict configuration adapter maps this document into the typed policy and named receipt contracts above; missing or unknown required fields are errors, not defaults to success.

0x0ED Release Transitions Must Be Idempotent

Controllers retry. A read followed by an unconditional atomic write is not compare-and-swap: two readers can both pass the revision check and overwrite one another. The comparison, legal-edge check, mutation, operation receipt and audit append must commit in one storage transaction. etcd transactions and Kubernetes conditional updates illustrate conditional mutation; no particular backend is required here.

The reference below requires a serializable transaction adapter with a real conditional update. Its local evidence uses SQLite transactions and a revision-predicated UPDATE, not an in-memory imitation of an atomic write.

FORWARD = {
    "ASSEMBLED": "OFFLINE_ACCEPTED", "OFFLINE_ACCEPTED": "SHADOW",
    "SHADOW": "CANARY_1", "CANARY_1": "CANARY_2",
    "CANARY_2": "RAMP", "RAMP": "FULL",
}
LIVE = {"SHADOW", "CANARY_1", "CANARY_2", "RAMP", "FULL", "PAUSE"}


@dataclass(frozen=True)
class StageCommand:
    release_id: str
    operation_id: str
    candidate_id: str
    policy_hash: str
    expected_revision: int
    next_stage: str


def compare_and_set_stage(store, command):
    if (not isinstance(command, StageCommand)
            or type(command.expected_revision) is not int
            or command.expected_revision < 0
            or any(type(x) is not str or not x for x in (
                command.release_id, command.operation_id, command.candidate_id,
                command.policy_hash, command.next_stage))):
        return {"status": "INVALID_COMMAND"}
    with store.transaction() as tx:
        prior = tx.operation(command.release_id, command.operation_id)
        if prior is not None:
            if prior["command"] != command:
                return {"status": "OPERATION_ID_CONFLICT"}
            return dict(prior["result"], replayed=True)
        record = tx.read_release(command.release_id)
        if (record is None or record["candidate_id"] != command.candidate_id
                or record["policy_hash"] != command.policy_hash):
            return {"status": "IDENTITY_CONFLICT"}
        if record["revision"] != command.expected_revision:
            return {"status": "STALE_REVISION"}
        stage, target = record["stage"], command.next_stage
        latched = record["rollback_requested"]
        if target == "REQUEST_ROLLBACK":
            if stage not in LIVE or latched:
                return {"status": "INVALID_TRANSITION"}
            next_stage, next_latch = stage, True
        elif target == "ROLLBACK":
            if stage not in LIVE or not latched:
                return {"status": "INVALID_TRANSITION"}
            next_stage, next_latch = target, True
        elif target == "QUARANTINE" and stage == "ROLLBACK":
            next_stage, next_latch = target, True
        else:
            if latched:
                return {"status": "ROLLBACK_LATCHED"}
            legal = FORWARD.get(stage) == target
            legal = legal or (stage in LIVE - {"PAUSE"} and target == "PAUSE")
            if not legal:
                return {"status": "INVALID_TRANSITION"}
            next_stage, next_latch = target, False
        if not tx.cas_update(command, next_stage, next_latch):
            return {"status": "STALE_REVISION"}
        result = {"status": "APPLIED", "revision": record["revision"] + 1,
                  "stage": next_stage, "rollback_requested": next_latch}
        tx.record_operation_and_audit(command, result)
        return result

The adapter rolls back on exceptions and exposes success only after commit. It persists operation IDs and their complete command payloads, so a restart or duplicate delivery returns the earlier result; changing the payload under the same successful operation ID is a conflict. Rejected attempts do not consume an operation ID. An operation receipt records a historical outcome, not permission to replay an old routing effect.

CAS does not choose rollback precedence. First commit REQUEST_ROLLBACK as a durable latch, retrying stale-revision conflicts after rereading the same immutable candidate. From that commit onward no forward promotion can commit, including a promotion prepared earlier. Then commit ROLLBACK. If a promotion committed before the latch, the latch applies at the successor revision; it does not pretend to undo the earlier history. The latch cannot be cleared for this failed release. A new attempt needs a new release record and its own authorization. This minimal machine deliberately has no automatic PAUSE-to-promotion or QUARANTINE-to-live edge.

The decision helper request_rollback submits that latch and subsequent ROLLBACK transition; it does not call cache cleanup directly. Only after ROLLBACK is durably committed does the reconciler invoke reconcile_rollback with its current fence. This separates requesting authority from executing the idempotent work authorized by it.

Keep desired stage separate from observed routing. A reconciler resumes after partial failure and applies each side effect idempotently under the current durable revision/fence. Routers and cache owners reject stale effects at their mutation boundary, even if an old command once returned APPLIED. Stage authorization also verifies the sealed eligibility/stage evidence, approver and metric window before issuing a command; this small storage reference models concurrency and identity, not authentication or a complete production release service.

0x0F Failure Lab

0x10 What M8 Now Means

M8 is not a list of bit widths. It is a chain of contracts:

M8.1  numerical code, reconstruction, accumulation, packing, kernel contract
M8.2  calibration evidence, sensitivity, and mixed-precision allocation
M8.3  outlier geometry and function-preserving transforms
M8.4  scalar weight-only PTQ: RTN, OBS/OBQ, GPTQ, AWQ
M8.5  sparse, codebook, additive, incoherent, and vector representations
M8.6  request-dependent activation and persistent K/V quantization
M8.7  fake-quant graphs, STEs, learned qparams, distillation, QAT export
M8.8  native binary/ternary training and converted low-bit architectures
M8.9  container, logical tensor, graph, loader, topology, and kernel ABI
M8.10 quality, serving, resources, release, monitoring, and rollback

The central distinction across the module is:

logical bit width
    is not effective artifact size
    is not loaded memory
    is not executed kernel precision
    is not service capacity
    is not quality preservation
    is not energy reduction
    is not release eligibility

Compression succeeds only when the complete deployment tuple preserves required behavior, executes the intended low-bit path, improves the target resource objective under representative load, and can be observed and reversed safely.

0x11 Claim Status And Scope

0x12 Publish And Release Gate

This is a topic-coverage checklist, not a certificate that every historical review finding is closed or that a real deployment has passed these gates. The release-safety reference tests are synthetic contract evidence only. The measurement-contract tests likewise verify scoped arithmetic and adapter behavior, not bootstrap coverage, observed serving capacity, production release readiness, or publication closure.

• The candidate is an immutable artifact/runtime/kernel/topology/config/workload tuple.
• Baseline, release candidate, mechanism ablation, and compound product change are distinguished.
• Perplexity is derived and scoped; task metrics, structured validity, critical buckets, macro/micro aggregation, and failure tags are explicit.
• Large-scale source evidence is converted into a method-by-scale-by-task evaluation matrix, with difficulty strata and no universal model-size ranking.
• Paired differences, standard error, paired bootstrap, clustered units, confidence intervals, and non-inferiority are derived.
• Development diagnostics, frozen promotion data, and production monitoring are separate evidence layers.
• TTFT, queue, TPOT, ITL, E2E, offline throughput, open-loop load, raw throughput, and goodput are distinct.
• Workload length, output, modality, arrival, concurrency, and cache distributions are part of the benchmark.
• Artifact, host startup, loaded HBM, dynamic HBM, workspace, rollout peak, and SLO-constrained capacity are separate ledgers.
• Energy has a device/system boundary, synchronized window, repetitions, variance, token/output accounting, and quality eligibility.
• Hard gates run before Pareto or scalar utility ranking.
• A source-reported W4A16/W8A8 latency-throughput crossover is scoped to its exact model, task, GPU, and runtime version.
• Within-format algorithm tuning is separated from between-format deployment selection.
• The corrected k-bit scaling-law source is scoped to fixed parameter bits, studied weight-only methods, 16-bit inputs, model families, and zero-shot tasks.
• Evidence bundles bind raw outputs and traces to policy, candidate, control, and rollback hashes.
• Shadow, sticky canary, staged ramp, absolute/relative gates, and rare-event limitations are explicit.
• Production metrics retain candidate/control identity, bounded labels, workload distributions, quality delay, burn rate, and kernel/fallback telemetry.
• Rollback handles new traffic, in-flight work, capacity, cache/session state, configuration, persistent side effects, recovery timing, and quarantine.
• Interactive, batch, edge, long-context, multimodal, and multi-adapter deployment modes are compared without declaring a universal winner.
• M8 presents a ten-article path from numerical representation to reversible production release.

0xFF References

Local Series And Transformer System Sources

• M4.1 - Transformer Cost Model - FLOP, byte, arithmetic-intensity, prefill, and decode cost boundaries. [TRANSFORMER SYSTEM]
• M5.1 - Autoregressive Serving State - K/V lifecycle, scheduler pressure, TTFT/TPOT/E2E distinctions, and request state. [TRANSFORMER SYSTEM]
• M5.3 - K/V Layout And Exact Reuse - normative compatibility manifest, namespace generations, legal references and physical reclamation. [TRANSFORMER SYSTEM]
• M5.6 - Prefill/Decode Disaggregation And K/V Placement - tail-SLO goodput and serving placement constraints. [TRANSFORMER SYSTEM]
• M8.1 - Quantized Numbers And The Kernel Contract - quantized representation, reconstruction, metadata, accumulation, packing, and operator contract. [TRANSFORMER SYSTEM]
• M8.6 - Live-Data Quantization - activation and K/V state, cache ABI, append/fork/reorder/truncate/reuse, and runtime consumers. [TRANSFORMER SYSTEM]
• M8.9 - Quantized Artifacts And Runtime Integration - immutable manifests, artifact schemas, topology, loader/parity, dispatch, fallback, and resource handoff. [TRANSFORMER SYSTEM]
• Compression Series (2) - Reproducible Baseline And Measurement Harness - B0 denominator, controls, token/latency/memory/energy windows, repetitions, and registry fields. [SOURCE-LOCAL]
• Compression Series (3) - Proxy Evaluation Design - bucket coverage, critical floors, proxy gaps, and failure taxonomy. [SOURCE-LOCAL]
• Compression Series (13) - Experiment Registry And Ablation Logbook - candidate lineage, raw/aggregate records, eligibility, and decision state. [SOURCE-LOCAL]
• Compression Series (14) - Submission Packaging - immutable candidate/runtime/config package, dry-run evidence, and fallback. [SOURCE-LOCAL]
• Transformer Series (34) - Quantitative Fundamentals - memory, precision, quantization, latency, throughput, power, and hardware motivation. [SOURCE-LOCAL]
• Transformer Series (35) - Fundamentals Of Large Model Quantization - prefill/decode bottlenecks, weight/activation/KV choices, and deployment tradeoffs. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - method families, implementation variants, and deployment context. [SOURCE-LOCAL]

Evaluation And Benchmarking

• “Give Me BF16 or Give Me Death?”: Accuracy-Performance Trade-Offs in LLM Quantization - accuracy, latency, throughput, cost, hardware, workload, and quantization-format comparisons; source Figure 2 is CC BY 4.0. [PAPER, SOURCE-REPORTED RESULTS]
• Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant - quantization evaluation across model sizes up to 405B, 13 benchmarks, six task categories, several quantization methods, and task/method/scale interactions. [PAPER, SOURCE-REPORTED RESULTS]
• The case for 4-bit precision: k-bit Inference Scaling Laws - more than 35,000 fixed-model-bit zero-shot experiments across 19M—176B models, 3—8-bit weights, datatype/block-size choices, and the source-scoped four-bit optimum. [PEER-REVIEWED PAPER, CORRECTED SOURCE LINK]
• lm-evaluation-harness Task Guide - versioned task configuration, dataset, prompt, few-shot, generation, and metric definitions. [OFFICIAL PROJECT DOCUMENTATION, VERSION-SPECIFIC]
• lm-evaluation-harness Evaluator - exact sample selection, bootstrap iterations, sample logging, model/runtime configuration, seeds, git hash, environment, and tokenizer metadata. [OFFICIAL CODE, VERSION-SPECIFIC]
• MLPerf Inference Datacenter - scenario-specific performance and full-system power measurement boundary. [OFFICIAL BENCHMARK DOCUMENTATION, VERSION-SPECIFIC]
• vLLM Serve Benchmark - request rate, burstiness, arrival distributions, trace timing, and ramp-up controls. [OFFICIAL CODE, VERSION-SPECIFIC]

Runtime Metrics And Energy

• vLLM Per-Request Metrics - current per-request queue, TTFT, generation, inter-token, and throughput fields. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• vLLM Metrics Design - current production histograms, counters, gauges, token distributions, queue/prefill/decode timing, K/V use, and metric history. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• Prometheus Histograms And Summaries - bucket-based quantiles, aggregation, native/classic histogram considerations, and the inability to aggregate precomputed summary quantiles. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]
• Zeus Measuring Energy - begin/end windows, synchronized asynchronous GPU execution, device scope, distributed measurement, and power/energy telemetry. [OFFICIAL PROJECT DOCUMENTATION, VERSION-SPECIFIC]

Release And Rollback

• etcd v3.6 Transaction API - atomic predicate evaluation and conditional mutation; an example storage contract, not a required backend. [OFFICIAL DOCUMENTATION, VERSION-SCOPED, CHECKED 2026-09-25]
• Kubernetes API Conditional Updates - resource-version preconditions and stale-update conflicts. [OFFICIAL DOCUMENTATION, CHECKED 2026-09-25]
• Google SRE Workbook: Canarying Releases - reproducible/automated releases, partial time-limited canaries, control comparison, gradual stages, representative and attributable metrics, error-budget risk, and rollback. [PRIMARY ENGINEERING SOURCE]
• Kubernetes Deployments - controlled rolling updates, rollout status, pause/resume, revision history, progress deadlines, canary pattern, and rollback to prior Pod-template revisions. [OFFICIAL DOCUMENTATION, VERSION-SPECIFIC]