M7.8 - Speculative Serving: Scheduling, Evaluation, Release, and Rollback

M7.8 - Speculative Serving: Scheduling, Evaluation, Release, and Rollback

0x00 Overview And Background

The previous M7 articles established algorithms that can reduce the number of serial target-model rounds. They did not establish that enabling speculation on a shared server improves latency or capacity.

At production scale, speculation competes with batching:

Inter-request parallelism:
    put more requests in the target batch

Intra-request parallelism:
    put more candidate positions behind each request

The first source of work always advances another live request. The second source is useful only when candidates are accepted. Once the GPU is already well utilized by concurrent requests, speculative positions can displace useful request work and increase time per output token.

The owner question is:

How should a continuous-batching runtime choose speculative work, maintain exact per-request state, prove a serving win under changing load, and disable or roll back the feature without corrupting output or cache state?

Historical path to production control

The speculative-decoding lineage in A30 starts from an algorithmic asymmetry: an autoregressive target is expensive when asked for one new token, yet can score several already supplied candidate positions in parallel. M7.1 made the resulting probability contract explicit. M7.2-M7.7 then changed where candidates come from and how they are arranged. None of those facts determines how many speculative rows a shared server should schedule.

Production serving introduces a second lineage. Iteration-level batching and PagedAttention made request admission and KV ownership dynamic. The Synergy study then showed that batching and speculation are not additive knobs: the best draft length changes with batch size, model, and GPU. TurboSpec moved that observation into a closed loop combining an offline execution profile with online acceptance feedback. Current vLLM and TensorRT-LLM releases expose batch-size-dependent draft schedules, including an explicit zero-depth region. These are runtime mechanisms, not a proof that a particular schedule transfers between deployments.

Two later systems results sharpen the production contract. Batch Speculative Decoding Done Right identifies ragged acceptance as a synchronization problem: token positions, attention masks, and KV rows must all be realigned to the same logical prefixes. SPECTRE studies a different axis, placing the drafter on a remote shared service and overlapping drafting with verification. Together they show why “turn on speculative decoding” is not one operation. It is a versioned scheduler, tensor-layout, state-transaction, and placement decision.

The article therefore follows four questions in dependency order:

semantic:
    which final tokens and cache rows does one round own?

physical:
    which query rows, temporary buffers, and devices execute that round?

control:
    which measured feasible action maximizes useful service under load?

operational:
    which evidence permits release, disable, or rollback?

Background knowledge: the serving unit is not “one request, one token” anymore

Ordinary decode contributes one query token per active request. A speculative round contributes a variable number of candidate query rows per request.

For active request set $\mathcal{R}$ and request-specific speculative depth $K_i$:

$$N_{query} = \sum_{i\in\mathcal{R}}(K_i+\chi_i)$$

$\chi_i$ represents method-specific anchor or frontier rows included in the target call. Its exact value depends on how logits and KV are aligned. The implementation must profile its actual query count rather than assuming every method maps to exactly $B\times K$ rows.

The useful output from that work is ragged: one request may accept four candidates, another none, and a third may not speculate at all.

0x01 Claim Status And Scope

This article owns deployment integration and release operations. It does not re-derive the proposal algorithms in M7.7 or the exact one-sequence sampler in M7.1.

M5.2 owns the ordinary versioned iteration plan, queue/admission policy, fairness service, and SLO denominators. M7.8 extends that decision surface with optional candidate rows, temporary branch state, and K=0; it does not replace M5.2’s base scheduler contract.

0x01A Ownership firewall

The runtime boundary is especially important. M5.2 decides which ordinary requests can receive service. M7.8 can spend optional rows inside that decision, but cannot let speculative work bypass age, priority, tenant, or SLO accounting. A request that receives one useful token with K=0 has received service; a request that consumes five candidate rows and accepts none has consumed resources without equivalent useful progress.

0x01B Immutable Serving Manifest And Round Plan

A serving profile is valid only for the exact objects that produced it. Model name strings are insufficient: two files with the same architecture can differ in vocabulary mapping, proposal layers, quantization, kernels, graph captures, or scheduler semantics.

@dataclass(frozen=True)
class SpecServingManifest:
    target_checkpoint_hash: str
    target_config_hash: str
    tokenizer_hash: str
    proposal_method: str
    proposal_artifact_hash: str | None
    verifier_algorithm: str
    sampler_contract_hash: str

    engine_version: str
    kernel_build_hash: str
    hardware_topology: tuple[str, ...]
    parallel_topology: tuple[int, int, int]  # TP, PP, EP
    kv_dtype: str
    graph_bucket_hash: str

    profile_generation: str
    profile_hash: str
    controller_version: str
    acceptance_schema: str
    release_epoch: int

The mutable request state points at that immutable manifest:

@dataclass
class RequestSpecState:
    request_id: str
    manifest_hash: str
    committed_token_count: int
    target_kv_length: int
    draft_kv_length: int | None
    pending_frontier_token: int | None
    proposal_ready: bool
    finished: bool
    rng_counter: int
    state_generation: int
    last_committed_round: int


@dataclass(frozen=True)
class SpecRoundPlan:
    iteration_id: int
    request_ids: tuple[str, ...]
    K_by_request: tuple[int, ...]
    query_rows_by_request: tuple[int, ...]
    target_blocks_reserved: tuple[tuple[int, ...], ...]
    draft_blocks_reserved: tuple[tuple[int, ...], ...]
    graph_bucket: str
    profile_generation: str
    controller_observation_cutoff: int

The plan is immutable after workers begin execution. Completions, cancellations, new arrivals, or controller updates belong to the next scheduler iteration. This is the same iteration-plan rule used by M5.2, extended with speculative shape and temporary-state ownership.

Before dispatch, validate:

def validate_round(manifest, requests, plan, profile):
    assert hash_manifest(manifest) == requests.common_manifest_hash()
    assert plan.profile_generation == manifest.profile_generation
    assert profile.supports_exact_shape(plan.graph_bucket, plan.K_by_request)
    assert all(k >= 0 for k in plan.K_by_request)
    assert target_blocks_fit(plan.target_blocks_reserved)

    for req, k in zip(requests, plan.K_by_request):
        assert not req.finished
        if k > 0 and method_needs_draft_kv(manifest.proposal_method):
            assert req.proposal_ready
            assert req.draft_kv_length == req.target_kv_length == req.committed_token_count
            assert req.pending_frontier_token is None

The draft/target/output equality is the strict dispatch convention used here. Other runtimes may fuse repair into the next forward, but must declare each model’s pending suffix. A one-token target frontier does not imply a one-token draft deficit: full acceptance can leave two draft rows missing. Cache presence alone is not proposal readiness.

0x02 Derive Useful Tokens Per Speculative Round

Assume a constant conditional acceptance probability $\alpha$ for each proposed token and a maximum draft length $K$. This independence assumption is a teaching and control-model approximation; real acceptance is depth-, context-, and method-dependent.

The following full-round output moments assume no EOS or length limit suppresses an accepted token or the final correction/bonus. Section 0x09 handles stopped rounds separately; their output growth is not unconditionally $A+1$.

Let $A$ be the accepted draft-prefix length. Token $j$ is accepted only if the first $j$ proposals are all accepted:

$$\Pr(A\ge j)=\alpha^j$$

Use the tail-sum identity for a non-negative integer random variable:

$$\mathbb{E}[A] = \sum_{j=1}^{K}\Pr(A\ge j) = \sum_{j=1}^{K}\alpha^j = \frac{\alpha(1-\alpha^K)}{1-\alpha}$$

The target verifier also emits one target-governed frontier token: either a correction at the first rejection or a bonus token after all drafts are accepted. Therefore:

$$\mathbb{E}[N_{commit}] = 1+\mathbb{E}[A] = \frac{1-\alpha^{K+1}}{1-\alpha}$$

with limit $K+1$ as $\alpha\rightarrow1$.

Renderer-safe form:

E[accepted_drafts] = alpha + alpha^2 + ... + alpha^K

E[committed_output_tokens]
    = 1 + E[accepted_drafts]
    = (1 - alpha^(K+1)) / (1 - alpha)

0x02A Worked Acceptance Ledger

For $\alpha=0.7$:

Each additional candidate has declining expected utility. Its compute and temporary-state cost does not decline in the same way.

0x02B Derive The Full Accepted-Length Distribution

The mean hides the raggedness that the scheduler must execute. Let $\alpha_j$ be the conditional probability that candidate $j$ is accepted given that candidates $1,\ldots,j-1$ were accepted. Define the survival probability:

$$S_j = \Pr(A\ge j) = \prod_{m=1}^{j}\alpha_m, \qquad S_0=1.$$

For $0\le a<K$, exactly $a$ drafts are accepted when the first $a$ survive and candidate $a+1$ fails:

$$\Pr(A=a) = S_a(1-\alpha_{a+1}).$$

All $K$ drafts are accepted with probability:

$$\Pr(A=K)=S_K.$$

The masses telescope:

$$\sum_{a=0}^{K-1}S_a(1-\alpha_{a+1})+S_K = \sum_{a=0}^{K-1}(S_a-S_{a+1})+S_K =1.$$

One target-governed frontier token is emitted in either case, so the committed output count is $C=A+1$. The first two moments are:

$$\mathbb{E}[C] = 1+\sum_{j=1}^{K}S_j,$$ $$\operatorname{Var}(C) = \sum_{a=0}^{K}(a+1)^2\Pr(A=a) -\mathbb{E}[C]^2.$$

The constant-$\alpha$ expression earlier is the special case $S_j=\alpha^j$. A controller needs the general form because depth-4 acceptance is conditioned on a request surviving depths 1-3; it is not estimated from all requests in the batch.

def accepted_prefix_pmf(alpha_by_depth, K):
    pmf = []
    survival = 1.0
    for j in range(K):
        pmf.append(survival * (1.0 - alpha_by_depth[j]))
        survival *= alpha_by_depth[j]
    pmf.append(survival)  # all K accepted
    assert abs(sum(pmf) - 1.0) < 1e-12
    return pmf

Why retain the variance of committed output? Two actions can have the same mean goodput but different round-time and progress variance. The higher-variance action can amplify tail inter-token latency, graph-bucket imbalance, and cross-request synchronization. Mean-only control is therefore a first approximation, not a release criterion.

0x02C Goodput And Three Different Token Rates

For one serving step:

$$\operatorname{Goodput} = \frac{\text{final committed output tokens}} {\text{wall-clock execution time}}$$

For batch $\mathcal{R}$ with per-request acceptance estimate $\alpha_i$ and depth $K_i$:

$$\widehat G = \frac{ \sum_{i\in\mathcal{R}} \frac{1-\alpha_i^{K_i+1}}{1-\alpha_i} } {\widehat T_{draft}(\mathcal{R},K) +\widehat T_{verify}(\mathcal{R},K) +\widehat T_{select} +\widehat T_{state}}$$

Unlike raw processed-token throughput, goodput excludes rejected candidate rows.

Keep three counters separate:

processed query rows:
    target and proposal positions executed by kernels

committed output tokens:
    accepted drafts plus target-governed frontier tokens

SLO-qualified output tokens:
    committed tokens delivered while their request remains within policy

TurboSpec’s goodput uses the second numerator. A production dashboard may also need the third; otherwise a controller can improve aggregate token rate while starving an old request or worsening p99 inter-token latency. This article uses goodput for committed output per wall-clock time and names any SLO-filtered metric explicitly.

0x02D Estimation Bias And Censoring

Depth statistics are censored by earlier rejection. The estimator for $\alpha_j$ is:

$$\widehat\alpha_j = \frac{n_{accepted,j}} {n_{reached,j}},$$

not $n_{accepted,j}/n_{rounds}$. A controller that reduces $K$ also stops observing deeper positions, creating policy-dependent missing data. Therefore record the action that generated every observation and use controlled probes or hierarchical fallback before concluding that an unobserved depth is poor.

0x03 Continuous Batching Meets Ragged Acceptance

Continuous batching removes completed requests and admits new requests at scheduler iterations rather than waiting for a fixed request batch to finish. Speculation changes the scheduler’s work unit from one decode row to a request-specific block.

request r1: K=4, accepts 4 + bonus output
request r2: K=4, accepts 1 + correction output
request r3: K=0, ordinary one-token decode
request r4: prompt lookup proposes only 2 rows

The output lengths, target branch-cache commit lengths, draft-cache rollback points, and next input positions are all different.

0x03A Logical Request State

Each request needs at least:

committed_token_count
target_kv_length
pending_frontier_token or none

proposal_enabled
proposal_method
proposal_kv_length or retrieval cursor
proposal_checkpoint/index version

current_K
acceptance statistics by depth
temporary target-branch allocation
sampling RNG state

The invariant is not always:

committed_token_count == target_kv_length

When the verifier emits a correction or bonus token from a target distribution, that token can be appended to the output before the token itself has been processed as an input row. A safe state is:

$$\operatorname{committed\_tokens} = \operatorname{target\_kv\_length} + \mathbf{1}[\text{pending frontier token}]$$

The next valid target call must materialize the frontier token’s KV before later tokens depend on it.

0x03B Per-Request Commit Record

@dataclass
class SpecCommit:
    request_id: str
    accepted_draft_rows: list[int]
    emitted_frontier_token: int | None
    target_kv_rows_to_commit: list[int]
    target_kv_rows_to_discard: list[int]
    draft_rollback_length: int | None
    proposal_can_resume: bool


def apply_commit(req, commit, target_branch_cache):
    assert forms_prefix(commit.accepted_draft_rows)

    req.target_cache.commit_rows(
        target_branch_cache,
        commit.target_kv_rows_to_commit,
    )
    target_branch_cache.discard(commit.target_kv_rows_to_discard)

    if commit.emitted_frontier_token is not None:
        req.output.append(commit.emitted_frontier_token)
        req.pending_frontier_token = commit.emitted_frontier_token

    if req.draft_cache is not None:
        req.draft_cache.truncate(commit.draft_rollback_length)

    req.proposal_enabled &= commit.proposal_can_resume

This is a cache/frontier fragment of the larger atomic publication transaction: the omitted accepted-output step must already have appended the emitted accepted draft tokens before this fragment appends the frontier. It is not a complete standalone output publisher. The stopping rule must first truncate both the accepted-output list and its target-row commit map. proposal_can_resume must reflect actual cache repair and terminal status, not merely cache allocation.

The frontier’s KV materialization, unlike its output publication, can be eager or deferred. Section 0x09 uses deferred materialization and separate pending suffixes for the target and drafter.

0x03C Ragged Acceptance Is A Synchronization Problem

For a nonterminal full round, request $i$ advances by $A_i+1$ output tokens but only $A_i$ candidate-input rows are immediately available for target-KV commit. The batch is therefore ragged in two coordinates: logical output length and materialized target-KV length.

Source: Batch Speculative Decoding Done Right, ragged-tensor overview. The figure motivates the synchronization problem; its implementation and benchmark claims remain source-reported.

For each request, define:

$$L_i^{out}=L_i^{prefix}+A_i+1,$$ $$L_i^{kv}=L_i^{prefix}+A_i.$$

The final +1 is the pending correction or bonus frontier. A rectangular physical batch can pad these lengths, but padding must not become logical history. Three invariants close the mapping.

Position invariant. The visible non-padding tokens for request $i$ have contiguous logical positions $0,\ldots,L_i^{out}-1$. Padding locations never reuse or advance a logical position.

Mask invariant. A query row can see exactly the committed prefix and its allowed candidate ancestors. It cannot see batch padding, a rejected suffix, or another request’s realignment rows.

KV invariant. Every committed target-KV row identifies the same request, token, logical position, ancestor path, layer, dtype, and cache generation as the corresponding reference decode. The frontier token is not represented by the rejected candidate’s KV row.

These invariants explain why a shape-correct implementation can still generate wrong output. Position IDs may point through padding; a stale attention mask may expose rejected rows; or a copied KV block may retain the old physical offset after the logical sequence moves.

0x03D Rectangularization Alternatives

The correctness-first unpad/append/repad construction is:

def synchronize_verified_batch(batch, verification):
    logical = []
    for req, result in zip(batch.requests, verification):
        prefix = read_logical_prefix(req)
        accepted = result.accepted_draft_tokens
        frontier = result.frontier_token

        commit_target_rows(req, result.target_rows[:len(accepted)])
        discard_target_rows(req, result.target_rows[len(accepted):])
        logical.append(prefix + accepted + [frontier])

    width = max(map(len, logical))
    token_ids, position_ids, masks, block_tables = repad(logical, width)
    assert_positions_masks_and_kv_agree(
        token_ids, position_ids, masks, block_tables
    )
    return token_ids, position_ids, masks, block_tables

Source: Batch Speculative Decoding Done Right, unpad-append-repad process. The article uses the figure to expose state that must move together, not to prescribe one universal storage layout.

Realignment is not free. If $b_i$ physical KV blocks must be remapped or copied, a simplified cost ledger is:

$$T_{sync} = T_{compact}(N_{rows}) +T_{mask}(B,W) +T_{position}(B,W) +T_{kvmap}\!\left(\sum_i b_i\right).$$

This term belongs in the measured round cost. Treating it as scheduler overhead outside the speculative objective systematically overestimates goodput.

0x04 Batch Expansion And The $B\times K$ Pressure

Let $B$ be the number of active decode requests. A fixed-depth sequence proposer with $K$ candidate rows creates approximately:

$$N_{verify}\propto B(K+1)$$

target query rows, with method-specific alignment details. Current vLLM dynamic speculative-decoding documentation describes the same operational pressure as effective batch size $BS\times K$: beyond a hardware-dependent critical region, speculative verification hurts time per output token.

The important distinction is:

scheduler batch size:
    active requests

target query-row count:
    all anchor, candidate, and repair rows

useful output count:
    accepted prefixes + target frontier tokens

Two batches with the same request count can have very different target shapes if their $K_i$, tree sizes, prompt-lookup hits, or repair paths differ.

0x04A Why Larger Batches Prefer Smaller $K$

The Synergy paper models expected accepted draft tokens as a sublinear function:

$$l(K)\approx cK^{\gamma}, \qquad 0<\gamma<1$$

and target verification time at fixed batch size $B$ with a local linear approximation:

$$t_T(B,K)\approx a_BK+\beta$$

Including sequential draft cost $K t_D(B,1)$, time to generate $N$ output tokens is approximated by:

$$T(N,B,K) \approx N\, \frac{a_BK+\beta+K t_D(B,1)} {cK^{\gamma}+1}$$

As $B$ grows, the marginal target slope $a_B$ and draft cost generally grow because more parallel rows already occupy compute. The numerator’s penalty for another proposal rises, while the denominator remains sublinear. The minimizing $K$ therefore moves downward.

This derivation is a local model, not a universal latency law. Real kernels have plateaus, graph buckets, memory cliffs, and topology effects. Its value is explaining the direction that profiling repeatedly observes.

Source: The Synergy of Speculative Decoding and Batching in Serving LLMs, Figure 1. Each curve is a batch size and each star is that curve’s best measured speculation length. Across the reported model/GPU combinations, larger batches usually shift the optimum toward shorter drafts. The exact optimum is configuration-specific; the recurring shape is the evidence for profiling $B$ and $K$ jointly.

0x05 Profile The Execution Surface

A production controller needs an offline latency surface, not a scalar “draft is 10x smaller” claim.

For target model/hardware configuration $M$, TurboSpec uses a profile over context and batched query tokens, with a fitted teaching form:

$$T_{fwd}(M,N_{context},N_{query}) \approx aN_{context}+bN_{query}+c$$

The terms summarize KV loading, query-token computation, and fixed model/system cost. Actual deployment should retain measured residuals and use richer buckets when the linear form misses graph or kernel transitions.

profile_key = (
    target_checkpoint,
    draft_checkpoint_or_method,
    dtype_and_quantization,
    cache_schema_and_qspec,
    allocation_and_retention_policy,
    tensor_parallel_degree,
    kernel_build,
    gpu_type,
    graph_capture_mode,
)

for context_bucket in CONTEXT_BUCKETS:
    for request_batch in REQUEST_BATCHES:
        for K in SPEC_DEPTHS:
            measure(
                target_verify_latency,
                draft_latency,
                temporary_kv_bytes,
                peak_memory,
                graph_capture_hit,
            )

0x05A Memory Feasibility Before Goodput

For homogeneous full K/V rows, request $i$ with candidate-node count $N_i$ has the following approximate temporary target code payload, with $b$ in bytes per scalar:

$$M_{temp,i} \approx N_iL\,2H_{kv}d_hb$$

This product is not a generic quantized/MLA/shared-page cache allocator. Admission replaces it with the actual incremental allocation-union delta for the schema’s per-layer retained segments: codes, qparams, residual/sparse state, page rounding and sharing counted once. M8.6 owns quantized representation, M5.8 latent representation, and M5 retention/publication/lifetime.

The following capacity ledger requires disjoint categories: resident excludes separately listed committed K/V and workspace, temporary K/V means new allocations not already in committed/shared state, and logits/sampling/ proposal workspace is charged once. The candidate action set must first satisfy:

$$M_{resident} +M_{committedKV} +\sum_iM_{temp,i}(K_i) +M_{workspace} \le M_{device}-M_{reserve}$$

Only feasible $K$ values should enter the goodput optimizer. Relying on OOM recovery in the critical path is not a control policy.

The paper statement that draft length does not enlarge the persistent KV cache and the ledger above are compatible. Candidate depth does not permanently append $K$ rows per round. It can still increase transient target rows, attention workspace, logits, branch block reservations, and allocator rounding. Measure persistent KV, reserved branch KV, and ephemeral workspace separately.

0x05B A Profile Is A Measured Domain, Not A Formula

The profile key must include every variable that can change kernel selection or state cost:

ShapeKey(
    proposal_method,
    request_batch_bucket,
    context_token_bucket,
    query_rows,
    candidate_nodes,
    K_bucket,
    target_dtype,
    kv_dtype,
    cache_schema_and_qspec,
    allocation_and_retention_policy,
    TP, PP, EP,
    graph_bucket,
    adapter_mix_bucket,
    prefix_cache_state,
)

For each shape, collect more than one mean:

warm-up until graph capture and allocator state are stable
record proposal / transfer / target / select / sync / repair separately
record p50, p95, p99 and sample count
record peak allocated and peak reserved memory
record graph hit, kernel IDs, and target query-row count
retain residuals from any fitted model

Use blocked randomization when profiling several $K$ values so thermal state, clock drift, and background load do not always favor the same action. Store raw samples or sufficient statistics with the profile generation.

TurboSpec’s linear teaching model can be written:

$$\widehat T_{fwd} = aN_{context}+bN_{query}+c.$$

For measured sample $n$, the residual is:

$$e_n=T_n-\widehat T_n.$$

The model is acceptable only inside a declared support region and while its residual distribution remains bounded. Graph-capture transitions, kernel changes, allocator cliffs, and expert-routing changes can produce piecewise behavior that one global plane cannot represent.

def predict_round(profile, shape):
    bucket = profile.exact_or_declared_interpolation(shape)
    if bucket is None:
        raise OutOfProfileSupport(shape)
    estimate = bucket.predict(shape)
    return Estimate(
        mean=estimate,
        upper=estimate + bucket.residual_quantile(0.95),
        profile_generation=profile.generation,
    )

An out-of-support shape must fall back to K=0, a conservative captured bucket, or an explicitly labeled exploration request. Silent extrapolation is not a performance optimization; it is an unreviewed policy change.

Source: An Interpretable Latency Model for Speculative Decoding in LLM Serving, Figure 4. One shared cost pair fails across draft lengths and acceptance rates; stage-aware effective costs recover the measured relationship. The fitted coefficients are configuration-specific.

0x05C Profile Invalidation

Invalidate or quarantine a profile after any manifest change affecting execution:

target or proposer weights / quantization
tokenizer or sampler contract
engine, compiler, kernel, driver, or graph build
GPU type, TP/PP/EP topology, or placement
KV dtype, block size, memory reserve, or adapter path
material workload shift outside calibrated context and shape buckets

Online residual monitoring can detect drift without pretending to identify its cause. If observed round time repeatedly exceeds the stored upper envelope, disable speculation first, preserve evidence, and re-profile offline.

0x06 Load Emerges From Arrival Rate

Batch size in a continuous server is an outcome of arrivals, service time, KV pressure, and scheduler policy. A 2026 serving study models stable pre-saturation request latency as:

$$L=C_1+BC_2$$

where $C_1$ is effective load-independent cost and $C_2$ is marginal cost per effective active request. By Little’s Law:

$$B=\lambda L$$

for arrival rate $\lambda$. Substitute:

$$L=C_1+\lambda LC_2$$ $$L = \frac{C_1}{1-\lambda C_2}$$

This is a pre-saturation steady-state approximation. It is not valid across the instability boundary where queues, preemption, and cache churn change sharply.

Source: An Interpretable Latency Model for Speculative Decoding in LLM Serving, Figure 1. Measurements from several model and prefill/decode configurations collapse after normalization by fitted fixed and marginal service costs. This supports the Little’s-Law derivation inside the measured stable regime; it does not remove the need to detect saturation or refit after a runtime change.

0x06A Speculation Changes Fixed And Load-Dependent Costs Differently

Let $E=\mathbb{E}[N_{commit}]$ and let a request need $g$ output tokens. It needs about $g/E$ speculative cycles. Split fixed costs into prefill, verifier, and drafter components:

$$C_{1,eff} = c_{1,p} + \frac{g}{E} \left(c_{1,v}+Kc_{1,d}\right)$$

and load-dependent costs:

$$C_{2,eff} = c_{2,p} + \frac{g}{E} \left(c_{2,v}+Kc_{2,d}\right)$$

Then:

$$L_{SD} = \frac{C_{1,eff}} {1-\lambda C_{2,eff}}$$

Large $K$ can improve the low-load fixed-cost term by reducing verifier rounds, yet worsen the load-dependent term through more draft work and expanded target batches. This formally explains why batch-size-1 tuning often over-speculates at high QPS.

0x06B MoE Targets Add Expert-Coverage State

For an MoE verifier, expanding $B\times K$ candidate rows can activate more distinct experts. At low load and small trees, only a sparse subset may be touched; wider verification increases expert coverage, communication, and device imbalance.

The 2026 latency paper models this as a correction to dense scaling. The operational lesson is sufficient here:

profile MoE target verification with real routing;
do not derive K from active parameter count alone;
record expert coverage, all-to-all bytes, and straggler rank time.

This reconnects M7 serving to M6.2 for routing and logical traffic, and to M6.6 for physical placement, collectives, kernels, and measured runtime.

Under the latency paper’s simplifying independent-uniform routing assumption, one expert is never selected by one routed token with probability $1-m/M$, where $m$ experts are selected from $M$. Across $T$ routed tokens:

$$\Pr(\text{expert untouched}) = \left(1-\frac{m}{M}\right)^T.$$

Therefore the expected fraction of distinct experts touched is:

$$\phi(T) = 1-\left(1-\frac{m}{M}\right)^T.$$

This is a modeling baseline, not a claim that trained routers are uniform or independent. Production traces should replace $\phi$ with measured expert sets, tokens per expert, all-to-all bytes, and slowest-rank time.

0x06C Stability Margin And Saturation Gate

The closed form has a stability condition:

$$\lambda C_2<1.$$

Define the stability margin:

$$m_{stable}=1-\lambda C_2.$$

Then sensitivity to arrival rate is:

$$\frac{\partial L}{\partial\lambda} = \frac{C_1C_2}{(1-\lambda C_2)^2} = \frac{C_1C_2}{m_{stable}^2}.$$

As the margin approaches zero, tiny load or model errors produce large latency errors. The controller must not optimize a smooth model through that boundary.

def stable_latency(arrival_rate, C1, C2, min_margin):
    margin = 1.0 - arrival_rate * C2
    if margin < min_margin:
        raise SaturationRisk(margin)
    return C1 / margin

Operational saturation signals include sustained queue growth, rising preemption, cache churn, admission delay, profile residual drift, and a widening p95/p99 gap. On entry, force conservative shapes or K=0; do not wait for the mean-latency model to predict an impossible denominator.

0x06D Arrival Processes And Burst Limits

Little’s Law relates long-run averages under a stable system; it does not say that arrivals are Poisson, that service times are independent, or that a burst has already reached steady state. Evaluate at least:

closed-loop concurrency:
    next request begins after one completes

open-loop stationary arrivals:
    fixed rate with controlled inter-arrival distribution

bursty arrivals:
    on/off or trace replay with the same mean rate

correlated long outputs:
    service demand and active batch remain coupled for many iterations

The same average QPS can yield different queue and batch distributions. A release profile must name its arrival process rather than reporting “QPS=8” as a complete workload specification.

0x07 A Goodput Controller

TurboSpec combines an offline execution profile with online acceptance feedback. At each generation step it evaluates candidate speculative depths and selects the highest estimated goodput, including $K=0$.

Source: TurboSpec, Figure 7. The offline profiler estimates execution cost; the online adapter combines that profile with observed acceptance; the scheduler chooses a supported shape; and propose, score, and accept remain separate runtime stages. Draft and target KV state are shown as distinct resources.

def choose_spec_depth(batch, profile, acceptance_model, memory):
    best = Decision(K=0, predicted_goodput=baseline_goodput(batch, profile))

    for K in supported_depths(batch.method):
        if not memory.fits(batch, K):
            continue
        if not profile.has_shape(batch, K):
            continue

        expected_tokens = sum(
            acceptance_model.expected_commits(req, K)
            for req in batch.requests
        )
        predicted_time = profile.predict_round(batch, K)
        goodput = expected_tokens / predicted_time

        if goodput > best.predicted_goodput + HYSTERESIS_MARGIN:
            best = Decision(K=K, predicted_goodput=goodput)

    return best

0x07A Constrained Action Space

The controller does not optimize over integers in isolation. One action is a physical execution choice:

$$a_i=(K_i,N_i,g_i,p_i),$$

where $K_i$ is maximum path depth, $N_i$ is total candidate rows, $g_i$ is a captured shape bucket, and $p_i$ is a placement or coordination mode. A linear draft has $N_i=K_i$; a tree does not.

Define the feasible set $\mathcal{F}(s)$ for current server state $s$ by:

manifest compatibility
profile support for the exact physical shape
target and draft cache readiness
target token-row budget
temporary KV and workspace budget
captured graph / backend support
parallel topology and sampling-feature support
base scheduler fairness and SLO constraints

Then choose:

$$a^* = \arg\max_{a\in\mathcal{F}(s)} \frac{\operatorname{LCB}[U(a\mid s)]} {\operatorname{UCB}[T(a\mid s)]}.$$

LCB is a lower-confidence estimate of committed useful output and UCB is an upper-confidence estimate of round time. This conservative ratio prevents an uncertain action from winning merely because its point estimate is optimistic. K=0 must always remain in $\mathcal{F}(s)$ and carries directly measured baseline cost.

def choose_action(state, actions, profile, acceptance):
    feasible = [a for a in actions if all_constraints_hold(a, state)]
    assert any(a.K == 0 for a in feasible)

    scored = []
    for action in feasible:
        useful_lcb = acceptance.committed_tokens_lcb(action)
        time_ucb = profile.round_time_ucb(action)
        scored.append((useful_lcb / time_ucb, action))

    score, action = max(scored, key=lambda pair: (pair[0], -pair[1].K))
    return action, DecisionEvidence(score=score, profile=profile.generation)

The controller stores the complete candidate set, rejected constraints, point estimates, uncertainty bounds, and selected action. Without that decision record, an operator cannot distinguish a model error from an action that was never feasible.

0x07B Acceptance Feedback Should Be Segmented

A single global acceptance moving average is simple but can mix incompatible traffic:

code completion vs conversation
temperature 0 vs temperature 1
short vs long context
adapter A vs adapter B
target checkpoint before vs after upgrade
prompt lookup hit vs miss

A production estimator should key statistics by the smallest set of dimensions that materially changes acceptance while retaining enough samples:

(proposal version, target version, sampler bucket,
 workload class, context bucket, adapter id optional)

Use hierarchical fallback when a bucket is sparse. Record acceptance by depth, not only one average:

$$\hat\alpha_j = \frac{\text{accepted proposals at depth }j} {\text{requests that reached depth }j}$$

Then:

$$\mathbb{E}[A] = \sum_{j=1}^{K} \prod_{m=1}^{j}\hat\alpha_m$$

This removes the constant-$\alpha$ approximation from the controller.

For uncertainty, a Beta-Binomial teaching model is useful. With prior $\alpha_j\sim\operatorname{Beta}(a_j,b_j)$ and observed accepted/rejected counts among requests that reached depth $j$:

$$\alpha_j\mid\mathcal{D} \sim \operatorname{Beta} \left(a_j+n_{accepted,j}, b_j+n_{rejected,j}\right).$$

Use a posterior lower quantile for conservative useful-output prediction, not only the posterior mean. The model assumes exchangeability inside its bucket; distribution drift, correlated requests, and controller-selected censoring can violate that assumption. Those are reasons to segment, decay, and probe, not to interpret the posterior as a universal law.

0x07C Step-Level Versus Per-Request Control

The Synergy strategy and current dynamic-runtime schedules are naturally batch- or step-level. Per-request control is not automatically better: its extra raggedness and graph misses can cost more than the acceptance gain. The controller must optimize the implemented physical action, including grouping and padding, rather than an ideal independent $K_i$ vector.

Fairness remains request-based. Charge speculative rows to the request that caused them, but advance age/service accounting by committed useful progress. When reducing an over-budget plan, remove the row with the lowest conservative marginal utility while respecting minimum service for old or priority requests.

0x07D Hysteresis And Exploration

Without hysteresis, noisy estimates can alternate between $K=0$ and $K=1$ every step. Without occasional exploration, a controller that disables speculation after distribution drift may never discover recovery.

enable threshold:
    predicted gain > +epsilon_enable

disable threshold:
    predicted gain < -epsilon_disable

minimum dwell:
    hold decision for N rounds unless memory/SLO emergency

probe:
    route a small eligible fraction through candidate K

Exploration must respect request cache readiness. A model-based proposal cannot be re-enabled mid-request if its draft KV was allowed to fall behind, unless the runtime backfills it.

Every observation must carry:

manifest hash
controller and profile generation
chosen action and shape bucket
eligibility and fallback reason
depth reached / accepted / rejected
measured stage times and synchronization cost
request class and sampling bucket

Late results from an old policy epoch may be retained for forensic analysis but must not silently update the active controller.

0x08 Prefill And Draft-Cache Readiness

A separate draft model needs its own prefix KV cache. There are three policies.

Always prefill the draft

advantage:
    speculation may start at any decode round

cost:
    draft prefill runs even for requests that never speculate

Prefill only eligible requests

advantage:
    avoids work when high load predicts K=0

cost:
    request cannot later speculate without backfill/re-prefill

Lazy prefill on transition

advantage:
    pays only when the controller enables speculation

cost:
    transition latency spike; needs admission/SLO handling

TurboSpec describes a policy that can skip proposer prefill above a disable threshold. Because the draft KV is then absent, speculation remains disabled for those requests unless the system later resets/rebuilds the state.

def can_enable_model_draft(req):
    return (
        not req.finished
        and req.proposal_enabled
        and req.draft_cache is not None
        and req.draft_cache.token_count == req.target_cache.token_count == len(req.output)
        and req.pending_frontier_token is None
        and req.draft_version == ACTIVE_DRAFT_VERSION
    )

The equality may be replaced by a proved frontier convention, but it cannot be assumed away.

0x08A Explicit Cache-State Machine

Use named states instead of inferring readiness from nullable pointers:

Allowed transitions are explicit:

ALIGNED --spec round--> TARGET_FRONTIER
TARGET_FRONTIER --target append--> ALIGNED or DRAFT_BEHIND
DRAFT_BEHIND --draft repair--> ALIGNED
ALIGNED --K becomes 0, maintain draft--> SPEC_DISABLED_SYNCED
ALIGNED --K becomes 0, skip draft--> SPEC_DISABLED_UNSYNCED
SPEC_DISABLED_UNSYNCED --re-prefill/backfill--> ALIGNED
any committed state --preempt--> PREEMPTED
PREEMPTED --validated restore--> prior logical state, new generation
any live state --EOS/length stop--> FINISHED (never proposal-ready)

No transition commits a correction or bonus token’s target KV from the rejected candidate row at the same physical position.

0x09 Draft, Target, And Frontier Cache Transitions

Source: TurboSpec §5.1 and Figure 8. A sequential drafter has not yet processed its final sampled proposal. Under partial acceptance (panel a), that proposal is rejected or discarded, so the draft model resumes from the correction token and materializes that token’s missing cache row. Full acceptance (panel b) instead requires draft-side processing of the final proposal as well as the bonus token. The dashed boxes are newly generated KV state, which is why output-token count and cache-row count cannot be conflated. The ledgers below derive the corresponding prefix lengths locally.

Cache convention and three separate prefixes

Start from a prefix of length $P$ already materialized in both models, with next-token logits available. A standard sequential drafter samples $K\ge1$ proposals without an extra forward on its final sampled token. It has therefore materialized only $K-1$ proposal rows. Target verification processes all $K$ candidate inputs, but only rows belonging to emitted accepted tokens may be committed.

For $A$ accepted proposals and an emitted correction/bonus ($F=1$), before repair:

$$L_{out}=P+A+1,\qquad L_{target}=P+A,\qquad L_{draft}=P+\min(A,K-1).$$

Here $A$ counts accepted draft tokens actually emitted; the no-stop examples below emit the whole accepted prefix. A cache row is model-specific: target verification does not create a draft-model row. An eager extra draft forward can make the final proposal row available, but its execution and cost must be recorded explicitly. Do not assume that variant silently.

Partial acceptance

Suppose $K=4$, two drafts are accepted, the third is rejected, and correction token $c$ is emitted without stopping.

output append:       draft_1, draft_2, correction c
target KV retained:  prefix + draft_1 + draft_2
draft KV retained:   prefix + draft_1 + draft_2
target pending:      [c]
draft pending:       [c]
proposal ready:      false until both pending suffixes are processed

Discard rejected target rows and truncate the draft suffix before repair. Even if $c$ happens to equal a rejected token ID, do not treat token equality as proof that a discarded branch row has the required identity. Reject-first ($A=0$) uses the same rule with both retained caches ending at $P$.

All accepted

Suppose all four drafts are accepted and the target emits bonus token $b$, again without an EOS or length stop.

output append:       draft_1, draft_2, draft_3, draft_4, bonus b
target KV retained:  prefix + draft_1 + draft_2 + draft_3 + draft_4
draft KV retained:   prefix + draft_1 + draft_2 + draft_3
target pending:      [b]
draft pending:       [draft_4, b]
proposal ready:      false

The fully accepted drafter is two rows behind output, not one. Process draft_4 and then b through the draft model in causal order (or an equivalent causally masked two-token forward). Materializing only the target’s bonus row does not repair the draft cache.

For the synthetic $P=10$, $K=4$, $A=4$ example:

Repair appends cache rows, never duplicate output tokens. The target and draft repairs can be scheduled separately, but the strict readiness gate opens only when both caches represent the entire visible prefix.

EOS and length-limit stopping

Apply the stopping rule before publication. Emit no tokens after EOS, and no more than the remaining output budget. If stopping occurs within accepted drafts, omit the correction/bonus and discard all later candidate rows.

Let $E$ be the number of accepted drafts actually emitted after stopping truncation and let $F\in\{0,1\}$ indicate an emitted frontier. Under the same sequential-drafter convention:

$$L_{out}=P+E+F,\qquad L_{target}=P+E,\qquad L_{draft}=P+\min(E,K-1).$$

For example, a length cap reached exactly at draft_4 yields $(L_{out},L_{target},L_{draft})=(14,14,13)$ for $P=10$, without a bonus. EOS or a length cap at the bonus instead leaves $(15,14,13)$.

A terminal request is never proposal-ready, even if all lengths happen to match. Retained caches may still lag the final output; keep that state explicit until release. No continuation repair is required merely to finish the request. A separate cache-reuse/export policy may choose additional work, but must not silently reopen decoding. A zero remaining budget dispatches no round.

No speculation

K = 0:
    ordinary target token, no proposal branch
    its target row may still be pending after output publication
    if a compatible draft cache exists, maintain or backfill it explicitly
    if draft KV is absent, target repair does not recreate it

The toy cache oracle is scoped to a separate sequential autoregressive drafter. Retrieval proposers, integrated heads, tree layouts, and runtimes with an incoming anchor offset need their own explicit state mapping. This prefix ledger is not a claim that all proposal methods share the same cache layout.

0x09A Prepare, Execute, Publish

Cache mutation is a transaction:

def run_spec_round(req, action, workers):
    snapshot = req.snapshot_identity_and_lengths()
    reservation = reserve_branch_blocks(req, action)
    try:
        proposal = workers.propose(req, action)
        verification = workers.target_verify(req, proposal, reservation)
        decision = exact_accept_or_method_verify(req, proposal, verification)

        commit = build_commit_record(snapshot, proposal, verification, decision)
        validate_commit(commit, snapshot, req.manifest_hash)
        publish_atomically(req, reservation, commit)
        return commit
    except BaseException:
        reservation.discard_all()
        assert req.matches(snapshot)  # no partial publication
        raise

publish_atomically means that output tokens, target block table, draft rollback point, pending frontier, RNG counter, and state generation become visible together. The GPU operations may complete at different times; the request-visible metadata publication may not.

0x09B Preemption, Resume, And Duplicate Completion

A preemption snapshot contains logical state, not raw pointers:

request and manifest hash
committed output token IDs or durable reference
target/draft logical KV lengths and block generations
pending frontier token
sampler RNG counter
last committed iteration and state generation
controller eligibility state

On restore, physical blocks may change. Rebuild block tables, then validate the same token/position/layer generation contract before dispatch. A worker result is publishable only when its (request_id, iteration_id, state_generation) matches the active request. Duplicate or late completion is discarded idempotently.

def may_publish(result, req):
    return (
        result.request_id == req.request_id
        and result.iteration_id > req.last_committed_round
        and result.base_generation == req.state_generation
        and result.manifest_hash == req.manifest_hash
    )

0x0A Request Scheduling Policy

A scheduler should treat speculative work as optional query-token demand.

def build_iteration(waiting, running, token_budget, temp_kv_budget):
    batch = admit_prefill_and_decode(waiting, running, token_budget)

    spec_candidates = []
    for req in batch.decode_requests:
        if not request_is_spec_eligible(req):
            req.K = 0
            continue

        req.K = controller.choose_K(req, batch)
        spec_candidates.append(req)

    while estimated_query_rows(batch) > token_budget:
        req = lowest_marginal_goodput_request(spec_candidates)
        req.K = reduce_K(req.K)

    while estimated_temp_kv(batch) > temp_kv_budget:
        req = lowest_marginal_goodput_request(spec_candidates)
        req.K = reduce_K(req.K)

    return bucket_for_executable_shapes(batch)

0x0AA Marginal Utility

For request $i$, the expected output gain from increasing $K$ to $K+1$ under depth-specific acceptance is:

$$\Delta U_i(K+1) = \prod_{j=1}^{K+1}\hat\alpha_{i,j}$$

The scheduler should compare that gain with marginal target, draft, memory, and shape-fragmentation cost. Requests with poor next-depth acceptance should lose speculative rows before high-confidence requests.

0x0AB Shape Buckets And CUDA Graphs

Dynamic $K_i$ creates many execution shapes. Padding every request to the maximum wastes work; fully ragged execution can defeat graph capture and kernel regularity.

Design choices:

Current vLLM documentation exposes batch-size ranges mapped to a configured optimal $K$, including $K=0$. It also documents method and CUDA-graph limitations. These are version-specific implementation facts, not speculative-decoding laws.

0x0B Placement And Distributed Execution

Colocated draft and target

advantages:
    low proposal transfer latency
    simpler request ownership

costs:
    draft weights/KV reduce target KV capacity
    draft kernels contend with target work

Dedicated draft device

advantages:
    draft can overlap or avoid target-memory pressure

costs:
    tokens, probabilities, state, and control cross devices
    synchronization can dominate a very small drafter

CPU or host retrieval proposer

advantages:
    no neural draft weights on GPU

costs:
    index latency and host-to-device candidate transfer

Integrated head, EAGLE, or MTP

advantages:
    target representation already resident

costs:
    proposer checkpoint is tightly coupled to target architecture/version
    feature or module state needs its own runtime path

Distributed compatibility must be tested against the exact engine release. Current vLLM documentation lists known version-specific incompatibilities and supported methods; do not convert that table into a permanent architectural claim.

0x0BA Ordinary/Parallel Coordination

Serial draft/verify execution preserves the latest target result before the next proposal. Overlapped execution can hide draft time, but it may prepare a continuation from a prefix later invalidated by verification.

SPECTRE analyzes a batch of size $B$, candidate segment length $\gamma$, draft step time $T_D$, target verification time $T_T$, accepted output per round $L$, and invalidated-request fraction $r$. Under its stated assumptions, ordinary speculative decoding has:

$$T_{ord}=T_T+(\gamma-1)T_D,$$ $$\operatorname{Thr}_{ord} = \frac{BL}{T_T+(\gamma-1)T_D}.$$

When $\gamma T_D<T_T$, parallel drafting can be hidden by verification. The fraction $r$ that cannot reuse its pre-generated segment contributes one fallback token, while the rest contributes $L$:

$$\operatorname{Thr}_{par} \approx \frac{B[r+(1-r)L]}{T_T}.$$

Solving $\operatorname{Thr}_{ord}>\operatorname{Thr}_{par}$ gives the paper’s critical invalidation ratio:

$$r^* = \frac{(\gamma-1)LT_D} {[T_T+(\gamma-1)T_D](L-1)}.$$

Parallel coordination is favored only when $r\le r^*$ under this model. The derivation assumes a specific pipeline, steady stage times, $L>1$, and hidden draft work. Network delay, queueing at a shared drafter, variable batch shape, and failed overlap require measured terms rather than an unqualified reuse of the threshold.

Source: SPECTRE, ordinary/parallel coordination figure. The diagram makes overlap and invalidated draft work visible; its timing comparison inherits the paper’s pipeline and workload assumptions.

Source: SPECTRE, Figure 1. The system reuses a small-model service as a remote drafter and switches coordination modes. Reported throughput and resource results remain specific to the paper’s implementation and workload.

0x0BB Remote Protocol And Failure Epoch

Remote proposals need a typed envelope:

RemoteDraftRequest(
    request_id,
    target_iteration,
    base_state_generation,
    manifest_hash,
    prefix_digest,
    max_depth,
    deadline,
)

RemoteDraftResult(
    request_id,
    target_iteration,
    base_state_generation,
    proposal_tokens,
    proposal_probabilities_or_absent,
    drafter_queue_ms,
    drafter_compute_ms,
)

The target never blocks past the request’s speculation deadline. Timeout, version mismatch, stale generation, malformed probability semantics, or remote overload falls back to the already feasible local action, usually K=0. Cancelling the remote request is best effort; a late response from the previous failure epoch cannot mutate current state.

This ordinary/parallel coordination contract is asymmetric: remote work may improve a future target round, but it never acquires authority over emitted tokens, target KV, request completion, or the target-only fallback path.

Remote placement adds metrics absent from a colocated design:

network serialization and transfer latency
remote queue and service time
timeout / stale-result / cancellation rates
native small-model traffic interference
draft reuse and invalidation fraction
target idle time waiting for proposals

0x0C Evaluation Is An Experimental Contract

A throughput number is not evidence unless the treatment, baseline, workload, arrival process, stopping rule, and unit of analysis were fixed before the run. The experiment manifest is immutable and content-addressed:

experiment:
  manifest_version: m7.8-eval-v1
  trace_hash: sha256:...
  split_hash: sha256:...
  target_release_hash: sha256:...
  candidate_release_hash: sha256:...
  request_seed_manifest: sha256:...
  arrival_process: open_loop_trace_replay
  warmup_requests: 500
  measured_requests: 10000
  policies: [target_only, fixed_k_1, fixed_k_3, batch_schedule, adaptive]
  primary_endpoints: [final_output_goodput, p99_tpot, correctness]
  inferential_unit: request
  confidence_level: 0.95
  multiplicity: holm
  stopping: fixed_horizon

The split is made at a dependency-preserving unit such as conversation, repository, tenant trace, or session. Splitting tokens or turns from one conversation between calibration and evaluation leaks proposal difficulty and arrival structure. Controller thresholds, acceptance priors, shape profiles, and graph buckets are frozen before the evaluation split is opened.

0x0CA Correctness Hierarchy Before Performance

Correctness has several non-interchangeable levels:

Greedy token equality is stronger than necessary for a mathematically exact sampler but useful for integration debugging. Conversely, matching a few greedy outputs does not test stochastic exactness. Current vLLM documentation also warns that floating-point order and batch shape can change observed outputs; the release must name whether its promise is token equality, distributional equivalence, or bounded numerical difference.

Cache cases must include all accepted, reject-first, reject-middle, EOS inside the proposal, correction and bonus frontiers, preemption/resume, K=0 state transitions, stale draft state, tree-sibling rollback, and duplicate remote completion.

0x0CB Paired Replay And The Unit Of Analysis

Replay the same request, seed, arrival timestamp, output cap, and admission class under target-only and candidate policies. For request $i$, define a paired difference for metric $Y$:

$$d_i=Y_i^{candidate}-Y_i^{baseline}.$$

The estimated treatment effect is:

$$\widehat\Delta = \frac{1}{n}\sum_{i=1}^{n}d_i.$$

Its classical paired standard error is:

$$SE(\widehat\Delta) = \frac{s_d}{\sqrt n}.$$

Tail latencies, controller actions, and requests sharing one arrival trace are not independent Gaussian samples. Use a paired bootstrap over the highest independent unit, usually request clusters, sessions, or whole replay blocks. Report the point estimate and confidence interval, not only the winning mean. Tokens from one request are never thousands of independent observations.

def paired_block_bootstrap(blocks, metric, draws, rng):
    effects = []
    for _ in range(draws):
        sample = rng.choice(blocks, size=len(blocks), replace=True)
        effects.append(mean(
            metric(block.candidate) - metric(block.baseline)
            for block in sample
        ))
    return quantile(effects, [0.025, 0.5, 0.975])

If five policies are compared with one baseline over several primary endpoints, unadjusted multiple comparisons inflate false discoveries. Predeclare the primary family and apply Holm correction or another named procedure. Exploratory segments remain useful, but their intervals are labeled exploratory rather than promoted to release gates after seeing the data.

Sequential dashboards create another trap. Looking every minute and stopping at the first favorable confidence interval invalidates a fixed-horizon test. Use a fixed sample size or a predeclared group-sequential design with alpha spending. A safety stop for corruption or severe SLO regression is always allowed; it is not evidence of efficacy.

0x0CC Workload And Policy Matrix

Cross the dimensions that change proposal quality or physical execution:

arrivals: closed loop, low/medium/near-saturation open loop, burst trace
lengths: short/short, long/short, short/long, long/long
workloads: code, chat, reasoning, retrieval-heavy, deliberate drift
sampling: greedy, temperature and top-p buckets, supported penalties
state: cold prefill, prefix hit, preempt/resume, adapter mix
placement: colocated, remote ordinary, remote parallel where supported
policies: K=0, each supported fixed K, batch schedule, adaptive controller

K=0 is the causal baseline, not merely one candidate in the adaptive policy. Fixed-$K$ arms reveal whether an adaptive win comes from control or simply from choosing one generally better depth. A batch-size schedule separates online acceptance adaptation from static shape adaptation.

Ragged batches also require a length-aware baseline. Batch Speculative Decoding Done Right groups or pools requests by compatible progress to reduce padding and synchronization work; such grouping changes the scheduler and must be enabled on both arms when the question is the incremental value of speculation.

Source: Batch Speculative Decoding Done Right, system illustration. The figure motivates progress-aware grouping; it does not establish that one grouping policy is universally optimal.

0x0CD Metrics, Denominators, And Release Gates

product:
    TTFT, TPOT, end-to-end latency, timeout and error rate

useful service:
    final-output goodput, committed tokens, completed requests

speculation:
    proposals and acceptance by depth, K histogram, K=0 share,
    query rows per committed token, correction/bonus frontier rate

physical cost:
    proposer/transfer/verify/select/sync/repair time,
    temporary and committed KV, graph hits, all-to-all bytes

queueing:
    arrivals, admission delay, active requests, queue depth,
    preemption and stability margin

Every ratio names its denominator. Acceptance is accepted proposal tokens over proposal tokens that were actually eligible at that depth. Final-output goodput excludes rejected rows. Tail latency is computed over completed requests with timeout and cancellation reported separately, not silently dropped.

A conservative release gate can be written:

$$\operatorname{LCB}(\Delta G)>\delta_G, \qquad \operatorname{UCB}(\Delta P99_{TPOT})<\delta_{SLO}, \qquad N_{correctness\ failure}=0.$$

The thresholds $\delta_G$ and $\delta_{SLO}$ are product decisions fixed before the run. Statistical significance without a material margin does not justify new runtime state.

0x0D Paper And Runtime Contracts Are Not Interchangeable

This table is a paper-by-contract comparison, not a leaderboard. Each source answers a different question. M7.8 composes them only after preserving their assumptions and assigning every state mutation to one runtime owner.

0x0E Release Artifact And Compatibility Contract

The deployable object is a content-addressed bundle, not a target checkpoint plus an informal flag:

speculative_release:
  schema: m7.8-release-v1
  release_id: sha256:...
  parent_release_id: sha256:...
  target:
    checkpoint: sha256:...
    config: sha256:...
    tokenizer: sha256:...
    dtype: bf16
    quantization: none
  proposer:
    method: eagle_or_draft_or_retrieval
    artifact: sha256:...
    feature_or_vocab_mapping: sha256:...
    maximum_topology: {depth: 5, nodes: 5}
  verifier:
    algorithm: rejection_sampling_vX
    sampler_features: [temperature, top_p]
    rng_contract: per_request_counter_v1
  runtime:
    engine_commit: ...
    kernel_build: sha256:...
    driver_and_gpu: ...
    topology: {tp: 1, pp: 1, ep: 1}
    graph_buckets: [0, 1, 3, 5]
  controller:
    profile: sha256:...
    acceptance_schema: v2
    policy: sha256:...
    stability_and_hysteresis: {...}
  evidence:
    immutable_eval_manifest: sha256:...
    correctness_report: sha256:...
    paired_replay_report: sha256:...
    fault_injection_report: sha256:...
  rollback:
    target_only_compatible: true
    prior_release_id: sha256:...

The release ID hashes the canonical manifest payload, excluding the digest field itself and any later operational receipts that refer to it. A profile, acceptance model, or graph bucket cannot be copied into a release whose target, runtime, placement, or physical shape differs.

For speculation inside an M8.10 low-bit deployment, this is a feature-level manifest and evidence boundary, not authority to promote the whole quantized tuple. Freeze the feature manifest, allowed speculative/target-only modes and switching rules in the parent configuration $C$ before evaluation. Operational child receipts then bind that frozen parent and feature identity; they are not inserted back into the parent hash. A registered, evaluated same-target disable can remain within $C$. Adding a mode, changing its switching policy or changing the target artifact/representation requires a new tuple and its release gates.

Compatibility checks occur before load and again before request eligibility:

tokenizer/vocabulary and special-token identity
target architecture, dtype, quantization, and parallel ownership
proposal feature layers, output projection, or retrieval-index lineage
sampler features and RNG semantics supported by the verifier
profile hardware, kernel, graph, KV, adapter, and placement identity
temporary KV/workspace reserve and remote protocol version
rollback target-only path tested for this exact bundle

A migration changes one compatibility generation at a time. Existing requests remain pinned to the generation that owns their target KV and RNG state; new requests enter the new generation. Cross-generation proposal results are stale even when request IDs happen to match.

0x0F Canary, Release, And Rollback

Each stage has an entry artifact, an exposure bound, an exit gate, and an automatic abort condition:

Do not reuse the same canary data to tune thresholds and certify them. A failed stage creates a new candidate release and a new evidence lineage; it does not rewrite the prior report.

0x0FA Rollback Is A State Transition, Not A Cache Clear

ENABLED
  -> DRAINING: stop new speculative rounds; allow owned target work to finish
  -> TARGET_ONLY: materialize visible frontier, discard provisional branches
  -> PROPOSER_QUIESCED: reject remote/local proposals and drain late results
  -> UNLOADED: free draft weights, retrieval index, draft KV, and workspaces
  -> PRIOR_RELEASE: route new requests to the previous compatible bundle

Rollback levels may stop at any safe state. TARGET_ONLY is often enough to restore service without unloading the proposer. No transition discards committed target KV or replays emitted output.

TARGET_ONLY here keeps the same target representation and consumes the registered fallback policy. PRIOR_RELEASE routes new requests; it does not assert that existing target K/V can be attached to a different model or quantized artifact. Existing sessions remain pinned or follow an explicit reconstruction/migration contract admitted by M5.3. Feature rollback cannot replace the M8.10 parent tuple’s quality, resource and operational gates.

def rollback_request_to_target_only(req, failure_epoch):
    assert not req.finished  # continuation path, not terminal cache release
    req.accepting_proposals = False
    cancel_unverified_proposal_work(req)
    discard_uncommitted_target_branches(req)

    if req.pending_frontier_token is not None:
        target_append_frontier(req)

    req.proposal_enabled = False
    req.current_K = 0
    req.failure_epoch = failure_epoch
    req.draft_cache = release_or_quarantine(req.draft_cache)

    assert req.target_cache.token_count == len(req.output)
    assert no_result_from_older_epoch_can_publish(req)

0x0FB Automatic Disable And Causal Evidence

Correctness-unknown state disables immediately. Performance gates use a dwell window and hysteresis unless an absolute SLO or stability boundary is crossed:

correctness:
    cache/position/mask/RNG invariant failure, unsupported sampler,
    unknown proposal provenance, stale generation publication attempt

capacity:
    temporary reserve breach, allocator failure, remote overload,
    MoE straggler/all-to-all limit

performance:
    goodput LCB below target-only margin, TPOT/E2E UCB above SLO,
    stability margin below floor, profile residual outside envelope

readiness:
    missing draft prefill, stale index/cache, graph/profile unsupported

Every trigger writes a causal record:

DisableEvent(
    release_id,
    request_or_server_scope,
    failure_epoch,
    reason_code,
    observed_value,
    threshold,
    profile_generation,
    controller_decision_id,
    state_before,
    state_after,
)

The fallback is fail-closed: unknown profile, cache provenance, proposal state, or verifier support yields K=0. It does not invent a conservative positive depth.

0x10 Failure Lab

0x11 Operational Dashboard And Evidence Closure

The dashboard connects decisions to user outcomes while retaining immutable raw evidence:

service: QPS, admissions, queue, TTFT, TPOT, E2E, timeouts
policy: feasible-action count, rejection reasons, K distribution, K=0 share
utility: committed output, final-output goodput, baseline delta and CI
acceptance: reached/accepted by depth, segment, proposal and target generation
cost: draft, transfer, verify, sync, repair, graph hit, temporary KV
state: frontier count, rollback level, cache invariant and stale-result failures
placement: remote queue/network/deadline, target wait, ordinary/parallel share

Alerts combine absolute SLOs with target-only deltas. A speculative path can be internally stable yet still be worse than ordinary decoding. Each dashboard point links to the release, profile, controller decision, and failure epoch that produced it.

0x11A Implementation References Are Versioned Evidence

The following references are examples of current implementation surfaces, not portable API promises:

vLLM:
    f57123aa2dbc1badb1865b18c724b308f90dceda

TensorRT-LLM:
    43edf6257bcbf55fb5ad27c3d3162351c2547f32

eBay spec_dec:
    4aa1e3424570f64381975576a71fcf0b18440a06

SGLang:
    bc7e1a07c3ae7733f639ab531fe42c084ed31d84

Pinning a commit makes a claim auditable. It does not imply that all four systems expose identical samplers, cache layouts, dynamic-depth policies, or graph behavior.

0x11B Visual And Source Provenance

Paper figures in this article are reproduced for explanation with source and figure context in their captions. SVGs authored for this program are redrawn semantic diagrams, not paper screenshots. Their editable generation script and manifest live under docs/numinens-evidence/m7-8-speculative-serving-lifecycle/. Formulae labeled as teaching models preserve the cited paper’s assumption boundary; production recommendations are identified as synthesis.

0x12 Red Flags

"Speculative decoding is lossless, so rollout is low risk."
    Algorithmic exactness does not validate cache, batching, RNG, or integration.

"Acceptance rose, therefore serving improved."
    Useful output must be divided by complete service cost under load.

"A latency model chose K, so the shape is supported."
    A model cannot authorize an unprofiled graph, memory, or placement shape.

"K=0 means the draft can resume immediately."
    Not after skipped draft prefill or stale cache lineage.

"Rollback means clear speculative caches."
    Materialize visible frontier and preserve committed target state first.

"Every token is an independent evaluation sample."
    Requests and traces own correlated execution and outcomes.

"One successful canary proves every traffic segment."
    Release claims are bounded by the immutable evaluation support.

0x13 Publish Gate

• The owner question and M5.2/M7.1/M7.7 boundaries are explicit.
• Immutable release, request, round, profile, and failure generations exist.
• The full accepted-prefix distribution and its variance are derived.
• Ragged position, mask, output, and KV synchronization are executable.
• Logical output, query rows, temporary state, and committed KV are separate.
• Profiles declare support, residuals, invalidation, and K=0 fallback.
• Queue stability and its pre-saturation assumptions are explicit.
• The controller optimizes a constrained physical action with uncertainty.
• Cache publication and repair distinguish output, target KV, and draft KV; full/partial/zero acceptance and stopping have explicit ledgers.
• Colocated and remote ordinary/parallel placement retain target authority.
• Evaluation uses immutable paired replay, confidence intervals, multiplicity, and a stopping rule.
• Release and rollback preserve generations, committed state, and causal evidence.
• Paper claims, runtime observations, and implementation versions are scoped.
• Formula, source, local-link, asset, desktop, and mobile audits close.

0x14 M7 Completion Boundary

M7 now spans the path from exact algorithm to reversible production system:

M7.1: one-sequence probability and correction contract
M7.2-M7.7: proposal, topology, training, retrieval, and verification choices
M7.8: synchronized service, measured control, evaluation, release, rollback

The deployment conclusion is conditional by design. Speculative decoding is not a property of a checkpoint. It is a joint property of proposal quality, target verification, tensor layout, cache transactions, traffic, scheduling, kernels, placement, and one versioned release artifact. When that joint contract stops holding, target-only decoding remains the ordinary, measurable, and state-preserving continuation.

References

1. Qidong Su, Christina Giannoula, Gennady Pekhimenko. The Synergy of Speculative Decoding and Batching in Serving Large Language Models.
2. Xiaoxuan Liu et al. TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput.
3. Linghao Kong et al. An Interpretable Latency Model for Speculative Decoding in LLM Serving.
4. Woosuk Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention.
5. Yaniv Leviathan, Matan Kalman, Yossi Matias. Fast Inference from Transformers via Speculative Decoding.
6. Ranran Haoran Zhang et al. Batch Speculative Decoding Done Right.
7. Jincheng Xie et al. SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference.
8. vLLM. Speculative Decoding and Dynamic Speculative Decoding, documentation inspected against commit f57123a.
9. NVIDIA. TensorRT-LLM Speculative Decoding, implementation inspected against commit 43edf62.
10. eBay. spec_dec, commit 4aa1e34.
11. SGLang. Speculative decoding implementation, commit bc7e1a0.
12. Immi’s Workshop. A30 - Decoding Speculation.
13. Immi’s Workshop. A31 - Medusa.
14. Immi’s Workshop. A32 - Lookahead Decoding.
15. Immi’s Workshop. A33 - DeepSeek MTP.