M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure

M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure

0x00 Overview And Background

0x00A From One Cached Loop To A Stateful Service

The mathematical Transformer describes a mapping from visible token rows to new hidden rows. It does not, by itself, specify how a service should preserve the intermediate state of an unfinished request. That distinction is easy to miss when inference is introduced with one prompt and one generation loop.

M3.10 owns that loop. It derives the single-request state transition from prompt prefill to a token decision, the next decode input, cache append, position advance, stopping, streaming, beam reorder, and cache/no-cache equivalence. Its most important boundary is the split between emitting a token and processing that token so its own K/V row becomes committed state.

M4.1 then supplies the rectangular accounting inputs: layer count, K/V-head count, head width, representation bytes, phase, and sequence extent. M4.2 through M4.6 explain how those logical operations become kernels, transfers, reductions, and hardware schedules. Together they establish two very different attention frontiers:

prefill:
    many new query rows
    many new key/value rows
    enough same-request work to tile across the prompt

incremental decode:
    a small new query frontier, ordinarily one row per request
    a long retained key/value history
    one new K/V row per processed input token and layer

Those modules intentionally stop before the serving problem. A rectangular formula can describe one request at one instant. A service sees many independently growing request lifetimes:

request A: prefill -> decode -> decode -> finish
request B:          prefill -> decode -> cancel
request C:                    prefill -> fork -> two decode branches
request D:                                      arrive and wait

The historical progression is easiest to understand as a change in the object being optimized. This is a problem-oriented reconstruction, not a claim that the ideas appeared in one strict chronological sequence.

Transformer Series A20 - KV Cache provides the first two stages in detail: it follows the growing autoregressive prefix, identifies repeated historical work, and explains why K and V are the ordinary retained attention state while the current Q row is transient. ORCA makes the next systems step explicit: a generative request spans many model iterations, so the scheduler can act at iteration granularity rather than treating a whole request batch as one indivisible job. PagedAttention then makes the memory consequence concrete: per-request K/V is dynamic, capacity-limiting state whose allocation and sharing affect feasible serving throughput.

M5.1 begins at the boundary shared by those observations. It does not reteach the full generation loop, select the next active batch, or choose physical pages. It asks:

What persistent model state does each live request create, grow, commit, fork, truncate, and release, and what resource interface does that state expose to a serving system?

The answer is broader than “keep two tensors.” M5.1 defines the logical, versioned per-request cache object that later serving mechanisms schedule, map, retain, move, reshape, or represent differently.

0x00B The Persistent Object

For one layer, incremental attention consumes a current query row and a visible history of K/V rows. Across a model, a live request therefore owns a collection of layer-indexed state. Across a service, that collection also needs identity, positions, schema, lifecycle, and ownership metadata.

logical request state:
    request identity
    prompt and committed token identity
    current position / cache position
    stop, fork, and policy metadata

model-derived persistent state:
    K and V rows for every cache-producing layer
    architecture-specific cache state where the model does not store full K/V

runtime ownership state:
    logical cache handle and generation/version
    lifecycle phase and worker ownership
    references needed to append, fork, truncate, or release safely

The object survives one kernel launch. Prefill produces its initial model state. A later decode call reads it and prepares an append. The request may wait while those bytes remain live. A cancellation must release its ownership; a fork must preserve the common committed prefix; a retry must not publish into a stale generation. The physical implementation may be a tuple, one static tensor, dynamic tensors, paged blocks, virtual pages, offloaded tiers, or a latent cache. Those are representations of the object, not its semantic definition.

This separation prevents three common category errors:

token history is not the same object as per-layer K/V state
logical K/V payload is not the same number as allocated or resident bytes
an emitted token is not necessarily a token whose own K/V row is committed

This section freezes that conceptual object before later sections make its manifest, append transaction, ragged byte trajectory, failure states, and runtime event surface executable.

0x00C Owner Contract: Consumes, Owns, And Defers

The ownership rule is not “mention a topic only once.” It is “derive each contract where its assumptions can be stated completely.” M5.1 may expose a pressure that a later article resolves, but it must not silently choose that article’s mechanism.

This article can therefore say that live state constrains scheduling without deriving ORCA or SARATHI policy. It can define fork without implementing M5.3 copy-on-write. It can require branch rollback without reproducing M7’s acceptance algorithms. It can count representation bytes without teaching the quantizer that M8.6 owns.

0x00D Source Continuity And Corrections

A20 remains useful because it introduces cache reuse from the computation the reader already understands. Two of its conclusions cannot be carried forward unchanged:

The correction is not a rejection of caching. It identifies the exchange precisely:

without retained K/V:
    less persistent request state
    more repeated historical computation

with retained K/V:
    less repeated historical computation during incremental decode
    more persistent state whose lifetime the service must manage

That exchange is the reason M5 exists.

Claim Status And Evidence Classes

This article uses five claim statuses.

The core M5.1 claim is:

KV cache is not only a tensor optimization.
It is the persistent request state that makes serving a scheduler problem.

Background knowledge: inference creates persistent request state

A training batch can release most forward activations after its optimization step. Autoregressive serving is different: every unfinished request carries information that the next iteration needs. The KV cache is the largest model-derived part of that state, but it is not the whole request.

logical state:
    prompt, emitted tokens, stop status, positions, policy metadata

model state:
    per-layer cached K/V and any architecture-specific latent cache

runtime state:
    allocation handles, scheduler phase, worker placement, ownership generation

This is why an inference server behaves like a stateful execution engine rather than a stateless collection of model calls. Cancellation, preemption, retries, prefix reuse, and admission all have to preserve or release the right state under explicit ownership rules.

The next section reconstructs the minimum request/cache trace needed to expose that serving object. M3.10 remains authoritative for the complete generation algorithm and policy semantics.

0x01 One Request Is A State Machine

The smallest useful serving trace is:

1. receive prompt tokens
2. run prefill over the full prompt
3. create K/V cache rows for every layer
4. sample or choose the first generated token
5. run one decode step
6. append one K/V row per layer
7. repeat decode until stop condition
8. release, reuse, page, or transfer the cache

A20’s request-level picture is a useful baseline because it shows why serving state exists before naming any cache implementation. The prompt is processed once, then every token-generation invocation depends on the prompt plus all tokens emitted so far:

Source: Exploring the Transformer Series (20) - KV Cache, request-lifecycle figure. The diagram shows the logical autoregressive dependency. It does not yet show which intermediate state is reused; the KV lifecycle below opens that missing object.

Without reuse, the growing prefix suggests recomputing historical projections at every invocation. KV caching changes the physical execution while preserving that logical dependency: old K/V rows remain live, and each decode iteration computes and appends only the new row.

The important transition is not only mathematical. It is operational:

The cache is not a side note. It is the state that makes step 5 cheap enough to repeat.

PagedAttention’s opening measurement makes the capacity consequence concrete. In its 13B-model/A100-40GB example, model parameters occupy about 65% of memory, while request-dependent KV state consumes more than 30%. The right panels connect the allocator to batching: reservation-heavy allocation reaches the memory ceiling after only a small batch increase, whereas demand-driven allocation grows more smoothly and admits more requests in that experiment.

Source: Kwon et al., PagedAttention, Figure 1. The memory shares, allocation curves, and throughput lines are from the paper’s 13B-model NVIDIA A100 40GB setup. The reusable conclusion is that persistent KV allocation can determine feasible batch size; the exact percentages and request counts are not portable to another model, dtype, allocator, or GPU.

During prefill, the model processes the prompt rows together. At each layer it projects hidden states into queries, keys, and values. The keys and values are stored because future generated tokens will need to attend back to them.

During decode, the model receives one new position. It still projects the new hidden row into Q, K, and V, but the attention calculation for that new query uses:

K_cache: all previous key rows for this request and layer
V_cache: all previous value rows for this request and layer
K_new:   key row for the current generated token
V_new:   value row for the current generated token

The current step then appends K_new and V_new to the cache. The next step sees a longer cache.

This is the key shape transition:

prefill cache after prompt length S:
    K_l: [B, H_kv, S, d_h]
    V_l: [B, H_kv, S, d_h]

after one decode step:
    K_l: [B, H_kv, S + 1, d_h]
    V_l: [B, H_kv, S + 1, d_h]

Here l is a layer index. The full cache is a list or block-managed collection of this object for every layer.

At implementation level, the loop is closer to this:

for layer l in transformer_layers:
    h = hidden_state entering layer l

    q_l = h @ Wq_l
    k_l = h @ Wk_l
    v_l = h @ Wv_l

    if prefill:
        cache[l].K = k_l for every prompt position
        cache[l].V = v_l for every prompt position
        h = attention(q_l, cache[l].K, cache[l].V)

    if decode:
        cache[l].K.append(k_l for current position)
        cache[l].V.append(v_l for current position)
        h = attention(q_l for current position, cache[l].K, cache[l].V)

Real implementations vary in the exact order. Some append before attention so the current token is present in the causal range. Some store separate tensors per layer. Some store paged blocks. Some use tuple-style caches, static caches, or block tables. But the dependency is the same:

layer l at decode step t cannot compute attention
unless the cache for layer l contains the previous positions.

That is why the cache is a per-layer object, not one global text buffer. A token does not have one K/V row for the whole model. It has one K row and one V row at every layer that uses self-attention.

The query side is the common source of confusion. In standard incremental decode:

q_t:
    computed for the current position
    used to read the K/V history
    not retained as the serving cache object

k_t, v_t:
    computed for the current position
    appended to the layer cache
    reused by future positions

A20 includes a subsection arguing that Q is “cached to some extent” because information flows through later layer representations. The rewrite should not turn that into an implementation claim. The persistent serving object exposed by mainstream cache APIs is K/V per layer. The current Q row is a transient consumer of that state.

Logical Cache Interface

Hugging Face’s cache documentation gives the clean implementation contract:

new_K, new_V = cache.update(k_t, v_t, layer_idx)
attn_output = attention(q_t, new_K, new_V)

The same docs describe the cache as a list of layers, where each layer contains a key cache and a value cache shaped like:

[batch_size, num_heads_or_num_kv_heads, seq_len, head_dim]

M5.1 uses that logical shape. M5.3 replaces the simple contiguous picture with block tables, virtual memory, prefix references, and exact reuse. Those mechanisms change allocation and addressing; they do not remove the per-layer K/V state requirement.

Immutable Request And Cache Manifest

The logical interface is still underspecified. Two caches can have the same rank and length while representing different model states. A row computed by a different model revision, adapter set, positional rule, or K/V representation is not compatible merely because its tensor shape fits.

The runtime therefore needs a record that answers two separate questions:

request record:
    which live attempt, lineage, input, policy, and state snapshot is this?

cache compatibility record:
    which exact model-state contract produced these persistent rows?

The article-local fixture represents both in one immutable manifest record:

request_identity
    namespace, request_id, attempt, lineage, parent request

artifact_identity
    model artifact, model configuration, adapter set, weight transforms

input_identity
    exact prompt token IDs, tokenizer/template provenance,
    side-input identity, cache-isolation salt

cache_schema
    cache family, layer count, K/V heads, head width, dtype widths,
    positional stage, logical axes, representation schema, parallel shard

position_and_visibility
    cache-position convention, RoPE configuration, visibility policy

request_policy
    context/generation limits, stopping policy, sampling policy

state_snapshot
    generation, phase, logical/committed/visible lengths,
    reserved capacity, resident bytes, reclaimable bytes

Why is this an immutable record rather than one mutable dictionary hanging off a request? Because identity and current state have different update rules. Model/cache compatibility should not drift during a request. Runtime state does change, but each change can produce a new generation-stamped snapshot rather than silently editing the evidence for an older publication or transfer.

manifest generation g:
    immutable envelope + immutable snapshot(g)

append transaction:
    references generation g and its cache ABI

successful publication:
    creates snapshot(g + 1)

late completion for g:
    cannot mutate snapshot(g + 1)

This is a semantic transaction log, not a demand that every runtime serialize JSON on its hot path. A production implementation can keep packed structs and integer IDs, provided those fields reconstruct the same contract.

Deriving Cache ABI And Prefix Identity

One giant hash would be easy to compute but hard to reason about. M5.1 instead derives two identities from projections of the manifest.

The cache ABI identifies everything required to interpret a K/V row:

cache_abi_sha256 = SHA256(canonical_json({
    artifact_identity,
    cache_schema,
    position_contract
}))

artifact_identity includes the base artifact, configuration, active adapter set, and weight transforms because each can change projected K/V values. cache_schema includes the logical tensor axes and local shard because equal byte counts do not imply equal head ownership. The position contract includes cache-position, RoPE, and visibility semantics because a numerically valid row at the wrong position is semantically invalid.

The prefix identity asks whether a specific prompt state may be reused:

prompt_token_ids_sha256 = SHA256(canonical_json(prompt_token_ids))

prefix_identity_sha256 = SHA256(canonical_json({
    cache_abi_sha256,
    prompt_token_ids_sha256,
    input_side_data_sha256,
    cache_isolation_salt_sha256
}))

This decomposition explains several deliberate inclusions and exclusions:

The exclusions are as important as the inclusions. Putting request_id in the semantic prefix key would prohibit all cross-request prefix reuse. Omitting an adapter or side-input identity would permit shape-correct but stale reuse. Recording tokenizer/template provenance still matters for replay and audit, even though exact token IDs are the semantic text input to this cache record.

This matches the contracts exposed by serving implementations. Hugging Face Transformers’ cache explanation defines absolute cache positions and per-layer K/V state. vLLM’s automatic prefix-caching design includes parent/block token identity plus extra hashes such as LoRA, multimodal, and cache-salt identity. TensorRT-LLM’s K/V reuse documentation likewise warns that identical pseudo-token IDs need extra identity when prompt embeddings differ. M5.3 owns their concrete block keys and lookup structures; M5.1 owns the compatibility facts those keys must not erase.

Source-To-Tensor Trace

The manifest is useful only if each field can be traced to the state it guards:

model artifact + adapter + transforms
    -> W_K^(l), W_V^(l) and all upstream hidden states

prompt token IDs + side inputs
    -> hidden row h_t^(l)

position / RoPE contract
    -> position-transformed key row K_t^(l)

cache schema + TP shard
    -> local K_t^(l), V_t^(l) shape and ownership

representation schema
    -> stored bytes plus scale/metadata interpretation

all of the above
    -> one compatible persistent K/V row

For example, suppose two requests contain token IDs [11, 22, 33, 44] and produce local rows shaped [2 K/V heads, 4 channels]. If one request activates a different LoRA adapter, their shapes and token IDs still match, but their hidden states or projection weights may differ. The ABI hash must differ, so a prefix directory cannot treat the rows as interchangeable.

Six State Predicates Are Not Synonyms

After identity is frozen, the current snapshot must keep six quantities separate. Collapsing them into one cache_length or memory_used field loses the state transitions later articles need.

The logical count may be one ahead of committed cache state. After the model emits token x_4, the authoritative output can contain five tokens while the cache still contains rows for positions [0, 4). Token x_4 becomes a cache row only when a valid subsequent model call processes it. That distinction is also why speculative correction output cannot borrow the rejected draft token’s row.

Worked Snapshot And Byte Ledger

Use a deliberately small full-K/V cache:

L = 2 cache-producing layers
H_kv = 2 local K/V heads
d_h = 4 channels per head
b_K = b_V = 2 bytes per scalar

One token contributes one K row and one V row at every layer, so:

$$B_{token} = L H_{kv} d_h (b_K+b_V).$$

The two dtype terms make the familiar factor of two explicit. They also allow K and V to use different representations instead of hiding that assumption:

B_token = 2 * 2 * 4 * (2 + 2) = 64 bytes/token

For the fixture snapshot:

logical token count:                 5 tokens
committed cache length:              4 tokens
visible positions:                   [0, 4)
reserved token capacity:             8 slots
resident payload:                    GPU 256 B, host 0 B
reclaimable resident payload:        GPU 0 B, host 0 B

Therefore:

$$M_{committed,logical}=T_{committed}B_{token}=4\times64=256\ \text{bytes},$$ $$M_{reserved,nominal}=T_{reserved}B_{token}=8\times64=512\ \text{bytes}.$$

Renderer-safe fallback:

B_token             = L * H_kv * d_h * (b_K + b_V)
M_committed_logical = T_committed * B_token = 256 B
M_reserved_nominal  = T_reserved  * B_token = 512 B

The reserved figure is a nominal capacity ledger, not a claim that the allocator consumed exactly 512 bytes. M5.3 must add block rounding, padding, metadata, fragmentation, and sharing. The resident figure is a declared state counter in this synthetic fixture, not a local GPU measurement. Its equality to logical committed payload here is intentional but not universal.

Validation And Negative Fixtures

The machine-readable fixture is validated before it is cited as evidence:

def validate_manifest(m):
    assert digest(m.prompt_token_ids) == m.prompt_token_ids_sha256
    assert digest(cache_abi_projection(m)) == m.cache_abi_sha256
    assert digest(prefix_projection(m)) == m.prefix_identity_sha256

    assert m.committed_cache_length <= m.logical_token_count
    assert m.next_cache_position == m.position_origin + m.committed_cache_length
    assert every_visible_position_is_committed(m)
    assert m.reserved_token_capacity >= m.committed_cache_length
    assert reclaimable_bytes(m, tier) <= resident_bytes(m, tier)
    assert recompute_ledger(m) == m.analytical_ledger

Eighteen negative fixtures mutate one contract at a time. They cover malformed artifact hashes, adapter drift without a new ABI, prompt-count/digest drift, zero layers or dtype width, shard mismatch, position gaps, visibility beyond commit, reservation below commit, impossible model limits, reclaimable bytes above resident bytes, ledger drift, and stale snapshot/prefix digests.

The executable evidence lives beside the article:

docs/numinens-evidence/m5-1-serving-state/
    request_cache_manifest.json
    invalid_manifest_fixtures.json
    cache_manifest.py
    test_cache_manifest.py

This oracle proves internal consistency of the declared fixture. It does not measure allocator behavior, physical residency, transfer latency, or hardware capacity.

Cache Position, Attention Mask, And Append Ordering

The logical cache.update call hides three pieces of runtime state that must agree:

cache_position:
    absolute slot occupied by each new token in the cache

position_ids / RoPE position:
    position used by the model's positional mechanism

attention mask:
    which cached and current positions the new query may see

They are related, but they are not interchangeable. cache_position describes storage order and is not changed by left padding. Position IDs can be derived under a model-specific padding or packing convention. The mask expresses visibility.

If the cache already contains T_past rows and the current call contributes T_new rows, a conventional causal cache update needs:

new K/V input:
    [B, H_kv, T_new, d_h]

updated logical K/V:
    [B, H_kv, T_past + T_new, d_h]

attention mask key length:
    T_past + T_new

cache positions:
    [T_past, ..., T_past + T_new - 1]

Why must the key-side mask cover T_past + T_new? The current queries score against both old cached keys and the keys produced by this call:

scores [B, H_q, T_new, T_past + T_new]

A mask of length only T_new does not describe the old prefix. A mask of length only T_past omits the new rows. Shape checks may catch this; broadcasting can also turn it into a quieter semantic bug.

One-Layer Tensor Trace, Then The Full Layer Chain

The compact statement “append one K/V row” hides two different dataflows:

within one layer:
    hidden rows -> normalization -> Q/K/V -> position transform
    -> current-call K/V view -> attention -> residual/MLP output

across layers:
    output of layer 0 -> input of layer 1 -> ... -> output of layer L-1

The second line is not bookkeeping. Layer ell+1 does not project its K/V from the token embedding or from layer 0’s input. It projects from the hidden rows produced by layer ell. A cache is therefore a stack of different representations of the same token positions, not one K/V tensor copied across layers.

For a decoder layer with input h_new^(ell):

x_new^(ell) = Norm(h_new^(ell))

Q_new^(ell) = reshape(x_new^(ell) W_Q^(ell))
             [B, H_q,  T_new, d_h]

K_raw^(ell) = reshape(x_new^(ell) W_K^(ell))
             [B, H_kv, T_new, d_h]

V_new^(ell) = reshape(x_new^(ell) W_V^(ell))
             [B, H_kv, T_new, d_h]

For a RoPE model, the stored key stage must be explicit:

Q_pos^(ell), K_pos^(ell) = RoPE(Q_new^(ell), K_raw^(ell), position_ids)

stored row under this article's ABI:
    K_pos^(ell), V_new^(ell)

not stored:
    Q_pos^(ell)

V is not rotated in ordinary RoPE attention. A runtime that stores pre-RoPE keys can be valid only under a different cache ABI that promises to apply the missing transform before use. Relabeling pre-RoPE bytes as post-RoPE bytes is a semantic error even when every shape matches.

The current layer needs both committed history and the rows produced by the current call:

K_used^(ell) = concat(K_committed^(ell), K_pos^(ell), axis=position)
V_used^(ell) = concat(V_committed^(ell), V_new^(ell), axis=position)

K_used, V_used:
    [B, H_kv, T_past + T_new, d_h]

For GQA, one K/V head serves a group of query heads. Let:

g = H_q / H_kv

Then the attention operation needs the logical head-expanded view:

K_attn = repeat_or_broadcast(K_used, groups=g)
V_attn = repeat_or_broadcast(V_used, groups=g)

K_attn, V_attn:
    [B, H_q, T_past + T_new, d_h]

scores = Q_pos @ transpose(K_attn) / sqrt(d_h) + causal_mask
scores:
    [B, H_q, T_new, T_past + T_new]

That expansion is an attention-interface operation. It does not change the persistent cache schema from H_kv to H_q. A backend may broadcast the groups, use a strided view, or materialize an expanded temporary. Those choices change temporary traffic and kernel behavior; they do not multiply the stored K/V payload. M5.7 owns the trained MHA/MQA/GQA mapping, while this article consumes its declared H_q and H_kv.

Worked Two-Layer Decode Trace

The executable fixture uses a deliberately small GQA model:

B = 1       L = 2       d = 16
H_q = 4     H_kv = 2    d_h = 4
T_past = 4  T_new = 1   fp16 K/V
cache_position = [4]

The exact trace is easier to audit as two layer-local ledgers.

Layer 0

• Hidden input: model input h_new^(0), shape [1, 1, 16].
• Query after reshape and RoPE: [1, 4, 1, 4].
• Pending K and V: two tensors of shape [1, 2, 1, 4].
• Usable K/V before query-head expansion: [1, 2, 5, 4].
• Logical K/V after group expansion: [1, 4, 5, 4].
• Scores: [1, 4, 1, 5].
• Hidden output: layer 0 residual/MLP output, which becomes layer 1 input.

Layer 1

• Hidden input: layer 0 output h_new^(1), shape [1, 1, 16].
• Query after reshape and RoPE: [1, 4, 1, 4].
• Pending K and V: two tensors of shape [1, 2, 1, 4].
• Usable K/V before query-head expansion: [1, 2, 5, 4].
• Logical K/V after group expansion: [1, 4, 5, 4].
• Scores: [1, 4, 1, 5].
• Hidden output: layer 1 residual/MLP output leaving the two-layer stack.

The pending payload for one layer follows directly from the K and V shapes:

B_layer_append
  = B * T_new * H_kv * d_h * (b_K + b_V)
  = 1 * 1 * 2 * 4 * (2 + 2)
  = 32 bytes

B_all_layers_append
  = L * B_layer_append
  = 2 * 32
  = 64 bytes

Changing H_q from 4 to 8 while keeping H_kv=2 changes query and score shapes, but the oracle still reports 64 persistent pending bytes. Changing H_kv would change the payload. This is the accounting distinction that a single generic num_heads field cannot preserve.

Current-Call Use Is Not Next-Call Publication

There are two visibility boundaries in this trace:

inside layer ell during the current forward:
    committed rows [0, T_past)
    plus this layer's transaction-local rows [T_past, T_past + T_new)
    are usable by this layer's current attention

to a later independent model call:
    only the all-layer committed prefix is a legal snapshot

The distinction resolves an apparent contradiction. The current token’s K/V must participate in its own layer-level causal attention, so the layer-local K/V view has length 5 in the fixture. But the serving handle remains at committed length 4 until every cache-producing layer has a complete K and V row for position 4. After the model call completes successfully, one logical publication advances the next-call snapshot to length 5.

def model_call_with_staged_append(h_new, committed, positions, mask):
    pending = begin_append(base_length=committed.length, positions=positions)

    for layer_idx, layer in enumerate(model.layers):
        x_new = layer.input_norm(h_new)
        q_new, k_new, v_new = layer.project_qkv(x_new)
        q_new, k_new = layer.apply_rope(q_new, k_new, positions)

        # Same-layer attention sees committed history plus this call's rows.
        k_used, v_used = pending.stage_layer_and_build_view(
            layer_idx, k_new, v_new, committed.layer(layer_idx)
        )
        h_new = layer.finish(q_new, k_used, v_used, mask, residual=h_new)

    # A later call may advance only after every layer interval is complete.
    return h_new, publish_all_layers(pending)

This is a serving contract, not a claim that every framework cache already implements an atomic transaction. In Transformers v4.57.0, Cache.update mutates one layer object while the model walks the layer stack. Ordinary single-threaded generation naturally waits for the forward call to return. A concurrent service that can cancel, retry, migrate, or expose handles across workers needs a wrapper that prevents a partially advanced framework object from becoming the next request snapshot. Loop 6 proves the generation fence, cancellation, retry, and ABA-resistant publication rules.

Prefill And Decode Use The Same Append Interface

The difference between prefill and decode is not a different K/V schema. It is the width of the new-row frontier and the resulting attention program:

first prompt prefill:
    T_past = 0
    T_new  = prompt length P
    cache_positions = [0, ..., P-1]
    Q/K/V contain P new rows per layer
    causal mask is triangular over the new prompt rows

incremental decode:
    T_past = committed prefix length
    T_new  = 1 in the ordinary loop
    cache_positions = [T_past]
    one Q/K/V row is produced per layer

cached continuation or chunked prefill:
    T_past > 0
    T_new  > 1
    current queries see committed history plus causally legal current rows

The local test suite executes both T_past=4, T_new=1 and T_past=0, T_new=4 through the same shape and publication validator. This is a contract test, not a latency measurement and not M5.2’s chunk-scheduling policy.

Dynamic, Static, Sliding, And Offloaded Cache Choices

Transformers v4.57.0 cache documentation and its pinned cache_utils.py make the logical/physical split unusually visible. Cache.update selects one layer, passes it the new key/value states, and returns the K/V tensors that the current attention call should use. The layer class decides how those states are retained.

DynamicLayer. The per-layer update concatenates new K/V on the sequence axis and returns observed history plus current rows. Logical valid length and backing length grow together.

StaticLayer. The update performs an indexed in-place write at cache_position and returns fixed max_cache_len backing tensors. The valid mask and committed length, not allocated tensor shape, identify history.

Dynamic or static sliding layer. The update retains a bounded suffix with offset/window logic. The current usable view can differ from the retained post-call suffix while the update crosses the window boundary. Architecture changes visible history here; this is not merely an allocator choice.

Cache with offloading=True. The cache prefetches a layer, updates it, prefetches the next layer, then offloads the completed layer according to policy. After synchronization, attention sees the same logical K/V interface. Residency changes; logical token identity and history do not.

Static cache exposes why these predicates were separated in Loop 2. A tensor with shape [B, H_kv, max_cache_len, d_h] does not mean every slot is a valid or committed token. In the v4.57.0 implementation, StaticLayer writes with index_copy_ when possible and returns the fixed backing tensors; the causal mask and cache-position contract prevent unused slots from becoming visible.

Sliding cache exposes a different boundary. Once saturated, the retained physical suffix can remain bounded even though the cumulative logical position keeps advancing. The cache layer therefore reports mask sizes and an offset. M5.4 owns the consequences of changing retained/visible history; M5.1 records the position and visibility policy identity required to interpret that state.

Offloading is orthogonal in this framework revision. It can wrap dynamic or static layer objects. The top-level cache waits for the prefetch stream, prefetches a later layer, updates the current layer, and then offloads according to policy. That is evidence that the logical layer update can survive a residency change, not a proof that transfers are free. M5.5 owns transfer, prefetch, fallback, and tier-policy mechanics.

Pinned Framework Source Contract

Implementation claims in this section are pinned rather than attributed to a floating main branch:

project: huggingface/transformers
release: v4.57.0
commit:  8ac2b916b042b1f78b75c9eb941c0f5d2cdd8e10

cache_utils.py SHA-256:
f4e11caf14c75bdefa5e3003b93ef0e8b97fb6103375705ad944b0b344d7da8a

modeling_llama.py SHA-256:
31bf660a663259134324bc65da4e155951dc89c5ca46471d2325a9938e859e26

The pinned modeling_llama.py provides the source-to-tensor order used above:

LlamaModel:
    embed input -> derive positions/mask -> loop over decoder layers

LlamaDecoderLayer:
    normalize -> self-attention -> residual
    -> normalize -> MLP -> residual -> next layer

LlamaAttention:
    Q/K/V projections -> RoPE(Q,K) -> cache.update(layer_idx)
    -> attention backend -> output projection

eager attention backend:
    expand K/V groups with repeat_kv -> QK^T -> mask/softmax -> PV

This source snapshot is implementation evidence, not a universal API. A20’s older StaticCache excerpt remains useful historical context, but constructor arguments and internal organization have changed. In v4.57.0, StaticCache is built from config and max_cache_len, chooses per-layer static or sliding objects from the model configuration, and lazily infers batch/head/device/dtype properties from the first key tensor. Copying an unpinned framework excerpt into production code would erase exactly the version boundary this article is trying to preserve.

A Complete Logical Cache Needs More Than append

Greedy decoding makes a cache look append-only because one live sequence grows by one position at a time. A serving object also has to survive branching, selection, rollback, cancellation, retries, and delayed completions. The pinned Transformers v4.57.0 Cache already exposes reorder_cache, crop, batch_repeat_interleave, and batch_select_indices in addition to per-layer update. That is useful implementation evidence that the state surface is broader than append. It is not evidence that the framework object itself provides concurrent, all-layer-atomic serving transactions.

The logical operations and their mechanism owners are:

This table is the abstract object. M5.3 owns how a paged or virtual-memory runtime implements shared prefixes, physical references, leases, copy-on-write, and atomic mapping replacement. A logical fork may be implemented by copying, sharing, or remote references as long as the visible result is the same.

Versioned Transactions And Lifecycle Operations

The object needs two related state machines. A snapshot is immutable and readable by later model calls. A transaction may contain incomplete work that must not become a snapshot yet.

request snapshot:
    DECODE_READY --finish--------> FINISHED --release--> RELEASED
          |                             ^
          +--cancel_request--> CANCELLED+--release------>
          |
          +--append/truncate--> new DECODE_READY generation
          +--fork-----------> child generation 0; parent unchanged

append transaction:
    OPEN --all L layers complete--> PREPARED --publish--> PUBLISHED
      |                                  |
      +-------------cancel--------------+-----------> ABORTED
      +-------------stale base---------------------> REJECTED

FINISHED and RELEASED are deliberately separate. Finish is a generation decision: no later append should begin. Release is an ownership decision: the logical handle becomes terminal. Even then, physical bytes may wait for kernel fences, child references, transfers, or allocator policy. Conversely, a transaction can be aborted without cancelling the request; the previous committed snapshot remains valid.

A compact logical schema is:

@dataclass(frozen=True)
class LogicalKVSnapshot:
    handle_id: str
    lineage_id: str
    manifest_record_sha256: str
    cache_abi_sha256: str
    snapshot_sha256: str
    generation: int
    committed_length: int
    layer_state_handles: tuple[LayerStateHandle, ...]
    phase: Literal["DECODE_READY", "FINISHED", "CANCELLED", "RELEASED"]

@dataclass
class KVAppendTxn:
    txn_id: str
    idempotency_key: str
    handle_id: str
    base_generation: int
    base_snapshot_sha256: str
    cache_abi_sha256: str
    position_start: int
    position_count: int
    pending_layers: dict[int, PendingLayerRows]
    state: Literal["OPEN", "PREPARED", "PUBLISHED", "ABORTED", "REJECTED"]

The snapshot digest is provenance, not a synchronization primitive. Publication still needs a lock, compare-and-swap, database conditional write, or another mechanism that makes one transition appear atomic. This follows the linearizability contract of Herlihy and Wing: a concurrent operation must have a legal sequential interpretation and an instantaneous effect point between invocation and response. Here, that effect point is the successful publication of the successor snapshot, not the first layer write and not transaction creation.

Deriving The All-Layer Publication Invariant

Let S_g be the committed snapshot at generation g, with committed length T_g. Its cache invariant is:

$$\mathcal{I}(S_g): \quad \forall \ell \in \{0,\ldots,L-1\}, \quad |K_\ell| = |V_\ell| = T_g.$$

An append transaction stages the half-open interval [T_g, T_g + Delta) for every layer. Before publication, a later independent model call still resolves handle_id to S_g; pending layer rows are reachable only through the transaction. A transaction is prepared exactly when:

$$\operatorname{prepared}(X) \iff \bigwedge_{\ell=0}^{L-1} \left( K_{\ell,X}\ \text{complete} \land V_{\ell,X}\ \text{complete} \land I_{\ell,X}=[T_g,T_g+\Delta) \right).$$

Publication compares the complete logical base, not a memory address:

expected = (handle_id, generation g, snapshot_sha256 s_g)
desired  = (handle_id, generation g+1, snapshot_sha256 s_(g+1))

CAS_or_locked_publish(expected, desired)

If it succeeds, the successor points to layer states whose lengths are all T_g+Delta. If it fails, no pending row becomes part of the current logical snapshot. Therefore a reader that resolves the handle before the publication sees S_g, while a reader after it sees S_(g+1); neither legal read can see layer 0 at T_g+Delta and layer L-1 at T_g. This is the article’s abstract all-layer atomicity proof. M5.3 must still realize the pointer/block mapping change safely for its chosen physical representation.

def publish_append(live, txn, num_layers):
    expected = (
        txn.handle_id,
        txn.base_generation,
        txn.base_snapshot_sha256,
    )

    assert txn.state == "PREPARED"
    assert txn.cache_abi_sha256 == live.cache_abi_sha256
    assert txn.position_start == live.committed_length
    assert set(txn.pending_layers) == set(range(num_layers))

    for layer_idx, rows in txn.pending_layers.items():
        assert rows.cache_abi_sha256 == live.cache_abi_sha256
        assert rows.interval == (
            txn.position_start,
            txn.position_start + txn.position_count,
        )
        assert rows.k_complete and rows.v_complete

    successor = build_unpublished_successor(live, txn)
    if not live.compare_and_publish(expected, successor):
        txn.state = "REJECTED"
        discard_pending_rows(txn)
        return STALE_BASE

    txn.state = "PUBLISHED"
    return durable_receipt(txn.txn_id, txn.idempotency_key, successor)

The compare_and_publish call is one logical step. It does not require one particular hardware CAS, and build_unpublished_successor does not authorize readers to follow an incomplete mapping. Those are implementation choices.

Fork, Reorder, And Truncate Are Different Operations

A logical fork creates new identities with an exact prefix relation:

parent H, generation 2, committed [0,5)

fork at 5:
    parent H remains generation 2
    child A starts generation 0 with semantic prefix [0,5)
    child B starts generation 0 with semantic prefix [0,5)

The children must have distinct handle and snapshot identities even if their initial K/V values are equal. Once child A appends or truncates, its generation advances independently. M5.3 may map both children to shared physical blocks, but that sharing is not part of the semantic postcondition.

Reorder does not create a new token history. A pure reorder applies one permutation to a coupled row:

@dataclass(frozen=True)
class LiveSequenceRow:
    token_state: TokenState
    cache_handle: LogicalKVSnapshot
    next_position: int
    sampler_state: SamplerState

def reorder(rows, parent_index):
    assert sorted(parent_index) == list(range(len(rows)))
    return [rows[parent] for parent in parent_index]

Permuting token IDs without cache handles makes a beam attend to another beam’s history. Permuting cache handles without positions or sampler state is equally wrong. Beam expansion may select the same parent more than once; that is a selection plus logical fork, not a bijective reorder. M3.10 owns the beam score and token decision. M5.1 owns the requirement that selected semantic state and cache ownership remain aligned.

Truncate creates a successor snapshot at T_new <= T_committed and changes all layer lengths together:

before: S_g     every layer exposes [0, T)
after:  S_(g+1) every layer exposes [0, T_new)

not allowed:
    T_new > T
    output IDs rolled back but K/V suffix still visible
    layer-local crop published one layer at a time

The truncated suffix becomes logically unreachable. It is not necessarily physically free at that instant: a child, reader lease, kernel, or transfer may still own it. Physical reference closure belongs to M5.3/M5.5.

Cancellation, Retry, And Release Races

There are two cancellation scopes. cancel_txn(txn_id) aborts one speculative or failed append while preserving the request’s previous snapshot. cancel_request(handle_id) advances the request to CANCELLED, aborts its open transactions, and prevents new work. A late GPU completion may still arrive; completion is not permission to publish.

def cancel_request(live, expected):
    cancelled = derive_successor(live, phase="CANCELLED")
    if not live.compare_and_publish(expected, cancelled):
        return RETRY_OR_ALREADY_CHANGED
    abort_open_transactions(handle_id=live.handle_id)
    return cancelled

If append publication wins first, cancellation must act on the new generation. If cancellation wins first, the append’s base comparison fails. The runtime must not silently pretend both orders happened.

Retries need an identity stronger than “same request ID”:

operation fingerprint = hash(
    txn_id, idempotency_key, handle_id,
    base generation, base snapshot,
    cache ABI, append interval
)

The same committed fingerprint returns the original receipt without incrementing the generation again. Reusing a transaction ID with a changed key, interval, or base is a conflict. An old receipt can still be returned after the live handle has advanced or been released; it describes the historical committed operation, not the current readable snapshot.

Release publishes a terminal tombstone. Keeping generation, digest, length, and lineage in that tombstone is useful for rejecting late completions and auditing what was released. It does not keep the cache readable, and it does not claim that physical bytes have already returned to a free list.

Why Generation Fencing Prevents The ABA Mistake

Suppose delayed transaction X captured physical slot 17 while handle H was at generation 2. H later finishes and releases at generation 4. The allocator may eventually reuse slot 17 for unrelated handle J. A pointer-only check sees 17 before and after and can mistake the state for unchanged: the classic “A-B-A” shape.

M5.1 compares:

(handle_id H, generation 2, snapshot s_2)
    versus
(handle_id H, generation 4, tombstone s_4)

The delayed publication fails even if an observed physical address is again 17. The generation namespace must itself be non-reused over the maximum stale-work lifetime; a narrow wrapping counter would reintroduce the problem. The snapshot digest strengthens provenance but does not replace the atomic comparison. M5.3 adds physical block generations and leases for the lower-level allocation lifetime.

Worked Synthetic Lifecycle Trace

The article-local fixture starts from the Loop 2 snapshot at generation 1 and committed length 4:

1. Two append transactions capture generation 1. txn-win stages layer 0; an intervening read still sees [4,4]. After layer 1 completes, one publication creates generation 2 with [5,5].
2. Retrying txn-win returns its generation-2 receipt. The competing transaction finishes later but is rejected as stale.
3. Generation 2 forks into child A and child B. Reorder swaps their coupled cache/position/sampler rows. Child A truncates to length 3; child B cancels.
4. A partially staged child-B transaction is aborted. Its late layer completion and publication are both rejected.
5. A parent append is fully prepared, but finish and release advance the parent to generations 3 and 4. Slot 17 is observed under another owner; the delayed append still fails on generation, not address.

Every stored snapshot in the oracle satisfies equal per-layer lengths. The fixture is a deterministic synthetic contract test, not evidence that a particular serving runtime uses these Python objects or achieves a measured latency.

The oracle rejects one-field mutations for a stale base, unknown snapshot, append holes, incomplete or duplicate layers, non-bijective reorder, split sampler/cache state, fork beyond the committed prefix, truncate-as-growth, release before finish/cancellation, retry identity changes, late publication, and a released-handle read. These checks prove the declared abstract history; they do not prove a production implementation linearizable.

Cache Integration Failure Lab

0x02 Ragged Live-State And Byte Trajectory

M4.1 derives a rectangular logical K/V ledger from one immutable architecture, numerical-state, workload, and ownership manifest. Its estimator deliberately labels that result “logical K/V payload only; allocator and request lifecycle excluded.” M5.1 now adds those lifetimes without changing what the original number meant.

The order matters. Starting from a device-memory reading and dividing by a convenient cache formula can silently mix replicas, unused capacity, allocator state, and workspace. The safer path is:

cache ABI
    -> bytes per committed token
        -> one request at one time cut
            -> ragged set of cache-owning requests
                -> separately declared physical terms

Deriving One Token From The Cache ABI

At one cache-producing layer, one committed token contributes one K row and one V row. On the rank whose local cache shard owns H_kv,local K/V heads, their shapes are:

K_row[layer]: [H_kv,local, d_h]
V_row[layer]: [H_kv,local, d_h]

If keys use b_K bytes per scalar and values use b_V, one layer contributes:

B_token,layer
  = H_kv,local * d_h * b_K
  + H_kv,local * d_h * b_V
  = H_kv,local * d_h * (b_K + b_V)

Summing the row over L_cache cache-producing layers gives the local-shard coefficient:

B_token
  = L_cache * H_kv,local * d_h * (b_K + b_V)

The familiar leading 2 is therefore not a fundamental independent factor. It is the special case b_K = b_V = b, which reduces the expression to:

B_token = 2 * L_cache * H_kv,local * d_h * b

This derivation also explains why the manifest must name local ownership. If K/V heads are partitioned exactly once across cache shards, summing B_token across those shards recovers the global logical payload. If state is replicated, the extra copies are a second multiplicity; they must not be smuggled into H_kv,local. An architecture that stores latent state or quantized K/V needs a different cache ABI rather than a correction factor applied after this calculation.

The Loop 2 fixture makes the coefficient executable:

L_cache      = 2 layers
H_kv,local   = 2 heads
d_h          = 4 scalars/head
b_K = b_V    = 2 bytes/scalar

B_token = 2 * 2 * 4 * (2 + 2) = 64 bytes/token

This is a synthetic local-shard fixture, not a local GPU measurement.

From One Request To A Ragged State-Owning Set

Let T_i(tau) be request i’s committed cache length at lifecycle cut tau. It is not automatically the number of token IDs returned to the user. Section 0x03 shows why an emitted final token can remain one position ahead of committed K/V until a valid append or repair call processes it.

One request’s semantic K/V payload is therefore:

B_logical,i(tau) = T_i(tau) * B_token

Let A_KV(tau) be the set of request handles that still own committed cache state at that cut. The ragged aggregate is:

B_logical(tau)
  = sum over i in A_KV(tau) of T_i(tau) * B_token
  = B_token * sum over i in A_KV(tau) of T_i(tau)

This sum is the lifecycle extension of M4.1’s rectangle. Only when all N(tau) state-owning requests have equal length T may it be simplified to:

B_logical(tau) = N(tau) * T * B_token

Padding every request to the longest live length is an upper bound, not the semantic payload:

B_rectangle(tau)
  = N(tau) * max_i T_i(tau) * B_token

B_padding-phantom(tau)
  = B_rectangle(tau) - B_logical(tau)

For lengths [5, 2] in the 64-byte fixture, the real sum is 7 * 64 = 448 B. A two-row rectangle of width five is 2 * 5 * 64 = 640 B; it invents 192 B of token state. A physical static layout may genuinely reserve that space, but then those bytes belong to the reservation ledger, not the logical one.

Numeric Scale Check: GQA Versus MHA

Use a larger Llama-like GQA shape:

L_cache = 32
H_q     = 32
H_kv    = 8
d_h     = 128
b_K     = b_V = 2 bytes
T       = 4096 committed tokens

Then the global single-copy logical payload for one request is:

B_token
  = 32 * 8 * 128 * (2 + 2)
  = 131,072 bytes/token
  = 128 KiB/token

B_logical
  = 4096 * 131,072
  = 536,870,912 bytes
  = 512 MiB

If the same example used full MHA with H_kv = H_q = 32:

B_token = 32 * 32 * 128 * (2 + 2) = 512 KiB/token
B_logical at T=4096 = 2 GiB

The fourfold difference follows directly from the fourfold K/V-head count. M5.7 owns the architectural derivation and quality/throughput tradeoffs of MHA, MQA, and GQA. M5.1 only consumes the resulting H_kv field.

Stored Payload, Read Traffic, And Capacity Are Different Questions

The same tensor dimensions can appear in several serving ledgers without making the ledgers interchangeable:

stored semantic K/V at length T:
    T * B_token

new persistent semantic payload for one ordinary committed decode row:
    B_token

simplified logical historical K/V read payload for one decode frontier:
    approximately T * B_token

The third line is not an HBM profiler prediction. A real backend may tile, split, cache, quantize, page, fuse, or reuse data. M4.5 owns decode-kernel traffic and execution schedules. The purpose of the line here is narrower: adding one row is constant-size while reading the visible history grows with T.

Likewise, neither stored semantic payload nor logical reads answer “how much device memory is currently unavailable?” That requires more ledgers.

Logical, Materialized, Reserved, Physical, And Resident

For each cache-owning request, declare:

r_i(tau) = number of materialized copies, including the primary
C_i(tau) = reserved token capacity per copy, with C_i >= T_i

The logical state counts one semantic request history. Materialized payload counts every declared copy:

B_materialized(tau)
  = B_token * sum_i r_i(tau) * T_i(tau)

B_replication-overhead(tau)
  = B_materialized(tau) - B_logical(tau)

Reservation answers a different question: how much payload-equivalent capacity has been set aside under the declared layout?

B_reserved-capacity(tau)
  = B_token * sum_i r_i(tau) * C_i(tau)

B_reservation-slack(tau)
  = B_reserved-capacity(tau) - B_materialized(tau)

Under the synthetic Loop 5 allocation contract, physical allocated cache bytes are the reserved payload capacity plus an explicit metadata charge B_allocator(tau):

B_physical(tau)
  = B_reserved-capacity(tau) + B_allocator(tau)

This equality belongs to the teaching fixture, not to every allocator. Paged blocks, virtual reservation, internal and external fragmentation, alignment, block tables, pools, and delayed frees can produce a different physical relation. M5.3 owns those mechanisms. A production record should report its actual policy and observed allocator counters instead of pretending the synthetic 16 B/copy metadata charge is universal.

Residency is then a partition of physical state, not another addend:

B_physical(tau)
  = sum over declared tiers j of B_resident,j(tau)

For example, GPU 800 B + host 272 B = physical 1,072 B. Adding all three numbers would double-count the same physical objects. M5.5 owns how and when state moves between tiers; this section only requires the partition to close.

Finally, event-local workspace can coexist with persistent cache:

B_instantaneous-cache-surface(tau)
  = B_physical(tau) + B_temporary-workspace(tau)

That sum still excludes model weights, non-cache request metadata, sampler buffers, communication staging, and any other state not explicitly declared. M4.1’s rule survives intact: state classes need separate count, dtype, placement, replication, and lifetime records before they may be added at one time cut.

Event Deltas: Arrival Is Not Prefill, Finish Is Not Release

The byte trajectory follows lifecycle publication, not token IDs in a queue:

• Request arrives and waits: semantic K/V delta 0; physical cache may remain zero, although a separately declared policy could reserve capacity.
• Prefill publishes P rows: semantic delta +P * B_token; request storage and any declared copies become populated.
• Ordinary decode publishes one row: semantic delta +B_token; the physical path may consume existing reservation slack or request more capacity.
• Request waits: semantic delta 0; existing state remains owned.
• Request reaches a stop condition: semantic delta 0 by itself; finished state can remain retained until ownership closes.
• Request releases T_i rows: semantic delta -T_i * B_token; references, fences, copies, and allocator state must close before reclamation.

This distinction catches two common accounting bugs. Counting arrival as prefill invents state that has not been produced. Counting finish as immediate release hides a real interval in which output is complete but cache ownership or an in-flight consumer is not. Loop 6 will deepen generation fencing, cancellation, retry, fork, truncate, and release safety; Loop 5 freezes only the byte effects.

Worked Two-Request Trajectory

The executable fixture binds to the Loop 2 cache ABI and uses:

B_token = 64 B

request A:
    one materialized copy on GPU
    reserved capacity = 8 tokens

request B:
    two materialized copies: one GPU, one host
    reserved capacity = 4 tokens per copy

synthetic allocator metadata:
    16 B per live materialized copy

The full event trace follows. Each record lists committed lengths (A,B), then logical / materialized / reserved / physical / temporary bytes:

• tau=0, A arrives: (0,-); 0 / 0 / 0 / 0 / 0 B.
• tau=1, A prefill publishes 4: (4,-); 256 / 256 / 512 / 528 / 96 B.
• tau=2, B arrives: (4,0); 256 / 256 / 512 / 528 / 0 B.
• tau=3, B prefill publishes 2: (4,2); 384 / 512 / 1,024 / 1,072 / 96 B.
• tau=4, A decode publishes 1: (5,2); 448 / 576 / 1,024 / 1,072 / 32 B.
• tau=5, B decode publishes 1: (5,3); 512 / 704 / 1,024 / 1,072 / 32 B.
• tau=6, A finishes but retains state: (5,3); 512 / 704 / 1,024 / 1,072 / 0 B.
• tau=7, A releases: (-,3); 192 / 384 / 512 / 544 / 0 B.
• tau=8, B decode publishes 1: (-,4); 256 / 512 / 512 / 544 / 32 B.
• tau=9, B finishes but retains state: (-,4); 256 / 512 / 512 / 544 / 0 B.
• tau=10, B releases: (-,-); 0 / 0 / 0 / 0 / 0 B.

At tau=5, every number can be reconstructed:

B_logical
  = (5 + 3) * 64
  = 512 B

B_materialized
  = (1 * 5 + 2 * 3) * 64
  = 704 B

B_replication-overhead
  = 704 - 512
  = 192 B

B_reserved-capacity
  = (1 * 8 + 2 * 4) * 64
  = 1,024 B

B_reservation-slack
  = 1,024 - 704
  = 320 B

B_allocator
  = 3 live copies * 16 B/copy
  = 48 B

B_physical
  = 1,024 + 48
  = 1,072 B

resident partition
  = GPU 800 B + host 272 B
  = 1,072 B

B_instantaneous-cache-surface
  = 1,072 + 32 B decode workspace
  = 1,104 B

The peak persistent physical allocation is 1,072 B, but the peak cache-plus-workspace cut is 1,168 B during B’s prefill. One scalar called “KV memory” cannot preserve both facts.

Tensor And Code-Level Ledger Trace

The implementation does not begin with physical bytes. It binds the request manifest, derives one row from tensor shapes, and then folds live requests:

def per_token_payload(cache_schema):
    return (
        cache_schema.num_cache_layers
        * cache_schema.kv_heads_per_rank
        * cache_schema.head_dim
        * (
            cache_schema.key_dtype.bytes_per_scalar
            + cache_schema.value_dtype.bytes_per_scalar
        )
    )


def snapshot_ledger(requests, token_bytes, metadata_per_copy, workspace):
    logical = 0
    materialized = 0
    reserved = 0
    live_copies = 0
    resident_by_tier = zeros_for_declared_tiers()

    for request in requests:
        if not request.owns_cache_state:
            continue

        T = request.committed_cache_length
        C = request.reserved_token_capacity
        r = request.replica_count_including_primary
        assert 0 <= T <= C
        assert len(request.copy_residency_tiers) == r

        logical += T * token_bytes
        materialized += r * T * token_bytes
        reserved += r * C * token_bytes
        live_copies += r

        per_copy_physical = C * token_bytes + metadata_per_copy
        for tier in request.copy_residency_tiers:
            resident_by_tier[tier] += per_copy_physical

    replication_overhead = materialized - logical
    reservation_slack = reserved - materialized
    allocator_overhead = live_copies * metadata_per_copy
    physical = reserved + allocator_overhead

    assert sum(resident_by_tier.values()) == physical
    return {
        "logical": logical,
        "materialized": materialized,
        "replication_overhead": replication_overhead,
        "reserved": reserved,
        "reservation_slack": reservation_slack,
        "allocator_overhead": allocator_overhead,
        "physical": physical,
        "resident_by_tier": resident_by_tier,
        "instantaneous": physical + workspace,
    }

The actual oracle additionally validates strictly ordered events, arrival before prefill, one-row ordinary decode publication, capacity bounds, finish-before-release, declared residency tiers, zero terminal allocation, and every expected event ledger. Twenty one-field-invalid fixtures prove that a drifted cache ABI, digest, token coefficient, replica count, tier partition, reservation, event order, workspace, or expected byte value fails closed.

Sensitivity: Which Variable Moves Which Ledger?

Sensitivity should vary one declared input while holding the others fixed:

• Double L_cache, H_kv,local, or d_h: B_token and every payload-equivalent term double; this makes no latency claim.
• Halve both b_K and b_V: tensor payload terms halve if the runtime really stores that ABI; quality and kernel support still require M8.6 gates.
• Add one committed ordinary decode row: logical payload adds one B_token; materialized payload adds r_i * B_token, but existing slack can absorb it without another physical allocation.
• Add a replica: logical payload is unchanged while materialized, reservation, metadata, and tier-residency terms increase; no scheduling benefit is implied.
• Increase C_i only: logical payload is unchanged while reservation slack and possibly physical allocation increase; no useful token state was added.
• Change allocator layout: logical payload is unchanged while physical bytes and fragmentation can change; M5.3 owns the mechanism.
• Change workspace at one event: persistent ledgers are unchanged while only the instantaneous cut changes.
• Delay release: per-request bytes are unchanged, but the area under the byte-time curve grows; the ledger does not claim the delay is necessary.

The executable sensitivity test doubles each of L_cache, H_kv,local, and d_h independently and obtains 128 B/token; changing both K and V from two-byte to one-byte scalars obtains 32 B/token. Separate tests show that replication changes materialized but not logical payload, reservation changes physical but not logical payload, and workspace changes the instantaneous cut without changing persistent allocation.

This is the contract later articles consume. M5.2 may decide which request to run or admit, M5.3 may map capacity through blocks or virtual pages, and M5.5 may move residency. None may call those choices “logical K/V payload” without discarding the distinctions derived here.

0x03 Cache/No-Cache Handoff And Saved-Work Boundary

M5.1 does not need a second proof that cached and uncached generation can represent the same autoregressive model. M3.10 already owns the stepwise test:

LM_full(x[0:t])[-1]
    approximately equals
LM_cache(x[t], cache_for_x[0:t-1])[-1]

The comparison requires the same checkpoint, token prefix, positional policy, mask semantics, adapter state, and output position. “Approximately” is deliberate: different kernel schedules and reduced-precision accumulation can prevent bit identity. The cache contract is semantic parity within a declared tolerance, not byte-for-byte equality between every backend.

This section imports that result and asks a different question:

Once parity is established, which historical operations disappear, which operations remain, and how far may output-token state advance ahead of committed K/V state?

Deriving The Saved Historical Row Executions

Let:

P = prompt-token count
G = number of output tokens selected and returned
g = zero-based output-decision index

An uncached loop that emits G outputs makes G model calls over growing prefix widths:

call g processes P + g token rows
prefix widths: P, P+1, ..., P+G-1

Therefore each decoder layer receives this many token rows across the request:

N_rows,no-cache
  = sum(g=0..G-1) (P + g)
  = G*P + G*(G-1)/2

The ordinary cached path still makes G decision-producing model calls. It does not make future tokens parallel and it does not remove the first prefill. It changes the width of calls after prefill:

prefill rows:       P
decode calls:       G - 1
rows per decode:    1

N_rows,cache = P + G - 1

The subtraction by one is not a cosmetic convention. Prefill processes the prompt and selects the first output token. Only the first G-1 selected outputs are later fed back to produce outputs two through G. If generation stops immediately after selecting the final output, that token has not been processed as model input.

Subtracting the two paths gives the historical row executions avoided at every layer:

N_rows,saved
  = N_rows,no-cache - N_rows,cache
  = (G-1)*P + (G-1)*(G-2)/2

For P=2048 and G=128:

no-cache rows per layer:  270,272
cached rows per layer:      2,175
saved row executions:     268,097
row-count reduction:       124.26x

This is a semantic row ledger, not a latency claim. The calls have different matrix shapes, kernels, occupancy, memory traffic, and launch costs. M4.1 owns the operation/FLOP conventions, while M4.3-M4.6 own attention kernels and hardware schedules.

Deriving The Saved Old-Query Attention Work

The row count alone hides another repeated object. A dense causal call of width S has this many legal query-key score pairs per layer and query head:

N_pairs(S) = S*(S+1)/2

The uncached loop recomputes that causal triangle for every growing prefix:

N_pairs,no-cache
  = sum(g=0..G-1) (P+g)*(P+g+1)/2

The cached path computes the prompt triangle once. Each later decode call has one current query and a key history whose width grows by one:

N_pairs,cache
  = P*(P+1)/2
  + sum(j=1..G-1) (P+j)

For the same P=2048, G=128 fixture:

no-cache causal pairs per layer/query head:  285,562,176
cached causal pairs per layer/query head:      2,366,400
old-query pairs not re-executed:              283,195,776
pair-count reduction:                             120.67x

The cached count still grows with context. Each new query must score against visible historical keys and combine historical values. KV cache removes old query-row recomputation; it does not make historical K/V reads disappear. Multiplying these semantic pairs by head counts or operation conventions to obtain FLOPs belongs to M4.1, and translating them into measured memory transactions belongs to the selected backend and profiler.

Source-local visual from A20 - KV Cache, used here for critical commentary. Red annotations identify repeated historical rows; blue annotations identify the new current-row work. The original labels are partly Chinese, so the operation boundary is restated explicitly below.

Saved, Retained, And Newly Introduced Work

Under the ordinary full-K/V cache ABI, the avoided work is broader than only the old W_K and W_V multiplications. Because causality guarantees that later tokens cannot alter earlier hidden rows, an uncached full-prefix call would repeat these old rows at every layer:

saved historical work:
    historical embedding and layer-row execution
    historical normalization and Q/K/V projection rows
    historical query attention rows, softmax rows, and P*V combinations
    historical output-projection, normalization, MLP, and residual rows
    historical vocabulary rows if the implementation materializes them

The cache path still performs:

unavoidable current-frontier work:
    current token embedding and complete one-row layer stack
    current Q/K/V projections and positional transform
    current query scores against all visible historical/current keys
    current probability/value combination
    current output projection, normalization, MLP, residual, and logits
    token selection and stop-policy evaluation

cache-specific work:
    read historical K/V
    build the attention-visible K/V view
    write and publish the new per-layer K/V rows
    preserve the cache while the request waits between calls

Caching old hidden rows instead would still require deriving or retaining the layer-specific K/V interface. Caching old Q rows does not help an ordinary next token because old queries never need to attend to a future token. Caching the old score or probability triangle is also the wrong reusable object: the next decision adds a new query row, while its required historical K/V already form the stable causal sufficient state for ordinary dense attention. Alternative architectures can declare a different recurrent or latent state, but that is a different cache ABI rather than an exception hidden inside this one.

Output Progress And Cache Progress Are Different Counters

Let E be emitted output count and E_processed the number of generated tokens that have subsequently passed through a complete cache-producing model call. For an ordinary full cache:

logical token count       = P + E
committed cache length    = P + E_processed

while generation continues normally:
    E_processed = E - 1

after stopping immediately after the final selection:
    committed cache length = P + E - 1

A tiny P=4, G=3 trace makes the frontier visible:

prefill [x0 x1 x2 x3]
    selects/emits y0
    cache commits [x0 x1 x2 x3]                 length 4
    unprocessed output frontier: [y0]

decode input [y0]
    selects/emits y1
    cache commits y0                            length 5
    unprocessed output frontier: [y1]

decode input [y1], then stop
    selects/emits y2
    cache commits y1                            length 6
    returned logical tokens: [x0 x1 x2 x3 y0 y1 y2], length 7
    unprocessed output frontier: [y2]

Nothing is wrong with a finished request ending in that state. If the cache is released, no future model call can depend on y2’s absent row. But an API that promises a reusable cache containing every returned token must close the gap explicitly:

def append_output_frontier_for_reuse(model, state):
    assert state.unprocessed_output_frontier == [state.output_tokens[-1]]

    repair = model(
        input_ids=state.unprocessed_output_frontier,
        past_key_values=state.committed_cache,
        attention_mask=extend_mask_for_frontier(state),
        position_ids=[state.committed_cache_length],
        cache_position=[state.committed_cache_length],
        use_cache=True,
    )

    publish_cache(repair.past_key_values)
    discard(repair.logits)  # not another returned output-token decision
    state.unprocessed_output_frontier = []

That append/repair call is real model work. It processes the final token through every cache-producing layer, appends one K/V row, and returns logits that the snapshot-closing path deliberately ignores. Relabeling the previous call’s logits as K/V, advancing only a length counter, or copying an unrelated row cannot create the missing state.

Boundary Consumed By Speculative Correction

M7 changes how many candidate positions a target call verifies, but it does not change the token-to-cache identity invariant:

accepted draft token:
    may commit the matching target-verification K/V row

rejected draft token:
    its branch row must not enter the committed prefix

correction or all-accepted bonus token:
    may be emitted from target logits before its own K/V row exists
    must pass through a valid target append path before later decode uses it

For a synthetic base cache length of 100, two accepted draft tokens, and one correction token:

output progress after verification:             +3 tokens
branch K/V rows safe to commit:                  +2 rows
committed cache length after verification:      102
unprocessed output frontier:                    [correction]
committed cache length after correction repair: 103

M7.1 owns the draft distribution, target verification, acceptance probability, and correction distribution. M5.1 exports only this state rule: an emitted token may be one frontier position ahead of committed K/V, but future execution must never condition on a row produced for a different token.

Executable Saved-Work Contract

The deterministic Loop 4 fixture records the long work ledger, the small ordinary stop trace, the optional final-token repair, and the speculative correction handoff:

docs/numinens-evidence/m5-1-serving-state/
    saved_work_fixture.json
    invalid_saved_work_fixtures.json
    cache_saved_work.py
    test_cache_saved_work.py

Seventeen invalid fixtures reject row-sum drift, an invented first-prefill saving, disappearance of historical K/V reads, final-token cache over-commitment, repair calls that emit another output, discarded-logit drift, and correction rows relabeled from rejected branches. This is a synthetic contract oracle. It does not run a model, prove numerical parity, predict latency, or replace a backend profiler.

0x04 Storage Is Linear, Dense Attention Work Is Not The Same Statement

Another common mistake is to mix cache storage with dense attention compute.

For a standard dense decoder layer:

KV cache storage:
    linear in T

one decode step attention read:
    linear in T

full prefill attention score matrix:
    roughly quadratic in S for dense attention

Those are different claims.

The KV cache is not exponential. It grows linearly with:

active tokens
layers
KV heads
head dimension
bytes per scalar
number of active sequences

The serving pain comes from the coefficient and from concurrency. A single linear object can still be huge if it is multiplied by long context, many layers, and many simultaneous users.

Dense prefill attention has a different shape. It can form or logically compute relations between many pairs of prompt tokens. M4 handled how FlashAttention avoids materializing the full score and probability matrices in HBM. M5 cares about what remains after that call: the per-token K/V state that survives into decode.

0x05 Phase And Service Interface

The logical cache object now has enough structure to expose a service interface. That does not mean M5.1 should choose a schedule. It means the object can report which operation is legal, which rows already exist, which token is waiting at the output/cache frontier, and which bytes remain owned.

This distinction is the boundary between M5.1 and M5.2:

M5.1 exports facts about one versioned request/cache object.
M5.2 chooses among many such objects.

ORCA externally establishes why that second decision exists: generation is a multi-iteration workload, so a scheduler can reconsider the active set between iterations. SARATHI and Sarathi-Serve externally separate prompt prefill from one-token-at-a-time decode and show one possible scheduling response, chunked prefill. DistServe externally treats TTFT and TPOT as separate latency requirements. None of those sources implies that phase state and scheduling policy are the same object.

Phase Is A Projection, Not A Second Cache Truth

Loop 6 defined durable request snapshots and transaction states. A service phase is a projection over those states plus work currently in flight. It does not replace the versioned cache object:

• WAITING_PREFILL: the request exists and prompt rows remain unprocessed. Arrival alone creates no K/V. Prefill rows are available.
• PREFILL_IN_FLIGHT: a selected prompt interval executes against one base snapshot. Its rows are pending rather than published; the object waits for success, failure, or cancellation.
• OUTPUT_READY: a complete prefill or decode publication produced target logits. The input rows are committed, so one output may be selected and emitted.
• DECODE_FRONTIER_READY: one emitted token has not passed through its cache-producing call. Committed K/V is one output frontier behind; ordinary decode may process that exact token.
• FINISHED_RETAINED: stop is final but ownership has not closed. Committed rows remain live and the last emitted token may remain unprocessed; the owner can retain, repair for reuse, or release.
• RELEASED: generation-fenced ownership is closed. This handle no longer owns logical payload, and no later read or mutation is legal.

The in-flight label is deliberately not a newly readable snapshot. During an append, readers still see the old all-layer snapshot from Loop 6. Only the publication transition changes committed state.

Prefill also needs one subtle qualification. Semantically, a new prompt’s prefill is complete when all prompt rows have been processed and the first output distribution is available. An execution engine may split that work:

prompt length P = P_1 + P_2 + ... + P_m

after chunk c:
    prompt rows processed = sum(j=1..c) P_j
    committed K/V rows    = sum(j=1..c) P_j

first output becomes ready only after:
    sum(j=1..m) P_j = P

The logical interface can therefore publish a partial prefill interval and return to WAITING_PREFILL. M5.2 owns whether a prompt is chunked, how large each chunk is, which decode work runs beside it, and whether that choice is fair. M5.1 owns only the state transition caused by a chunk that was selected elsewhere.

Deriving The Service-Facing State View

A scheduler cannot safely act on the label decode alone. Two decode-ready requests can have different snapshot generations, cache lengths, residency, remaining prompt work, and output frontiers. The minimum interface is a typed projection of facts already established in Loops 2-6:

@dataclass(frozen=True)
class ServiceStateView:
    # Versioned identity: stale work must not target a recycled handle.
    request_id: str
    handle_generation: int
    snapshot_version: int

    # Semantic availability, not a scheduling decision.
    phase: Phase
    eligible_work: Literal[
        "PREFILL_ROWS_AVAILABLE",
        "DECODE_FRONTIER_AVAILABLE",
        "NONE",
    ]

    # Three counters that must not alias.
    prompt_rows_remaining: int
    emitted_output_tokens: int
    processed_generated_tokens: int
    committed_tokens: int
    output_frontier_count: int

    # Logical resource facts. Physical mechanisms remain later owners.
    logical_payload_bytes: int
    next_ordinary_append_bytes: int
    terminal_retained_logical_payload_bytes: int

    # Endpoint clocks used to derive service metrics.
    arrival_time: Timestamp
    output_times: tuple[Timestamp, ...]
    finish_time: Timestamp | None
    release_time: Timestamp | None

For the ordinary cache ABI, the resource fields follow directly from the Loop 2 coefficient B_token:

B_logical,i(t) = T_committed,i(t) * B_token

B_next-append,i(t) =
    B_token, if one ordinary emitted frontier token is decode-eligible
    0,       otherwise

B_terminal-retained,i(t) =
    B_logical,i(t), if state is FINISHED_RETAINED
    0,              otherwise

The final line reports bytes still owned after output completion. It is not a reclaimability predicate: FINISHED_RETAINED still has a semantic owner. Release can close that ownership; only then can reference and fence conditions make physical storage reclaimable. M5.3 owns physical blocks, sharing, copy-on-write, references, and reclamation. M5.5 owns residency and movement.

The view also does not contain these fields:

admit
priority
batch_id
prefill_chunk_size
preempt
victim_request
run_next

Those are decisions. Adding them to the request object would hide policy inside state and make it impossible to compare FCFS, SLO-aware, prefix-aware, or decode-prioritizing schedulers against the same factual input.

Facts constrain a decision without silently becoming the decision.

Deriving User Metrics From Endpoint Clocks

The word latency is too coarse. Let request i arrive at a_i, emit G_i output tokens at times

y_i,0, y_i,1, ..., y_i,G_i-1

finish at f_i, and release its logical cache ownership at r_i. The article uses the following explicit convention:

TTFT_i = y_i,0 - a_i

TBT_i,j = y_i,j - y_i,j-1              for j = 1..G_i-1

TPOT_i = (y_i,G_i-1 - y_i,0) / (G_i-1) for G_i > 1

E2E_i = f_i - a_i

release_lag_i = r_i - f_i

TPOT averages the G_i - 1 intervals between emitted output tokens. It is undefined for a one-output request. Replacing the denominator with max(G_i - 1, 1) silently invents a zero TPOT for a request that has no inter-token interval. A reporting system may choose another named convention, but it must publish that denominator before values can be compared.

TBT preserves the individual gaps that TPOT averages away:

TPOT_i = mean_j(TBT_i,j), for G_i > 1

Two requests can have the same TPOT while one contains a large streaming stall. Tail TBT or a maximum-TBT gate can therefore reject a request whose average appears acceptable.

TTFT is also an endpoint metric, not a causal attribution. It can include waiting, routing, prefix lookup, prefill execution, first-token sampling, and stream delivery. The exact identity is simply first output - arrival. Loop 8 will define event records that partition this interval. M5.2 owns the queueing and selection policies that create much of it, while M5.6 owns prefill/decode placement and transfer.

Throughput, SLO Attainment, And Goodput Are Different Ledgers

DistServe uses goodput as a capacity objective: the maximum request rate a system can serve while meeting configured TTFT and TPOT constraints for the required fraction of requests. That source-level definition should not be silently replaced by one universal per-window equation.

For the deterministic fixture below, M5.1 uses a narrower operationalization so the accounting is executable. For an observation window of width W, let

q_i = 1[
    request i completed successfully
    and TTFT_i <= tau_TTFT
    and every observed TBT_i,j <= tau_TBT
]

For a one-output request the set of TBT intervals is empty, so the TBT clause is vacuously satisfied; TTFT and successful completion still apply. Then:

request throughput = completed requests / W

request goodput = sum_i q_i / W

goodput fraction = sum_i q_i / completed requests

SLO-valid token goodput = sum_i (G_i * q_i) / W

Request goodput and token goodput answer different questions. A workload with one very long successful output can have modest request goodput and high token goodput. Neither should be called raw throughput without its unit and SLO predicate.

Worked Phase, Frontier, And Metric Trace

Use the same synthetic cache ABI as Loops 2 and 5:

B_token = 64 B per committed token on the local cache shard
observation window W = 100 ms
TTFT limit = 15 ms
maximum TBT limit = 10 ms

Request A has four prompt tokens and emits four outputs. Request B has two prompt tokens and emits one output:

Read every record as:

time | request | operation
     | emitted outputs / processed generated tokens | committed K/V
     | service consequence

• t=0, A arrives: 0 / 0 generated counters and 0 committed K/V. Prompt work is available; arrival creates no K/V.
• t=2, A prefill begins: counters remain 0 / 0 and K/V remains 0. Pending work is not readable state.
• t=5, B arrives: B independently has 0 / 0 generated counters and no committed K/V.
• t=8, B prefill begins: B has in-flight work but still no published state.
• t=12, A prefill publishes four rows: counters are 0 / 0, committed K/V is 4, and the first output becomes ready.
• t=14, A emits a0: counters are 1 / 0, committed K/V remains 4, frontier is [a0], and the next ordinary append is 64 B.
• t=15, B prefill publishes two rows: counters are 0 / 0, committed K/V is 2, and B’s first output becomes ready.
• t=18, B emits b0 and finishes: counters are 1 / 0, committed K/V remains 2, B retains 128 B, and TPOT is undefined.
• t=19, B releases: committed K/V becomes 0; B’s logical ownership closes.
• t=20, A processes and publishes a0: counters become 1 / 1, committed K/V becomes 5, and the matching row is now committed.
• t=21, A emits a1: counters become 2 / 1, committed K/V stays 5, and [a1] is the new frontier.
• t=29, A processes and publishes a1: counters become 2 / 2 and committed K/V becomes 6.
• t=30, A emits a2: counters become 3 / 2, committed K/V stays 6, and [a2] is the new frontier.
• t=41, A processes and publishes a2: counters become 3 / 3 and committed K/V becomes 7.
• t=42, A emits a3 and finishes: counters become 4 / 3, committed K/V remains 7, and the final output is an unprocessed frontier.
• t=50, A releases: committed K/V becomes 0. No repair is needed because no later call reuses the snapshot.

The metric derivation is now mechanical:

request A:
    TTFT = 14 - 0 = 14 ms
    TBT  = [21-14, 30-21, 42-30] = [7, 9, 12] ms
    TPOT = (42 - 14) / 3 = 9.333... ms
    E2E  = 42 - 0 = 42 ms
    release lag = 50 - 42 = 8 ms
    TTFT passes, max-TBT fails -> not SLO-valid

request B:
    TTFT = 18 - 5 = 13 ms
    TBT  = []
    TPOT = undefined
    E2E  = 18 - 5 = 13 ms
    release lag = 19 - 18 = 1 ms
    TTFT passes; no TBT interval exists -> SLO-valid

100 ms aggregate:
    throughput       = 2 / 0.1 = 20 requests/s
    request goodput  = 1 / 0.1 = 10 requests/s
    goodput fraction = 1 / 2   = 0.5
    token goodput    = 1 / 0.1 = 10 SLO-valid tokens/s

These rates are arithmetic over a synthetic 100 ms window. They are not a capacity benchmark and should not be compared with a paper’s measured goodput.

The same timeline exposes a resource-time quantity that instantaneous peak bytes miss. Let the logical K/V byte-time be:

KV_byte_time = integral over t of sum_i B_logical,i(t) dt

For A:

4*64*(20-12)
+ 5*64*(29-20)
+ 6*64*(41-29)
+ 7*64*(50-41)
= 13,568 byte-ms

For B:

2*64*(19-15) = 512 byte-ms

Therefore:

aggregate logical K/V byte-time
    = 14,080 byte-ms
    = 14.08 byte-seconds

This is the area under the logical payload curve, not physical GPU byte-time. Copies, reservations, metadata, pages, and residency would require the separate Loop 5 ledgers and later physical owners.

Executable Phase And Interface Contract

The deterministic Loop 7 evidence records the event history, service-view fields, metric conventions, output/cache frontier, release lag, goodput, and logical K/V byte-time:

docs/numinens-evidence/m5-1-serving-state/
    phase_service_fixture.json
    invalid_phase_service_fixtures.json
    phase_service_interface.py
    test_phase_service_interface.py

Twenty-three one-field mutations reject cache-ABI drift, policy fields hidden inside the state interface, undefined TPOT relabeled as zero, work before arrival, publication before prefill begins, prompt overrun, output before logits are ready, non-contiguous output indexes, early finish, wrong frontier-token identity, release of a live request, and expected-metric drift. The oracle also checks a partial-prefill publication. It is a synthetic contract test, not a serving runtime, allocator, scheduler, or latency measurement.

0x06 Exact Handoff To M5.2

M5.1 no longer needs to reproduce the static-batching timeline, ORCA scheduler loop, admission alternatives, or chunk-size tradeoff. M5.2 owns those derivations in depth. Its input from this article is:

versioned request/cache identity
legal work kind and phase-produced state
prompt rows remaining and one-token output frontier
committed, reserved, resident, terminal-retained, and reclaimability facts
current logical bytes and declared growth bounds
arrival, output, finish, and release clocks
TTFT and TBT/TPOT SLO declarations

M5.2 then owns:

which request or token interval runs next
continuous-batch membership
prefill chunk boundaries
admission and reservation policy
fairness, priority, and starvation control
preemption action and victim selection

The separation is practical. The same ServiceStateView can feed a static request scheduler, ORCA-style iteration scheduling, decode-prioritizing continuous batching, or a chunked-prefill policy. Changing policy should not change what committed, frontier, finished, or released mean.

0x07 Runtime Integration And Observability

The abstract object is now detailed enough to meet a production runtime. This is where many explanations make one of two opposite mistakes:

mistake 1:
    present an abstract state machine with no place to instrument real code

mistake 2:
    rename one runtime's mutable fields as the universal cache semantics

M5.1 needs a stricter integration contract. It should show where a current runtime exposes corresponding state, where that correspondence is only approximate, which semantic events are absent, and what telemetry can and cannot prove.

The central rule is:

runtime state causes telemetry
telemetry does not become runtime state

An event record can be delayed, duplicated, reordered, sampled, or dropped. A metric can aggregate thousands of requests into one bucket. A trace can be disabled. None of those changes the legal transition of the request/cache object. Conversely, a perfectly shaped log line emitted before publication does not make pending K/V readable.

Pinning One Runtime Before Mapping It

The implementation comparison in this section is pinned to vLLM v0.23.0, commit 0fc695fc6d1d82e9a5ac6835ac8e4e1c83703665, checked on 2026-07-27. It is not a claim about every vLLM release, every serving engine, or even every cache backend in that release.

The exact source digests are renderer-safe in one scrollable ledger:

vllm/v1/request.py
    4b97f41fc0ea474149f78d52a44ce89c16a9921c0641c48ad2573bdb8b7f71a4
vllm/v1/engine/__init__.py
    6eb1175dd76d5099dd4b6297d12c682f13ca42e297968983bc433894c1b82245
vllm/v1/core/sched/scheduler.py
    9ae1491f57a6294e32416b93ef2ea68107fa0d65c3a4d7f745f1261012f735e9
vllm/v1/core/kv_cache_manager.py
    a69aa08062714b40b149648bbec031a42d22e422fbf9543cd10fbd12d2496edb
vllm/v1/metrics/loggers.py
    ce2a0535d2b2b69e4368f34b692dac081be764e8d0eadab4f791b1afdcf43df4
vllm/tracing/utils.py
    2463b04ccfa8fc97d5cdb423accf50bb601c422322e72e241ccc0b1b2610305d
docs/design/metrics.md
    5d5b8bab2b6d1e36a9ade2ed5974011dddbf7695dee4a59b9113870fd8339d72

The permitted source claims remain separate:

• Request carries identity, status, output IDs, num_computed_tokens, and engine events.
• EngineCoreEvent exposes QUEUED, SCHEDULED, and PREEMPTED with process-local monotonic timestamps.
• The scheduler separates scheduling bookkeeping, output reconciliation, finish, and delayed cache free.
• The KV cache manager owns physical slot allocation, free, and block-pool events.
• The metrics implementation exports bounded engine/model dimensions and aggregate request/cache/latency observations.
• The tracing utilities propagate W3C context and define request/latency attributes.
• The metrics design requires one process-local monotonic clock for intervals and describes some block-residency observations as sampled.

The local evidence file runtime_source_pins.json records the release, full commit, URLs, file digests, required source markers, and permitted claims. A local test re-hashes the downloaded release archive when that archive is available. The article still does not run vLLM or claim a measured overhead.

Mapping Fields Without Flattening Semantics

The useful comparison is not “M5.1 equals vLLM.” It is a set of qualified correspondences:

• Direct field — request identity: Request.request_id names the request; M5.1 additionally requires a handle generation to reject recycled-identity ABA races.
• Near correspondence — lifecycle: Request.status includes waiting, running, preempted, remote-KV, streaming, and finish states. M5.1 phases project different semantic questions and are not aliases for that enum.
• Direct field — output progress: output_token_ids records emitted token state; its count must not alias computed/cache progress.
• Near correspondence — processed progress: num_computed_tokens is advanced around scheduling and can later be reduced for speculative rejection. It is not by itself an all-layer publication receipt.
• Direct observation — queue/schedule/preempt clocks: EngineCoreEvent provides those events, but the pinned vocabulary lacks allocate, append-start, append-publish, fork, truncate, release, and append failure.
• Foreign owner — physical cache: KVCacheManager and request-to-block state expose allocator facts. M5.3 owns their mechanism.
• Partial observation — residency: connector and cache metrics can expose selected residency facts. M5.5 owns tiers, transfer, restore, and fallback.
• Absent stronger contract — versioned snapshot: the pinned runtime has no exact generation-plus-snapshot field. M5.1 proposes that discipline; it does not misdescribe it as an existing vLLM API.

The num_computed_tokens distinction is especially important. In the pinned scheduler, _update_after_schedule advances that field after work is scheduled so the next scheduling pass has progress bookkeeping. Later, update_from_output can subtract rejected speculative tokens. This is valid runtime machinery, but it means the field’s update site is not automatically the same thing as M5.1’s all-layer publication point.

Likewise, finishing and freeing are not one instruction. _free_request can delay block release while connector cleanup remains in flight. That source behavior supports the conceptual separation already derived here:

finished output
    is not the same fact as
closed logical ownership
    is not the same fact as
physical blocks observed free

The mapping is therefore useful in both directions. The runtime shows which real code paths need correlation; the abstract contract shows which fields cannot safely be inferred from one mutable counter.

Five State Planes, One Causal History

An observability design becomes ambiguous if every field is placed in one record and called “cache state.” Separate five planes:

1. Semantic plane: request identity and generation, snapshot version, committed token count, output frontier, phase, finish, and release.
2. Operation plane: operation identity, base snapshot, append interval, pending state, retry lineage, success, failure, or cancellation.
3. Allocator plane: physical blocks or slots, references, free-list state, and allocator generations. M5.3 owns these mechanisms.
4. Residency plane: GPU/CPU/remote tiers, resident byte partitions, transfers, and restoration. M5.5 owns these mechanisms.
5. Telemetry plane: source identity, event schema, clocks, sequence, delivery, sampling, and export status.

The last three can be correlated with the first two, but they do not collapse into them. For example:

semantic:
    committed_tokens = 5

allocator observation:
    allocated_slots = 8
    used_slots = 5

residency observation:
    resident_payload_bytes = 5 * B_token

Those values are compatible, not equal. The allocator may have tail slack, shared blocks, metadata, delayed frees, or reservations. Residency may include copies or omit non-resident logical payload. A drift monitor needs an explicit conversion model before it can compare planes.

Deriving A Mutation Event From The Transition

Let the authoritative semantic object be S_n. A legal operation op_n produces either a new published state or no semantic mutation:

success:
    S_(n+1) = delta(S_n, op_n)

failure before publication:
    S_(n+1) = S_n

An audit event is an observation of that outcome:

e_n = observe(S_n, op_n, outcome_n, S_(n+1))

It is safe to use the event for reconciliation only if all of these hold:

publication ordering:
    success event is emitted after S_(n+1) becomes the visible state

completeness:
    every audited mutation has an outcome record

idempotent identity:
    duplicate event_id means byte-equivalent event content

local order:
    source-local sequence has no unexplained gap or regression

causal correlation:
    cross-component observations name the semantic event or operation they follow

schema and provenance:
    producer revision and event schema are known

Without those assumptions, this implication is false:

replayed telemetry state == authoritative runtime state

The minimum event envelope in this article is:

@dataclass(frozen=True)
class RuntimeMutationEvent:
    # Stable delivery and correlation identity.
    event_id: str
    trace_id: str
    source_instance: str
    source_component: str
    local_sequence: int

    # Event time is source-local; observation time is for correlation.
    monotonic_ns: int
    observed_wall_time: str

    # Versioned semantic target.
    request_id: str
    handle_generation: int
    operation_id: str | None
    event_type: str

    # Event-specific base/result payload follows a versioned schema.
    base_snapshot: int | None
    result_snapshot: int | None
    outcome: str | None

Why keep both clocks? The pinned vLLM event source and metrics design use time.monotonic() for intervals and explicitly warn that monotonic timestamps from different processes cannot be compared. Wall time can help correlate records across machines, but NTP adjustment and clock skew make it unsafe as a mutation order. Therefore:

duration within one source:
    local monotonic timestamps

gap detection within one source:
    local_sequence

causality across sources:
    trace/span context + operation_id + caused_by_event_id

human and multi-system correlation:
    observed wall time, with skew caveat

OpenTelemetry’s trace API supports operation spans and timestamped events, and its log data model separates event time from observed time. That makes it a reasonable transport for correlations. It does not require every span to be recorded or exported, so a sampled OpenTelemetry trace is not a complete mutation journal.

Event Contracts At The Semantic Boundary

The event type determines which fields are legal:

• request_allocate: establish a generation and empty snapshot; do not imply prompt K/V already exists.
• append_start: target one generation, base snapshot, and interval [T,T+delta); pending rows are not readable.
• append_publish: pair with one start, advance the result snapshot, and commit T+delta; do not imply physical free elsewhere.
• failure: terminate one pending operation while leaving its base snapshot visible; carry no result snapshot or committed-row delta.
• output_emit: record an output index and token from one semantic snapshot; do not imply that token’s own K/V has already been appended.
• fork: establish child generation/snapshot from the logical prefix; make no eager-copy or shared-block claim.
• truncate: shorten one live handle and advance its snapshot; leave physical block policy to M5.3.
• finish: make output terminal while ownership may remain; do not claim release or reclamation.
• release: close logical ownership and fence the generation; immediate device free is a separate observation.

Allocator and residency observations use different event types and source components. They can say “two blocks observed allocated” or “320 payload bytes observed on GPU,” but they must carry a causal reference to the semantic event they follow. Putting allocated_blocks directly into append_publish would silently make one allocator representation part of the logical cache ABI.

The instrumentation order should follow the transition, not precede it:

def publish_append(txn, completed_layers):
    # Authoritative compare-and-publish from Loop 6.
    receipt = cache.publish_if_base_matches(txn, completed_layers)

    # Only now can a success event name the visible result snapshot.
    audit.emit(append_publish_event(receipt))
    traces.add_event("cache.append.publish", receipt.trace_attributes())
    metrics.append_outcomes_total.labels(outcome="success").inc()
    return receipt


def fail_append(txn, error):
    cache.abort_pending(txn)       # leaves the base snapshot readable
    audit.emit(failure_event(txn, error, result_snapshot=None))
    metrics.append_outcomes_total.labels(outcome="failure").inc()

If append_publish is logged before the compare-and-publish succeeds, a crash can leave a durable success event for a state that never became visible. If the event is emitted after publication but lost, the cache is correct while the audit stream is incomplete. Those are different failure classes and need different alerts.

Worked Failure, Retry, Fork, And Release Trace

The Loop 8 fixture starts request R at handle generation 7:

seq 0  request_allocate  -> generation 7, snapshot 0, T=0
seq 1  append_start      -> prefill-1, base s0, interval [0,4)
seq 2  append_publish    -> snapshot 1, T=4
seq 3  output_emit       -> output index 0; cache remains T=4
seq 4  append_start      -> decode-1, base s1, interval [4,5)
seq 5  failure           -> no result snapshot; visible state stays s1,T=4
seq 6  append_start      -> decode-2 retries from the same base
seq 7  append_publish    -> snapshot 2, T=5
seq 8  fork              -> child R.branch starts at child s0,T=5
seq 9  child truncate    -> child s1,T=3; parent stays s2,T=5
seq 10 parent finish     -> output terminal; T=5 retained
seq 11 parent release    -> generation 8, snapshot 3, T=0
seq 12 child finish      -> child s1,T=3 retained
seq 13 child release     -> child generation 1, snapshot 2, T=0

Three observations from other source-local sequences are correlated to the parent’s successful publications and release:

allocator observer:
    after prefill publish  -> 2 blocks, 8 slots, 4 used
    after decode publish   -> 2 blocks, 8 slots, 5 used
    after release          -> 0 blocks, 0 slots, 0 used

residency observer:
    after prefill publish  -> 256 payload bytes on GPU
    after decode publish   -> 320 payload bytes on GPU
    after release          -> 0 resident payload bytes

The 64 B/token values come from the synthetic Loop 2 cache ABI, so the residency arithmetic is exact for the fixture. The block counts do not derive from that coefficient. They are separate synthetic allocator facts and do not specify block size, sharing, or physical reclamation in a real runtime.

The local sequence is global to each producer instance, not to a request. This matters when many requests interleave. Requiring every request’s events to have contiguous producer sequence numbers would incorrectly flag normal interleaving. The audit consumer groups by source for gap detection and by (request_id, handle_generation) for semantic replay.

Audit Stream, Traces, Metrics, And Logs

One observability surface cannot satisfy every cost and correctness goal:

• Mutation audit stream: stable request, generation, operation, and event IDs support transition reconciliation, retry deduplication, and gap detection. This complete surface is operationally heavier than aggregate metrics.
• Distributed traces: trace/span/request attributes support causal and latency diagnosis. Sampling means absence is not proof, and spans are not the state store.
• Prometheus metrics: bounded model, engine, event-type, and outcome labels support rates, gauges, histograms, alerts, SLOs, and capacity trends. Aggregation discards per-request identity and order.
• Structured logs: request and operation context supports diagnosis and detailed errors where policy permits. Retention, rate limiting, parsing, and delivery do not constitute an audit guarantee.

Prometheus explicitly warns that every unique label-value combination creates a new time series. request_id, trace_id, operation_id, and event_id are therefore valid audit/trace correlation fields but invalid metric labels. A bounded metric family can instead use:

cache_append_outcomes_total{
    model_name,
    engine,
    event_type="append",
    outcome="success|failure|conflict|cancelled"
}

The denominator must exist beside the error count. A failure counter without attempts cannot produce a failure ratio. Gauges such as running requests or KV usage describe snapshots and may move down; counters such as append outcomes only increase until process restart; latency distributions belong in histograms. The pinned vLLM metrics surface follows this broad separation with running/waiting gauges, a KV usage gauge, token counters, and TTFT/inter-token/ E2E histograms.

Instrumentation in the inner scheduler loop also has cost. The pinned vLLM metrics design deliberately moves bookkeeping toward the frontend where possible. M5.1 therefore does not prescribe one exporter call per tensor row. It defines semantic mutation points. A runtime can batch their export, place them on a non-blocking channel, or project aggregates elsewhere, provided it measures drop, queue, and exporter-failure behavior and does not claim a sampled projection is complete.

Runtime Observability Failure Lab

• Success event before publication: audit says s2 while readers still see s1 after a crash. Instrumentation crossed the linearization boundary too early.
• Publication succeeds but audit is lost: a source-sequence gap or reconciliation mismatch means telemetry is incomplete; cache correctness must be checked at the source.
• Duplicate delivery with the same ID: deduplicate byte-equivalent content and do not double-count. Different content under one ID is corruption.
• Retry lacks handle generation: a stale completion can target a recycled request identity; reject it with the generation fence.
• Wall clocks order two processes: negative or impossible durations mean a total order was invented; use causal context.
• request_id becomes a metric label: time-series cardinality grows with traffic; keep identity in traces, audit records, or logs.
• Finish is counted as physical free: held blocks or connector delay reveal collapsed semantic, allocator, and residency planes.
• Block count is compared directly with logical bytes: slack or sharing causes a false drift alarm; define the conversion and ownership model first.
• A sampled trace lacks a failure: no conclusion about absence is valid.

The hardest observability bug is a plausible event stream. Syntactically valid records can still describe an impossible history: a publish with no start, a failure that advances snapshot state, a truncate that grows the cache, a release before finish, a physical-free observation caused by finish rather than release, or a residency observation in the semantic event itself. Those are state-machine violations, not dashboard issues.

Executable Runtime And Observability Contract

The Loop 8 evidence is deliberately more demanding than the runtime mapping:

docs/numinens-evidence/m5-1-serving-state/
    runtime_source_pins.json
    runtime_observability_fixture.json
    invalid_runtime_observability_fixtures.json
    runtime_observability.py
    test_runtime_observability.py

The validator binds the trace to the Loop 2 cache ABI and to the pinned vLLM release. It replays semantic transitions, pairs starts with exactly one publish or failure, preserves the base across retry, checks fork/truncate/release, validates source-local sequence and monotonic clocks, follows observation causes, keeps state planes separate, and rejects unbounded metric labels.

Thirty-four one-field mutations cover schema and source drift, sampled audit, metric-cardinality leaks, duplicate event IDs, missing trace identity, source/ component mismatch, local sequence gaps, clock regression, stale generations, bad append bases and intervals, orphan publication, failure mutation, fork aliasing, growing truncate, wrong finish/release state, allocator overflow, bad causal observations, negative residency, and expected-result drift.

This proves only the synthetic event contract. The following paths remain unmeasured:

actual exporter overhead and loss under load
real scheduler-to-worker and worker-to-frontend clock behavior
runtime-specific block allocation and delayed-free races
connector and residency transfer events
trace sampling and backend retention configuration
production alert thresholds and SLO attainment

The distinction is intentional: source behavior is pinned, local replay is tested, and unmeasured runtime paths remain labeled unmeasured.

0x08 Alternative Lifecycle Architectures And Translation Contracts

The vLLM mapping in 0x07 answered one implementation question deeply. It did not answer a different one: which parts of M5.1 survive when the runtime organizes ownership differently? A caller-held framework cache, a scheduler with a separate block manager, a request object attached to a prefix tree, and an executor-managed internal sequence do not expose the same fields or place their linearization boundaries in the same component.

This is not a framework benchmark. Loop 10 compares source contracts, not throughput, latency, memory efficiency, feature completeness, or code quality. Its purpose is to find the stable semantic object underneath four implementation layouts and to make every missing correspondence explicit.

The owner question is:

Given four different runtime ownership layouts,
which fields directly express M5.1 semantics,
which are only partial projections,
which belong to physical owners,
and which stronger contracts are absent?

Comparing Architectures, Not Brand Names

Let the M5.1 semantic state be the vector

$$\Sigma = (I, g_h, s, O, K_c, p, F, R_c),$$

where:

• $I$ is request identity;
• $g_h$ is handle generation;
• $s$ is snapshot version;
• $O$ is emitted-output progress;
• $K_c$ is all-layer committed K/V progress;
• $p$ is the derived lifecycle phase;
• $F$ records terminal output;
• $R_c$ records closed logical ownership.

A runtime architecture $a$ has a larger and differently shaped internal state $S_a$. Its source fields define at most a partial projection:

$$\phi_a : S_a \rightarrow \Sigma_{\bot},$$

where $\bot$ means that a semantic coordinate is absent or cannot be proven from the compared source. The projection is not expected to be bijective. A physical block table contains details absent from $\Sigma$, while a cache object may omit request generation or terminal state that $\Sigma$ requires.

This immediately rejects field-name comparison as a method. A field called kv_committed_len is useful evidence that a runtime distinguishes a K/V frontier. Its name alone does not prove the Loop 6 all-layer publication linearization point, generation fence, or idempotent receipt. Conversely, the absence of one M5.1 field does not make the runtime incorrect. It means an adapter cannot reconstruct that stronger semantic fact from that field.

Five statuses are therefore used throughout this section:

• Direct field: the named source field directly carries the compared fact.
• Derived projection: several fields or one ownership action jointly imply a useful lifecycle view.
• Near correspondence: a field resembles the semantic fact but has a different update contract or scope.
• Foreign owner: the fact belongs to a scheduler, allocator, prefix cache, executor, or framework allocator outside the logical object.
• Absent: no source surface in the audited set proves the semantic fact.

These categories are evidence labels, not scores. Counting more direct fields would not establish a better runtime architecture.

Pinning Two Additional Runtime Families

Loop 10 reuses the existing source pins for Hugging Face Transformers v4.57.0 and vLLM v0.23.0. It adds two stable release pins checked on 2026-07-27:

• SGLang v0.5.16, commit fdebc938f7f4d16fe6b9f55dcd9a767cf0899ea1;
• TensorRT-LLM v1.2.1, commit 376f7e1bd8ed543f75014309e3fd4b237e9b0e73.

The exact new source digest ledger is:

SGLang v0.5.16
  managers/schedule_batch.py
    27e20908294c9d76d0a8ae1199ecac4379232532878013404a8b4fb08842ac43
  mem_cache/base_prefix_cache.py
    f4b5387be62c2aa9d6d9134ea2be258a50b2eeb7296889612fb9e7397e8efadb
  mem_cache/radix_cache.py
    d88a676fff1edf8c61f77dbdd84bc3e1ecbe5f86eb35697618811c08a77641eb

TensorRT-LLM v1.2.1
  batch_manager/llmRequest.h
    68167660d94f0c0f3faf0f4cd57e716aeba75d785c27e9933f40d7643a5b9aaf
  batch_manager/kvCacheManager.h
    f50818b33715120d75c69722e98c37cf0d4d3ca3a6e325fec62031866d12675b
  executor/executor.h
    397a6d28ce4881410c6abbd316d73802376e57e5199bb303e6313618ab70c2f8
  docs/source/features/kvcache.md
    780cc5510af3ead29a97100701c717ab490f970221a3b443649fc0bde503188e

The SGLang Req source carries rid, append-only output_ids, kv_committed_len, finish state, request-pool indices, reusable-prefix indices, and radix-node references. Its BasePrefixCache separates prefix matching from finished- and unfinished-request handling, and the pinned RadixCache updates protected prefix length, node references, and physical indices. The SGLang paper establishes RadixAttention as an automatic K/V reuse mechanism. M5.1 uses this only to compare ownership; M5.3 still owns radix insertion, reference, and eviction mechanics.

TensorRT-LLM’s internal GenericLlmRequest separates request ID, token vectors, context progress, generated-token progress, and a context/generation/disaggregation state enum. Its BaseKVCacheManager separately exposes sequence registration, token growth, rewind, sequence removal, reuse, and K/V events. The public Executor enqueues a request, returns an ID, yields asynchronous responses, accepts cancellation, and can expose a K/V event manager. The pinned K/V cache documentation describes request-assigned block pools and cross-request radix reuse. Again, those block mechanics remain M5.3-owned.

One Semantic Vector, Four Partial Projections

The resulting implementation traces are deliberately asymmetric.

Caller-owned cache object — Transformers v4.57.0:

direct cache operations:
    Cache.update
    get_seq_length
    crop
    reorder_cache / batch_select_indices / batch_repeat_interleave

outside the Cache object:
    request identity and generation
    output token history and stopping
    service phase and release clocks
    all-layer publication receipt

get_seq_length() is a near correspondence for $K_c$, not the entire M5.1 object. Each model layer updates its own cache entry during the forward call, while the caller owns the generation loop. Loop 3 already traced why that ordinary framework path should not be silently relabeled as an atomic all-layer publication protocol. crop and batch-axis reorder methods are useful mutation primitives, but the caller remains responsible for applying the matching token, position, beam, and output-state mutation.

Request/scheduler/cache-manager split — vLLM v0.23.0:

Request:
    request_id · output_token_ids · status

scheduler:
    num_computed_tokens · finish/reconciliation paths

KVCacheManager:
    allocated request blocks · free · cache events

This split makes service identity and output progress explicit. It also makes it unsafe to infer the full lifecycle from one request field. As 0x07 showed, num_computed_tokens participates in scheduling and speculative reconciliation; it is a near correspondence, not the Loop 6 publication receipt. Request finish and physical free can occur on different paths or at different times.

Request plus prefix tree — SGLang v0.5.16:

Req semantic-facing fields:
    rid · output_ids · kv_committed_len · finished_reason

Req physical/reuse references:
    req_pool_idx · prefix_indices · cache_protected_len · last_node

prefix-cache owner:
    match/insert · unfinished handling · finished handling · node locks

This is the clearest source example of why one object can contain fields from multiple planes without making them synonymous. output_ids and kv_committed_len are separate. prefix_indices and last_node describe where reusable physical state is referenced, not how many outputs were emitted. On retraction, the pinned source resets cache-local references and kv_committed_len; generated output can remain available for recomputation in ordinary token-input paths, while input-embedding paths may need to discard generated IDs and restart. M5.1 may describe that state consequence. M5.2 owns the decision to retract, and M5.5 owns optional offload/restore.

Finished-request handling is also structurally different from immediate free. The radix cache may insert page-aligned state for reuse, free duplicate or unaligned pieces, release a request’s node lock, and keep shared tree state. Therefore:

request finished
    does not imply
request-local physical references all remain
    and does not imply
reusable prefix state is evicted

Executor-managed request and K/V sequence — TensorRT-LLM v1.2.1:

application surface:
    enqueueRequest -> request ID
    awaitResponses -> output and finish observations
    cancelRequest

internal request:
    LlmRequestState · token vectors · context current position

KV manager:
    addSequence · addToken · rewindKVCache · removeSequence
    request-to-block ownership · reuse state · K/V events

The public executor is intentionally more opaque than a caller-held tensor cache. That is an API design choice, not missing functionality. Applications observe responses and request IDs, while internal request state and K/V sequence operations remain engine-owned. rewindKVCache is a direct physical manager operation; it is not by itself a proof that output history, positions, sampling state, and snapshot generation were truncated consistently. The separate K/V event manager similarly reports cache-manager changes rather than becoming the semantic request state.

A Translation Must Preserve Transitions

A field projection is useful only if it preserves the operation being translated. Let $\delta_a$ be an architecture transition and $\delta$ the corresponding M5.1 transition. For an operation with a valid mapping, require:

$$\phi_a\!\left(\delta_a(S_a, op_a)\right) = \delta\!\left(\phi_a(S_a), op\right).$$

This is a commutation requirement, not a claim that the full runtime states are isomorphic. Consider output emission from the ordinary stop frontier:

before output_emit:
    O = 0, K = 4, snapshot = 1

after output_emit:
    O = 1, K = 4, snapshot = 1

Any adapter that derives both $O$ and $K_c$ from one token-list length maps these two semantic states incorrectly. Formally, if two runtime states collapse under the projection,

$$\phi_a(S_a^{(1)}) = \phi_a(S_a^{(2)}),$$

while their required M5.1 states differ, then the missing coordinate cannot be reconstructed from that projection. The remedy is an additional field, receipt, or wrapper-owned counter, not a guess.

The same rule applies to finish and release:

finish:
    F = true
    R = false
    K may remain 5

release:
    F = true
    R = true
    logical ownership closes

A runtime may keep reusable physical prefix state after both transitions. The physical tree or block pool then has a different owner and lifetime. This is not a violation of the semantic transition because logical request release never meant global cache eviction.

Code-Level Adapter: Facts With Qualifications

The adapter boundary should make uncertainty a value rather than erase it:

@dataclass(frozen=True)
class QualifiedFact:
    value: object | None
    status: Literal[
        "direct_field",
        "derived_projection",
        "near_correspondence",
        "foreign_owner",
        "absent",
    ]
    owner: str
    source: str


@dataclass(frozen=True)
class LifecycleProjection:
    request_identity: QualifiedFact
    handle_generation: QualifiedFact
    output_progress: QualifiedFact
    committed_kv_progress: QualifiedFact
    lifecycle_phase: QualifiedFact
    publication_receipt: QualifiedFact
    terminal_output: QualifiedFact
    logical_release: QualifiedFact
    physical_release: QualifiedFact

A source-bound adapter can then say what it knows:

def project_sglang(req) -> LifecycleProjection:
    return LifecycleProjection(
        request_identity=direct(req.rid, "Req.rid"),
        handle_generation=absent("not present in compared source"),
        output_progress=direct(len(req.output_ids), "Req.output_ids"),
        committed_kv_progress=near(
            req.kv_committed_len,
            "named committed frontier; writers not proven equivalent "
            "to the M5.1 all-layer receipt",
        ),
        lifecycle_phase=derived(req.finished_reason, req.is_retracted),
        publication_receipt=absent("no compared generation+snapshot receipt"),
        terminal_output=direct(req.finished(), "Req.finished_reason"),
        logical_release=foreign("scheduler release path"),
        physical_release=foreign("prefix cache / KV allocator"),
    )

Equivalent adapters for the other runtimes should not fill absent coordinates with plausible values. A Transformers adapter needs wrapper-owned request and output state. A vLLM adapter must retain the num_computed_tokens qualifier. A TensorRT-LLM application adapter can use request IDs and responses directly, but internal K/V sequence state remains engine-owned unless an explicit API or event exposes the required fact.

Alternative Design Choices And Their Costs

The four layouts make different engineering trades.

• Caller-owned tensor cache: transparent tensor shapes and mutations, simple single-process experimentation, and easy custom loops. Multi-request identity, atomic publication, lifecycle clocks, and reclamation discipline must be supplied by the caller or a wrapper.
• Split request and cache manager: service progress and physical memory can evolve under specialized owners. Cross-component reconciliation becomes necessary, and no single mutable counter should be declared authoritative for every plane.
• Request plus shared prefix tree: exact reuse is represented directly in request references and tree ownership. Finish, request release, reference release, and tree eviction become intentionally different events.
• Executor-managed sequence: the public API can remain stable while the engine changes scheduling and memory internals. Application-level cache introspection is weaker, so observability and correctness claims must stay within exposed responses, events, and documented manager boundaries.

Two common alternatives cut across those layouts:

co-located semantic + physical state
    fewer correlation edges
    stronger coupling to one allocator representation

split semantic + physical state
    independent evolution and scaling
    requires causal IDs, versioning, and reconciliation

eager free at finish
    simpler lifetime
    discards terminal reuse and may synchronize cleanup

retain/share after finish
    enables reuse and delayed cleanup
    requires explicit ownership, isolation, and eviction semantics

There is no context-free winner. The correct choice depends on whether the system optimizes for a custom local loop, continuous service, cross-request prefix reuse, stable executor APIs, disaggregated operation, or another workload contract. Loop 10 refuses to convert those design differences into a benchmark result that was never measured.

Worked Lifecycle Across All Four Patterns

The executable fixture uses one canonical trace:

allocate          s0  O=0  K=0  NEW
prefill_publish   s1  O=0  K=4  DECODE_READY
output_emit       s1  O=1  K=4  DECODE_READY
decode_publish    s2  O=1  K=5  DECODE_READY
finish            s2  O=1  K=5  FINISHED_RETAINED
release           s3  O=0  K=0  RELEASED

Each architecture can implement that lifecycle, but no audited source surface contains every coordinate:

• Transformers needs caller-owned identity, output, phase, and release state around the Cache object.
• vLLM needs a stronger generation/snapshot publication receipt if an external reconciler must prove the exact M5.1 transaction.
• SGLang directly separates output IDs and a committed-KV-length field, while generation fencing and the article’s receipt remain extensions.
• TensorRT-LLM applications see executor request IDs and responses; internal state and K/V sequence managers supply different parts of the projection.

The fixture does not execute these systems. It checks that the source mapping never claims a direct field where the source set only supports a partial, foreign, or absent correspondence.

Alternative Architecture Failure Lab

• Choose the runtime with the most direct fields: a documentation coverage count is presented as a performance or correctness ranking. No benchmark or end-to-end proof supports that conclusion.
• Map token-vector length to both $O$ and $K_c$: output emission and cache publication collapse; the ordinary one-token frontier cannot be represented.
• Trust kv_committed_len by name: a useful implementation field is overclaimed as generation-fenced all-layer publication without auditing all write sites.
• Treat rewindKVCache as complete truncation: K/V blocks rewind while token history, positions, sampling state, or output streams can remain inconsistent.
• Treat request finish as prefix eviction: shared reusable state is removed from the conceptual model even though the prefix-cache owner can retain it.
• Treat a K/V event as request truth: allocator/reuse observations replace semantic request state and inherit delivery or ordering gaps.
• Translate crop without the generation loop: the cache shortens while output, position, and beam state still describe the old sequence.
• Hide absent fields behind defaults: generation=0 or snapshot=len(KV) looks complete but cannot reject stale completion or partial publication.
• Compare current main branches: source behavior changes during writing; the article loses a reproducible implementation coordinate.

Executable Architecture Comparison Contract

The Loop 10 evidence is:

docs/numinens-evidence/m5-1-serving-state/
    architecture_source_pins.json
    lifecycle_architecture_fixture.json
    invalid_lifecycle_architecture_fixtures.json
    lifecycle_architecture_comparison.py
    test_lifecycle_architecture_comparison.py

The validator covers four architectures, twelve semantic dimensions, five operation families, and the six-step canonical lifecycle. It re-hashes seven new pinned source files when the local archives are available. Thirty one-field mutations reject source-binding drift, unknown correspondence status, missing dimensions, collapsed output/K/V progress, overclaimed generation or publication receipts, semantic/physical effect aliases, finish/release collapse, invalid lifecycle frontiers, and any attempt to emit a runtime ranking.

This is still source-bound synthetic evidence. It does not run any framework, exercise a GPU allocator, measure request latency, or prove that every writer of an implementation field satisfies the stronger M5.1 transaction. Its result is narrower and useful: different architectures can consume the same semantic contract only through explicit, qualified translations.

0x09 Full Failure And Sensitivity Closure

The preceding sections built the serving-state contract one coordinate at a time. A manifest fixes interpretation. A per-layer trace fixes tensor provenance. An append transaction fixes publication. The output/cache frontier fixes what has actually been processed. Lifecycle operations fix ownership, and the runtime comparison shows that implementations distribute those facts across different components.

That decomposition is useful while deriving the object, but a deployed failure does not respect article subsections. A stale append can also have the wrong mask. A beam reorder can preserve K/V bytes while corrupting token ancestry. A request can finish correctly and still trigger an invalid shared-prefix release. The final Loop 11 question is therefore compositional:

When every earlier M5.1 contract is active at once,
which failures are rejected, repaired, or handed to another owner,
and how does the logical cache ledger respond when exactly one
architecture or workload control changes?

This section closes the logical object. It does not claim that every failure has a local recovery path. Some states are rejected before publication, some require a new target call, and shared physical state remains the responsibility of M5.3.

One Invariant Chain, Nine Failure Classes

For append transaction $a$ against committed snapshot $S_s$, define the ordered predicate chain

$$\begin{aligned} \mathcal{C}(a,S_s) ={}& C_{\mathrm{ABI}} \land C_{\mathrm{position/mask}} \land C_{\mathrm{layer\ set}} \\ &\land C_{\mathrm{generation/snapshot}} \land C_{\mathrm{frontier}} \land C_{\mathrm{ownership}}. \end{aligned}$$

The order is operational, not merely stylistic. ABI and position checks can reject work before tensor rows are trusted. Layer rows are staged but remain invisible. Generation and base-snapshot checks guard the publication linearization point. Only after publication may the output/cache frontier and request ownership advance. A failure after output emission may need repair; it cannot be rewritten as a pre-publication rejection.

immutable ABI
    -> token / position / mask interval
    -> complete K and V rows for every cache-producing layer
    -> generation-fenced atomic publication
    -> separate emitted-output and committed-K/V frontiers
    -> coupled fork / reorder / truncate / cancel / finish / release

The nine required failure classes occupy different edges of that chain:

The table deliberately does not use one generic rollback word. Before publication, rollback means discarding unpublished rows. After a token has been emitted, repairing its cache state means processing that actual token in a valid target call. Releasing one logical owner does not mean erasing shared physical storage. These operations have different preconditions and different owners.

Position, Schema, And Publication Failures

The first three failures can all produce shape-correct tensors, which is why shape checks alone are insufficient.

Position and mask mismatch. Suppose the committed cache has $K=8$ rows and the call processes one current token. The append interval is $[8,9)$, and the current query must see the causal key extent declared by the model contract. Two superficially plausible calls are invalid:

cache_position = 7       # overwrites or reuses a historical slot
mask_key_length = 8      # omits the current K/V row when the call expects 9

The binding that must agree is not just a length:

$$(x_{\mathrm{current}},\ p_{\mathrm{current}},\ [K,K+q),\ \text{mask extent},\ \text{position stage}).$$

If any coordinate disagrees, pending layer rows are discarded and the call is rebuilt from the last committed snapshot. Model-specific mask construction still belongs to M3.10; M5.1 owns the fact that the append interval and mask contract travel together.

Mixed model or adapter schema. Two caches can have the same apparent [H_kv, T, d_h] shape while their rows were produced by different projection weights, adapters, RoPE conventions, K/V representations, or parallel shards. That is semantic aliasing, not compatibility. The cache ABI therefore binds at least the model artifact, adapter identity, tensor representation, positional convention, and shard geometry. Reusing a handle after changing one of those inputs without changing the ABI digest is rejected before the rows are read.

Partial-layer publication. Let $\mathcal{L}_{cache}$ be the exact set of cache-producing layers. A transaction may stage

$$P_a = \{(l,K_l^{new},V_l^{new}) : l \in \mathcal{L}_{staged}\},$$

but it may publish only when

$$\mathcal{L}_{staged}=\mathcal{L}_{cache} \quad\text{and}\quad \forall l,\;\operatorname{interval}(K_l^{new})= \operatorname{interval}(V_l^{new})=[K,K+q).$$

Publishing after layer 0 while layer 1 is absent creates no coherent next model call: the visible cache length depends on which layer reads it. The transaction therefore has one linearization point after complete staging. A failure retains the previous snapshot $S_s$ unchanged; it does not publish a shorter “best effort” state.

Collection Mutations, Stale Work, And Cancellation

The next three failures concern coordination rather than tensor arithmetic.

A beam or batch reorder is one permutation $\pi$ over every sequence-indexed field:

$$(X,C_{own},P_{pos},U_{rng},A_{anc})_{new}[i] = (X,C_{own},P_{pos},U_{rng},A_{anc})_{old}[\pi(i)],$$

where $X$ is token/output state, $C_{own}$ is cache ownership, $P_{pos}$ is position state, $U_{rng}$ is sampler/RNG state, and $A_{anc}$ is ancestry metadata. Reordering only the cache can leave every tensor shape valid while future logits condition on the wrong sequence. The operation must therefore validate that $\pi$ is a legal parent map for the generation algorithm and apply it as one coupled mutation. M3.10 owns beam scoring and parent selection; M5.1 owns the cache-side requirement that there is no split permutation.

Generation fencing closes the stale-work and ABA-shaped cases. A handle name alone is not identity:

worker W reads:       handle=h, generation=4, snapshot=12
request is released: handle=h, generation=4 -> terminal
slot is reused:       handle=h, generation=5, snapshot=0
worker W finishes:    publication for (h, 4, 12) arrives late

Without the generation coordinate, the final event appears to target the same handle. With it, the publication predicate fails before state changes. Snapshot version prevents a less dramatic but equally important lost update when two operations start from the same generation but only one may publish first.

Cancellation composes those predicates with transaction state. A legal order is:

mark request cancellation intent
    -> abort every unpublished transaction
    -> prevent new transaction admission
    -> advance or terminally close request generation
    -> release logical ownership when no legal publication remains

A row completion arriving after abort is not a partial success. A cancellation arriving after an append has already published cannot retroactively erase that linearization point; it closes the resulting snapshot through a later legal state transition. Scheduler policy decides when to cancel in M5.2. M5.1 defines what cancellation is allowed to mutate.

Final Output And Cache State Are Different Frontiers

The ordinary final-token case is a deliberate off-by-one, not an implementation accident. If the prompt has length $S$, and a generation loop processes $G_{in}$ generated tokens as inputs before selecting one final output, then immediately after selection:

$$O=G_{in}+1, \qquad K_c=S+G_{in}.$$

The user-visible output includes the final selected token. Its own K/V does not exist until that token is processed through the model. If a reusable snapshot must include it, a repair call processes the token, publishes one K/V row per layer, and discards the call’s logits rather than emitting another user token:

ordinary stop:  O = G_in + 1, K_c = S + G_in
repair append:  O = G_in + 1, K_c = S + G_in + 1

Setting K = S + O without that call invents state. Treating repair logits as another output changes the user-visible sequence. Both mutations are explicit negative fixtures.

Shared Prefix Release Is An Ownership Handoff

Shared state adds an ownership relation that is not reducible to request length. Consider an abstract prefix p0 referenced by requests A and B:

initial:       owners(p0) = {A, B}   eviction_eligible = false
release A:     owners(p0) = {B}      eviction_eligible = false
release B:     owners(p0) = {}       eviction_eligible = true

M5.1 can close the logical rule: finishing A, releasing A’s request-local handle, releasing A’s prefix reference, and evicting p0 are different events. It intentionally cannot implement the final line. Reference counts, radix nodes, page tables, copy-on-write, pinning, and eviction eligibility are physical mechanisms owned by M5.3.

That boundary is a correctness result, not missing detail. If M5.1 invented a reference-count algorithm, it would duplicate the allocator and prefix-index assumptions that M5.3 must derive. Its executable trace therefore models an abstract owner set and explicitly declares physical_mechanism_modeled=false.

Rejected Suffixes Stop At The Commit Frontier

Speculative verification exercises the same frontier with a branch. Let the committed prefix length be 100, let four candidate tokens be [d0, d1, d2, d3], and suppose only d0 and d1 are accepted before a correction token c2 is sampled.

verification output:
    accepted candidate outputs = [d0, d1]
    emitted correction output  = c2

branch-cache publication:
    commit K/V rows for d0 and d1
    discard rows for d2 and d3
    committed K after verification = 102

correction frontier:
    c2 is emitted but still unprocessed
    a later valid target append computes c2's own K/V
    committed K after repair = 103

The row computed for rejected d2 cannot be relabeled as c2; its hidden state was conditioned on a different token. The target call that computes the correction token’s K/V is therefore a real semantic operation, not bookkeeping. M7 owns candidate construction, acceptance, and correction sampling. M5.1 owns the rule that only accepted rows cross the cache publication frontier.

Deriving The Sensitivity Surface

Failure closure answers whether a state is legal. Sensitivity asks how the logical byte ledger changes when the legal manifest changes. Start with the per-committed-token K/V coefficient for the local cache shard:

$$\beta = L_{cache}\,H_{kv}^{local}\,d_h\,(b_K+b_V),$$

where $L_{cache}$ is the number of cache-producing layers, $H_{kv}^{local}$ is the number of K/V heads owned by this shard, $d_h$ is head width, and $b_K,b_V$ are representation bytes per K and V scalar. Query-head count $H_q$ is absent because persistent storage contains K/V rows. Holding $H_{kv}^{local}$ and the cache ABI fixed,

$$\frac{\partial\beta}{\partial H_q}=0.$$

That conditional clause matters. Changing model architecture from MHA to GQA often changes $H_{kv}$ with $H_q$ fixed, and then $\beta$ changes. M5.7 owns that architectural derivation; this sweep only prevents query heads from being accidentally counted as stored K/V heads.

For live requests $i=1,\ldots,N$ with committed lengths $T_i$ and materialized copy counts $c_i$:

$$\begin{aligned} B_{logical} &= \beta\sum_i T_i,\\ B_{materialized} &= \beta\sum_i c_iT_i,\\ B_{rectangular} &= \beta N\max_i T_i,\\ B_{phantom} &= B_{rectangular}-B_{logical}. \end{aligned}$$

B_rectangular is the no-copy padded logical upper bound; it is not an allocator measurement. B_materialized can exceed it when a request has more than one materialized copy, as in a beam or temporary duplication. There is no universal ordering between ledgers with different semantics.

Because $\beta$ is a product, the one-variable elasticities are one for $L_{cache}$, $H_{kv}^{local}$, $d_h$, and a common scale on $(b_K+b_V)$:

$$\frac{\partial\beta}{\partial L_{cache}}=\frac{\beta}{L_{cache}},\qquad \frac{\partial\beta}{\partial H_{kv}^{local}}=\frac{\beta}{H_{kv}^{local}},\qquad \frac{\partial\beta}{\partial d_h}=\frac{\beta}{d_h}.$$

Doubling any one of those controls doubles every byte ledger that is linear in $\beta$. Concurrency is different. Adding request $j$ changes logical bytes by

$$\Delta B_{logical}=\beta T_j,$$

not by a fixed factor unless request lengths are held equal. Its rectangular effect is non-local because the new request can also change the maximum:

$$\Delta B_{rectangular} =\beta\left[(N+1)\max(T_{max},T_j)-NT_{max}\right].$$

This is why context length and concurrency must be separate controls even though both can increase aggregate bytes.

Worked One-Control Sensitivity Suite

The synthetic base case is deliberately tiny enough to audit by hand:

L_cache = 2, H_q = 4, H_kv_local = 2, d_h = 4
b_K = b_V = 2 bytes

request A: T_A = 5, materialized copies c_A = 1
request B: T_B = 3, materialized copies c_B = 2

beta = 2 * 2 * 4 * (2 + 2) = 64 bytes / committed token
B_logical      = 64 * (5 + 3)       = 512 bytes
B_materialized = 64 * (1*5 + 2*3)   = 704 bytes
B_rectangular  = 64 * 2 * max(5, 3) = 640 bytes
B_phantom      = 640 - 512           = 128 bytes

Each row below changes exactly one declared control while holding the base fixture immutable:

What A One-Variable Sweep Does Not Prove

The table is a local derivative made discrete. It is useful precisely because its assumptions are narrow. It does not prove:

• how many requests fit on a GPU, because weights, allocator metadata, reservation, temporary workspaces, fragmentation, and safety margin are not included;
• latency, throughput, or goodput, because no kernel, scheduler, transfer, or queue is executed;
• that halving representation bytes is supported or quality-preserving, because quantization metadata, dequantization kernels, calibration, and quality gates belong to M8.6;
• that doubling query heads is free, because attention compute and projection costs still change even when the persistent K/V payload does not;
• that adding one request multiplies memory by a fixed concurrency ratio, because request lengths and copy counts are ragged;
• that one-parameter results compose under a multi-field architecture change.

A production capacity claim requires a complete memory inequality such as

$$B_{weights}+B_{runtime}+B_{allocator}+B_{KV,physical}+B_{workspace} \le B_{available}-B_{margin},$$

with every term measured or source-bound for the actual runtime. M5.1 exports the logical K/V inputs to that calculation. It does not fill in the other owners’ terms.

Executable Failure And Sensitivity Contract

The Loop 11 evidence is:

docs/numinens-evidence/m5-1-serving-state/
    failure_sensitivity_fixture.json
    invalid_failure_sensitivity_fixtures.json
    failure_sensitivity_closure.py
    test_failure_sensitivity_closure.py

The valid fixture hash-binds eight earlier manifests and negative-case sets. Its cross-oracle registry names nine failure classes and resolves seventeen specific prior mutations to their expected error codes. The validator does not duplicate those old algorithms; it proves that the closure points to immutable evidence:

for failure in required_failure_classes:
    for ref in failure.evidence_refs:
        bound_file = load_and_verify_sha256(ref.file)
        prior_case = find_case(bound_file, ref.case_id)
        assert prior_case.expected_error == ref.expected_error

It then derives all base and sensitivity ledgers from the declared geometry, representation, lengths, and copy counts. Every sensitivity case must mutate exactly one allowed control path, the base fixture must remain unchanged, and the query-head case must leave persistent K/V bytes invariant. Thirty-five one-field mutations reject binding drift, missing or duplicate failure classes, bad closure vocabulary, unknown evidence, false shared-prefix reclamation, rejected-suffix relabeling, impossible head geometry, zero copies, multi-control sweeps, expected-ledger drift, and any capacity, latency, throughput, scheduler, allocator, or quantization-kernel claim.

This is cumulative synthetic closure with no runtime benchmark. Its result is exact but narrow: the nine named logical failures map to tested rejection or handoff contracts, and the seven one-control byte sweeps follow from the declared ledger without pretending to measure a serving system.

Final Formula, Unit, And Symbol Registry

The displayed equations above are the derivations. This plain-text registry is the renderer-safe fallback and the article-wide notation contract:

cache payload on one local shard:
    B_token [bytes / committed token]
        = L_cache * H_kv_local * d_h * (b_K + b_V)
    M_committed_logical [bytes]
        = T_committed * B_token
    M_reserved_nominal [bytes]
        = T_reserved * B_token

all-layer publication:
    I(S_g) [boolean]
        = every cache-producing layer has equal committed K/V length T_g
    prepared(X) [boolean]
        = every cache-producing layer staged complete K/V over [T_g, T_g + Delta)

runtime translation:
    Sigma = (I, g_h, s, O, K_c, p, F, R_c)
    phi_a: runtime state S_a -> partial semantic state Sigma_bottom
    transition preservation [boolean]
        = phi_a(delta_a(S_a, op_a)) == delta(phi_a(S_a), op)

append compatibility:
    C(a, S_s) [boolean]
        = ABI and position/mask and layer-set and generation/snapshot
          and frontier and ownership compatibility

ordinary stop and repair:
    ordinary stop: O = G_in + 1; K_c = S + G_in
    repair append: O = G_in + 1; K_c = S + G_in + 1

analytical sensitivity:
    beta [bytes / committed token]
        = L_cache * H_kv_local * d_h * (b_K + b_V)
    B_logical [bytes]      = beta * sum_i(T_i)
    B_materialized [bytes] = beta * sum_i(c_i * T_i)
    B_rectangular [bytes]  = beta * N * max_i(T_i)
    B_phantom [bytes]      = B_rectangular - B_logical

The scoped symbols prevent three common collisions. $g_h$ is a handle generation, while $G_{in}$ counts generated tokens already processed as model inputs. $K_c$ is a committed-length counter, while $K_l$ is a layer key tensor. $R_c$ is closed logical ownership, while $U_{rng}$ is sampler state carried by a coupled reorder. bytes always means an analytical payload unless a section explicitly says reserved, physical, resident, transferred, temporary, or measured.

0x0A M5.1 Versus Later M5 Articles

M5.1 should introduce scheduler pressure, not consume the whole serving-systems map.

That boundary matters because otherwise M5.1 becomes a shallow survey of every KV technique. Its job is smaller and deeper:

define the state object
define the lifecycle
define endpoint metrics and the factual service interface
defer scheduling decisions to M5.2

0x0B What This Article Claims

The externally verified claims are:

• autoregressive serving has a multi-iteration structure that breaks static request-level batching, as described by ORCA;
• PagedAttention/vLLM identifies KV cache fragmentation, dynamic growth, and sharing as core serving problems;
• SARATHI identifies prefill and decode as distinct phases and uses chunked prefill to let decode piggyback on compute-heavy prefill work;
• DistServe separates TTFT and TPOT SLOs and treats prefill/decode interference as a serving-system problem;
• at pinned vLLM v0.23.0, request state, scheduler computed-token bookkeeping, cache-manager allocation/free, request events, metrics, and tracing are separate runtime surfaces rather than one authoritative cache-state record;
• the pinned runtime uses process-local monotonic request-event timestamps and warns against comparing them across processes;
• Prometheus treats each label-value combination as a separate time series, so unbounded request identity belongs in traces, logs, or audit records rather than metric labels;
• at pinned SGLang v0.5.16, request output, committed-K/V length, request-pool mappings, prefix indices, radix-node references, finish state, retraction state, and prefix-cache operations are distinct source surfaces;
• at pinned TensorRT-LLM v1.2.1, executor request/response operations, internal request phase and token state, K/V sequence management, rewind and sequence removal, reuse state, and K/V events are distinct source surfaces;
• MQA and GQA reduce KV cache size by reducing key/value heads, which M5.7 handles in detail; M5.8 handles MLA’s latent representation.

The imported single-request contracts are:

• cache and no-cache last-row logits should agree within the declared numerical tolerance when model, token, position, mask, and cache semantics agree;
• ordinary caching avoids re-executing historical rows but retains current-row execution, historical K/V reads, and cache append work;
• after stopping immediately after the final output selection, logical output length can be one token ahead of committed full-cache length;
• a reusable snapshot that includes the final output requires a valid append/repair call whose logits are not another returned token.

The local synthesis claim is:

M5 should be read as a change in ownership.
M4's attention kernel owns one call.
M5's serving runtime owns the cache state that lives across calls.
That ownership may be caller-held, split across managers,
attached to a shared prefix structure, or hidden behind an executor API.

That is the conceptual move. The KV cache is not just a tensor optimization. It is the serving ledger.

The cumulative synthetic claims are narrower:

• the nine named logical failure classes resolve to hash-bound rejection, repair, coupled-operation, or ownership-handoff evidence;
• the declared K/V byte ledgers are exactly reproducible from geometry, representation, request lengths, and materialized copy counts;
• query-head count does not alter persistent K/V payload when the local K/V-head count and cache ABI are fixed;
• none of those synthetic results is a measured allocator-capacity, latency, throughput, scheduler-quality, or quantization-quality result.

0x0C Publish-Gate Checks

This article should pass these checks before it is treated as publication-quality:

0xFF References

Local source articles:

• Transformer Series - KV Cache
• Transformer System M3.10 / M1.8 - Inference Execution
• Transformer System M4.1 - Transformer Cost Model
• Transformer System M7.1 - Speculative Decoding
• Transformer System M8.6 - Live-Data Quantization: Activations And K/V Cache
• Transformer Series - KV Cache Optimization
• Transformer Series - KV Cache Optimization: P/D Separation or Merging
• Transformer System M4.5 - Decode Attention Kernels, Mask Programs, and Backend Boundaries

Primary and high-quality sources:

• ORCA: A Distributed Serving System for Transformer-Based Generative Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Linearizability: A Correctness Condition for Concurrent Objects
• Hugging Face Transformers v4.57 - Cache explanation
• Hugging Face Transformers v4.57 - KV cache strategies
• Hugging Face Transformers v4.57 - cache_utils.py
• Hugging Face Transformers v4.57 - Llama implementation
• vLLM v0.23.0 - Request state
• vLLM v0.23.0 - Engine request events
• vLLM v0.23.0 - V1 scheduler
• vLLM v0.23.0 - KV cache manager
• vLLM v0.23.0 - Metrics implementation
• vLLM v0.23.0 - Metrics design
• vLLM v0.23.0 - Tracing attributes and propagation
• SGLang: Efficient Execution of Structured Language Model Programs
• SGLang v0.5.16 - Request and schedule-batch state
• SGLang v0.5.16 - Prefix-cache interface
• SGLang v0.5.16 - Radix-cache request lifecycle
• TensorRT-LLM v1.2.1 - Internal request state
• TensorRT-LLM v1.2.1 - K/V cache manager
• TensorRT-LLM v1.2.1 - Executor API and K/V events
• TensorRT-LLM v1.2.1 - K/V cache system
• OpenTelemetry - Tracing API
• OpenTelemetry - Logs data model
• Prometheus - Metric and label naming
• Prometheus - Instrumentation best practices
• vLLM - Automatic Prefix Caching design
• TensorRT-LLM - K/V cache reuse
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Fast Transformer Decoding: One Write-Head is All You Need
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints