M5.4 - Long-Context KV Policies: Retention, Compression, and Budget Allocation

M5.4 - Long-Context KV Policies: Retention, Compression, and Budget Allocation

0x00 Historical Introduction And Owner Contract

M5.1 established why autoregressive serving materializes K/V state in the first place. Without a cache, every decode step would project keys and values again for the entire accepted prefix. With a cache, each committed token contributes one persistent K/V row per layer and later queries reuse those rows.

That exchange removes repeated projection work; it does not make a long history free. Under full-cache attention, the persistent state grows with the accepted sequence, and every new query remains logically able to read every old row. M5.3 can allocate those rows in pages, map them through virtual memory, or reuse an exactly compatible prefix. Those mechanisms change where exact state lives and who owns it. They do not decide that an old token has ceased to be part of the model-visible history.

The owner question is:

Given a compatible model and a finite historical-state budget, which past representations remain visible to each layer and head, how is that decision updated, and what quality and runtime contract replaces full-cache attention?

M5.4 owns that decision as one conceptual KVVisibilityPolicy. Its thesis is:

Long-context K/V optimization is not one technique and not one byte count.

The exact baseline keeps every committed row visible.
A visibility policy may retain original rows, choose different rows per
layer or head, or replace several rows with another representation.

Each policy must name:
    the state it changes,
    the evidence used to change it,
    the work and metadata it adds,
    whether omitted history is recoverable,
    and the quality contract that is no longer exact full-cache attention.

0x00A The Cache Solved Recomputation And Created A Historical-State Problem

The historical progression matters because the word “cache” hides several different problems:

no K/V cache:
    repeatedly project old accepted tokens

exact full K/V cache:
    project each accepted row once
    retain every row
    expose every row to later attention

exact dynamic layout:
    retain the same logical rows and visibility
    allocate, share, and reclaim physical storage more precisely

long-context visibility policy:
    change which historical rows or representations attention may consume

The full cache is therefore the semantic baseline for this article:

for query position t and layer l:
    legal visible history = every committed position 0..t
    stored object = the checkpoint-compatible K/V representation
    omitted committed history = empty

This baseline does not promise that a model reasons perfectly over every visible token. It says only that the cache policy has not removed evidence that the checkpoint’s attention rule could otherwise read. That distinction is important: model capability, positional generalization, physical storage, and legal visibility are separate contracts.

A second distinction is equally important:

sparse attention execution:
    all historical K/V may remain stored
    a backend computes only a declared subset for one query
    generic kernel and mask execution remain M4.5 concerns

persistent K/V retention:
    omitted rows no longer belong to the resident visible set
    future queries cannot use them unless another contract recovers them
    this is M5.4

Both can reduce attention work. Only the second necessarily reduces persistent K/V rows, and only the second changes the future visibility state owned here.

0x00B From Attention Concentration To A Visibility Decision

The handwritten source A25 introduces sequence-length reduction through two empirical observations: attention can be concentrated on a subset of positions, and different tokens can receive very different attention mass. That is the right motivation, but it is not yet an eviction proof.

Attention is conditional on at least:

current query token
layer and attention head
prompt and generated history
task and data distribution
checkpoint and position regime
decoding step

The figure below is the A25 crop of Keyformer’s source-reported CNN/DailyMail summarization measurements. It shows that attention sparsity varies across models and layers and that substantial cumulative attention mass can be concentrated in a fraction of the context under that experiment. It does not establish one universal cache ratio, and it does not prove that a position with low attention now will remain irrelevant to an unknown future query.

Source: Keyformer, Figure 3(a,b), cropped as preserved in A25. The measurements are evidence for the named models, dataset, and evaluation setup, not a universal future-relevance law.

Turning concentration into a persistent cache policy adds a prediction problem:

observed fact:
    a row received little or much attention for queries already evaluated

policy inference:
    a row is unlikely or likely to matter to future queries

irreversible action:
    evict or merge the row so future full-cache attention is unavailable

That last implication is method-specific. Sliding windows use recency. StreamingLLM protects initial sink positions plus recent rows. H2O, Scissorhands, and Keyformer use different importance histories or score rules. SnapKV observes a prompt suffix and clusters selected prompt positions. PyramidKV, PyramidInfer, FastGen, ZigZagKV, and HeadKV make layer, head, phase, or budget heterogeneity explicit. DMC, MiniCache, and Layer-Condensed KV Cache change the stored representation rather than merely selecting an unchanged row set.

The governing warning is:

current attention is evidence about the current query;
it is not proof of future irrelevance

0x00C The Owner Object: KVVisibilityPolicy

The article uses one conceptual object to stop those methods from collapsing into a list of paper names:

KVVisibilityPolicy
    baseline:
        compatible full-cache attention contract

    selection unit:
        token | token group | head-token | layer-token | layer-head-token

    decision phase:
        prefill | decode | both

    visible-state rule:
        protected positions
        selected original rows
        replacement or merged representations
        per-layer and per-head budgets

    update rule:
        static | periodic | every decode step | learned transition

    evidence:
        recency | attention history | query similarity
        observation window | learned gate | architecture state

    recoverability:
        resident and visible
        permanently omitted from this cache generation
        nonresident but recallable through M5.5

    failure contract:
        invalid policy state
        insufficient physical capacity
        selector or representation failure
        quality threshold or fallback action

This is a teaching object, not a claim that the cited papers or serving engines expose one class with these fields. Loop 2 below turns the concept into an immutable manifest and rejects ambiguous or impossible combinations.

The policy output must also remain distinct from its consumers:

M5.4 decides the legal logical visible state
M5.3 decides how accepted state is physically mapped
M5.2 decides when admitted work is served
M4.5 decides whether a backend can execute the resulting layout and mask
M5.5 decides how recallable nonresident state is found and transferred

0x00D Four Reading Lanes

The methods in A25 and its primary sources become easier to compare when organized by the object they change:

The lanes are not a ranking. A system can compose them, but composition is a new contract. For example, per-head prompt selection over quantized K/V in an offload hierarchy has at least four independent sources of approximation, metadata, failure, and latency. A paper name does not remove the need to state those coordinates.

0x00E Owner Contract: Consumes, Owns, And Defers

M5.4 may consume a neighboring fact without re-owning the mechanism that produced it:

Three boundaries are especially easy to confuse:

exact layout:
    the same legal history exists
    storage and reuse change
    owner: M5.3

lossy visibility:
    some original history is no longer legal input to attention,
    or is replaced by another declared representation
    owner: M5.4

recallable nonresidency:
    state remains in a larger logical pool but is not currently local;
    lookup, transfer, timeout, and fallback decide whether it arrives
    owner: M5.5

0x00F A25 Source Continuity And Corrections

A25 is not discarded. Its mechanism depth remains the main source-local spine, but Transformer System assigns each part to one owner and tightens several statements:

All benchmark numbers remain source-reported. Later loops must preserve each paper’s checkpoint, task, context length, cache budget, baseline, hardware, runtime, and quality coordinate before comparing results.

0x00G Memory Management Can Preserve Or Change Model Context

An allocator can move exact K/V bytes without changing which tokens attention can read. A retention policy makes a stronger decision: some history may be discarded, summarized, compressed, predicted, or fetched only when needed. That changes the effective attention context even if the prompt was originally accepted in full.

exact layout optimization:
    same K/V values and visibility, different storage

lossy retention:
    fewer or compressed values remain visible

retrieval or offload:
    values exist elsewhere but may not arrive for every step

recomputation:
    trade additional model work for lower persistent storage

Long-context K/V methods therefore need more than one result column: persistent and temporary bytes, attention and policy work, metadata, latency, and task-quality change. A policy that works for local dialogue may fail when an answer depends on an early document span or a specific tool event.

0x00H Fixed Size Does Not Identify The Stored Object

The phrase fixed-size memory gives only a capacity bound. It does not say what one slot represents, whether old evidence remains individually addressable, or whether an unchanged checkpoint can consume the state.

The tensor paths expose why the labels cannot be exchanged:

raw K/V window:
    K_keep, V_keep                 [L, H_kv, W, d_h] each
    -> consumed directly by the checkpoint's attention kernel

merged K/V:
    merge(K_group, V_group)
    -> K_bar, V_bar                [L, H_kv, K, d_h] each, K < W
    -> original grouped rows are not recoverable from the cache contract

recurrent hidden memory:
    H_mem                          [L, M, d_model]
    concat(stop_gradient(H_mem), H_current)
    -> architecture-specific K/V projections for the next segment

local plus compressed memory:
    H_local                        [L, M_local, d_model]
    H_compressed                   [L, M_comp, d_model]
    -> attention over fine and coarse activation memories

accumulated segment summaries:
    sigma_<i                       [B, (i - 1) * kappa, d_model]
    -> prepended to segment i as soft-prompt embeddings
    -> current segment emits kappa new final-layer summary vectors

For a scalar width of b bytes, the first-order resident-state ledgers are therefore different:

$$M_{\mathrm{raw\ KV}} = L\cdot W\cdot 2H_{kv}d_hb,$$ $$M_{\mathrm{hidden\ memory}} = L\cdot M\cdot d_{model}b,$$

and a two-tier activation memory has the payload

$$M_{\mathrm{dual\ activation}} = L\cdot(M_{local}+M_{comp})d_{model}b,$$

before compressor parameters, temporary workspaces, positional state, allocator slack, or attention outputs. Equal slot counts do not imply equal bytes: 2 H_kv d_h can differ substantially from d_model, especially under MQA or GQA.

Transformer-XL is the recurrent-hidden-state example. It caches a sequence of previous-segment hidden states, stops gradients through that reused segment state during the next segment, and conditions the next segment’s keys and values on the extended hidden context. Its recurrence and relative-position contract are model semantics, not a K/V eviction plugin.

Compressive Transformer adds a second resolution level. When fine-grained memories age out, a learned or defined compression function maps them into a smaller compressed memory; attention can read both the recent granular memory and the older coarse memory. This is architecture-level lossy state trained for long-range modeling, not proof that averaging arbitrary serving-cache rows is safe.

The classification rule is:

name the stored object before naming the budget

A statement such as “memory size = 512” is incomplete until it says whether those 512 slots are K/V rows, hidden activations, compressed activations, learned summaries, or retrieval references.

0x01 Claim Status And Scope

M5.4 should be read as a contract map, not as an endorsement of one policy. The important questions are:

Which K/V entries remain visible?
Which byte term went down?
What scoring, clustering, or retrieval state was added?
What prompt can now fail?

0x01A Immutable Visibility Manifest, Policy State, And Tensor Identity

The owner object from 0x00 is still too informal for implementation. A runtime cannot safely execute “keep important tokens” until it can answer:

Which immutable policy meaning was admitted?
Which cache generation and committed frontier did the decision observe?
Which original token position does each packed K/V row still represent?
Which K/V head owns that row under MHA, MQA, or GQA?
Did selection preserve an original row or create a replacement representation?
Can omitted state be recovered, and which module owns that recovery?

Loop 2 freezes those answers before deriving any paper-specific score. The mechanical owner question is:

Given one logical cache ABI and one admitted visibility policy, how can a runtime update a compact per-layer/per-head view without losing row identity, position semantics, generation fencing, or the distinction between exact and approximate state?

Five Records, Not One Mutable Policy Dictionary

One dictionary containing configuration, selected rows, physical pointers, and temporary scores would give every subsystem authority to reinterpret every other subsystem’s state. M5.4 instead separates five records:

The first two records define meaning. The next two describe one state and one materialized view under that meaning. The fifth describes storage. Updating a selection produces a successor KVPolicyState; it does not edit the manifest or relabel existing physical bytes.

policy manifest p7:
    immutable selection and representation semantics

state s11:
    selected positions derived from committed cache generation c23

next decode decision:
    scores new evidence against s11 and c24
    publishes s12 only if its source generations still match

late worker derived from c23:
    cannot overwrite s12 or reinterpret c24

This generation split is normative. It does not require JSON or SHA-256 on the hot path; packed structs and integer handles are valid implementations if they reconstruct the same immutable contract.

KVVisibilityPolicyManifest: Freeze Meaning Before State

The article-local manifest contains:

KVVisibilityPolicyManifest
    schema_version
    policy_manifest_id
    policy_generation
    logical_cache_abi_sha256
    checkpoint_generation

    tensor_contract:
        cache-producing layers L
        query heads H_q
        K/V heads H_kv
        head dimension d_h
        K and V scalar representations
        logical source axes
        query-head -> K/V-head map
        key positional stage and value stage

    selection_contract:
        selection unit
        decision phase
        budget scope and maximum rows
        protected-position rule
        evidence family and update cadence
        ordering and duplicate rules

    representation_contract:
        original rows | replacement rows
        full-cache exact | lossy subset | lossy replacement
        replacement consumer schema, when required

    recoverability_contract:
        resident visible | omitted irrecoverable | recallable nonresident
        M5.5 handoff identity, when recallable

    quality_and_fallback:
        evaluation profile
        minimum quality gate
        invalid-state action
        physical-capacity action
        quality-regression action

Several combinations are impossible rather than merely undesirable:

full_cache_exact + omitted positions:
    reject; the state contradicts its semantic class

lossy_subset + replacement_rows:
    reject; original-row selection and representation replacement differ

lossy_replacement + no consumer schema:
    reject; the packed object has no declared interpretation

recallable_nonresident + no M5.5 handoff:
    reject; "recoverable" has no lookup, movement, or fallback contract

per_layer_kv_head selection + no q_to_kv_head_map:
    reject; query-head consumers cannot identify their K/V slice

The manifest identifies a policy family and contract, not the rows selected at one decode step. A state update caused by new attention evidence therefore does not advance policy_generation. Changing the selection unit, protected set, representation kind, K/V-head mapping, or quality/fallback contract does.

Canonical Identity And Deliberate Exclusions

The teaching fixture derives one immutable identity:

policy_manifest_sha256 = SHA256(canonical_json({
    schema_version,
    policy_manifest_id,
    policy_generation,
    logical_cache_abi_sha256,
    checkpoint_generation,
    tensor_contract,
    selection_contract,
    representation_contract,
    recoverability_contract,
    quality_and_fallback
}))

Canonicalization recursively sorts object keys and preserves array order. Array order is semantic for axes, head maps, and protected-position lists; a serializer must not sort those arrays as if they were sets.

The following mutable facts are deliberately excluded from the manifest hash:

Exclusion does not make those fields unimportant. It prevents a state update or physical move from pretending to be a new policy definition.

KVPolicyState: A Successor Snapshot, Not An In-Place Edit

The mutable process publishes immutable snapshots:

KVPolicyState
    request_id
    request_generation
    cache_generation
    policy_manifest_sha256
    policy_generation
    state_generation
    parent_state_generation

    source_committed_frontier
    source_position_origin
    decision_phase and decision_epoch
    evidence_receipt_id

    selected_original_positions[L][H_kv][]
    protected_original_positions[]
    omitted_interval_summary
    replacement_object_ids, if representation-changing

Every selected position must satisfy:

$$p_{origin} \le p < p_{origin}+T_{committed},$$

must occur at most once in one layer/head selection, must obey the manifest’s ordering rule, and must fit that layer/head’s budget. Every protected position inside the committed frontier must remain selected wherever the protected rule applies.

Renderer-safe fallback:

position_origin <= selected_position
selected_position < position_origin + committed_frontier

An asynchronous selector may read state s11 and cache generation c23 while decode publishes c24. Its result is not “almost current”: it is stale input to a state transition and must be retried or discarded. Likewise, a request generation mismatch is not repaired by matching token counts; it may identify a canceled, forked, or recycled request lineage.

Source Tensor To Ragged Packed View

Use the logical source tensors consumed from M5.1 and M5.7:

$$K,V \in \mathbb{R}^{L\times H_{kv}\times T\times d_h}.$$

For each layer l and K/V head h, the policy state publishes an ordered position set:

$$S_{l,h}=(p_0,p_1,\ldots,p_{K_{l,h}-1}).$$

Original-row subset packing is a gather, not a re-encoding:

$$K^{pack}_{l,h,r,:}=K_{l,h,S_{l,h}[r],:}, \qquad V^{pack}_{l,h,r,:}=V_{l,h,S_{l,h}[r],:}.$$

The compact row r is only a storage coordinate. Its semantic position is:

$$P^{pack}_{l,h,r}=S_{l,h}[r].$$

Renderer-safe fallback:

packed_K[l,h,r,:] = source_K[l,h,position_map[l,h,r],:]
packed_V[l,h,r,:] = source_V[l,h,position_map[l,h,r],:]

packed row r is not token position r

A practical ragged representation can flatten all selected head rows:

packed_K_rows          [R_total, d_h]
packed_V_rows          [R_total, d_h]
original_position_map  [R_total]
row_offsets            [L, H_kv + 1]

For one (l,h) slice:

begin = row_offsets[l, h]
end   = row_offsets[l, h + 1]

packed_K_rows[begin:end]
packed_V_rows[begin:end]
original_position_map[begin:end]

The fixture uses independent global offsets per layer row, so every offset is monotonic and the final offset equals R_total. Other legal ABIs can store a base offset per layer plus head-local offsets. The convention must be frozen; mixing the two can yield shape-correct reads from another head.

Position Identity Survives Packing

Suppose one head keeps original positions [0, 2, 5, 7]. Replacing them with dense positions [0, 1, 2, 3] changes more than metadata:

wrong interpretation:
    packed row 2 -> token position 2

correct interpretation:
    packed row 2 -> original token position 5

The original position can affect causal legality, positional bias, relative distance, rotary-key interpretation, diagnostics, later scoring, and any attempt to join policy state with the authoritative token stream. If keys were stored after RoPE, gathering preserves their already-applied rotary values; it does not authorize recomputing them at packed indices. If a backend needs positions for another mask or bias rule, it consumes original_position_map.

For a decode query at original position t, the legal selected keys satisfy:

$$P^{pack}_{l,h,r}\le t,$$

not r <= t. A policy can impose a stricter visible subset, but it cannot use the packed coordinate as evidence that a future original position is causal.

GQA Maps Query Heads To K/V Identity; It Does Not Duplicate Ownership

Under GQA, H_q can exceed H_kv. The manifest carries an explicit mapping:

q_to_kv_head_map[q_head] -> kv_head

The fixture uses:

H_q  = 4
H_kv = 2

q_to_kv_head_map = [0, 0, 1, 1]

Query heads 0 and 1 consume the packed slice owned by K/V head 0; query heads 2 and 3 consume head 1. The visibility budget is charged once per stored K/V head, not once per query head. Expanding K/V rows to [H_q, ...] for a kernel may be a transient execution view, but counting it as persistent selected state would erase the M5.7 architecture contract.

The manifest uses an explicit map rather than assuming q_head // group_size because that arithmetic is only valid for the declared contiguous grouping. The validator still checks the fixture’s named contiguous_gqa mode against that rule.

Complete Two-Layer, Two-K/V-Head Trace

The executable fixture freezes:

L = 2, H_q = 4, H_kv = 2, T = 8, d_h = 4
b_K = b_V = 2 bytes
maximum rows per layer/K/V head = 4
protected original positions = [0, 7]

The selected positions are:

Therefore:

R_total = 16 layer-head rows

original_position_map =
    [0,2,5,7, 0,1,6,7, 0,3,4,7, 0,2,6,7]

row_offsets =
    [[0,4,8],
     [8,12,16]]

The oracle constructs symbolic source values:

K[l,h,t,c] = 1000*l + 100*h + 10*t + c
V[l,h,t,c] = 5000 + 1000*l + 100*h + 10*t + c

For layer 1, K/V head 0, packed row 1 maps to original position 3:

packed K row = [1030, 1031, 1032, 1033]
packed V row = [6030, 6031, 6032, 6033]
position map = 3

That one check jointly catches wrong layer, wrong head, wrong token, K/V misalignment, and packed-index substitution.

The useful full-cache payload is:

M_full = L * H_kv * T * d_h * (b_K + b_V)
       = 2 * 2 * 8 * 4 * (2 + 2)
       = 512 bytes

The selected payload is:

M_selected = R_total * d_h * (b_K + b_V)
           = 16 * 4 * (2 + 2)
           = 256 bytes

The teaching metadata adds:

position map = 16 * 4 bytes = 64 bytes
row offsets  = 2 * 3 * 4 bytes = 24 bytes

The resulting 344 B is not an allocator or kernel measurement. It proves that selected payload, position metadata, and offsets are separate terms. A padded backend may spend more, and a fused implementation may encode offsets differently.

Packing Transaction And Publication Checks

The normative subset path is:

def materialize_subset_view(manifest, state, source_cache):
    require(state.policy_manifest_sha256 == digest(manifest))
    require(state.policy_generation == manifest.policy_generation)
    require(state.cache_generation == source_cache.generation)
    require(state.source_committed_frontier == source_cache.committed_frontier)
    validate_selection(manifest, state)

    packed_k, packed_v = [], []
    position_map, row_offsets = [], []

    for layer in range(manifest.L):
        offsets = [len(position_map)]
        for kv_head in range(manifest.H_kv):
            for position in state.selected_original_positions[layer][kv_head]:
                packed_k.append(source_cache.K[layer, kv_head, position, :])
                packed_v.append(source_cache.V[layer, kv_head, position, :])
                position_map.append(position)
            offsets.append(len(position_map))
        row_offsets.append(offsets)

    receipt = PackedKVViewReceipt(
        policy_manifest_sha256=digest(manifest),
        state_generation=state.state_generation,
        cache_generation=source_cache.generation,
        packed_k=packed_k,
        packed_v=packed_v,
        original_position_map=position_map,
        row_offsets=row_offsets,
    )
    validate_gather_identity(receipt, source_cache, state)
    return receipt

Publication occurs only after every K/V row, position entry, and terminal offset has validated. A partially filled packed buffer is workspace, not a visible successor state. Physical placement can then consume the immutable receipt without gaining authority to change its selected positions.

Alternative Materializations Are Different ABI Choices

The flat ragged gather is not the only implementation:

These are design choices, not aliases. A paper can prove a better selection rule while a runtime loses the expected latency benefit because its backend pads every head to K_max or repeatedly gathers noncontiguous rows. M4.5 owns the measured backend result; M5.4 must publish enough identity for that backend to be correct.

Executable Manifest, Oracle, And One-Field Failures

The article-local oracle canonicalizes the manifest, validates the state, constructs symbolic K/V tensors, materializes the ragged view, and verifies every packed scalar against its source coordinate. Negative fixtures mutate one contract at a time:

manifest digest or policy generation mismatch
stale request/cache/state generation
committed frontier mismatch
query-head map length or ownership error
axis, dtype, position-stage, or representation mismatch
duplicate, unsorted, protected-missing, over-budget, or out-of-range position
row-offset or position-map drift
K/V gather misalignment
full-cache exact label with omitted rows
replacement rows without a consumer schema
recallable state without an M5.5 handoff

All reject before the result becomes a PackedKVViewReceipt. The executable evidence lives in:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    visibility_policy_fixture.json
    invalid_visibility_policy_fixtures.json
    visibility_policy_oracle.py
    visibility_policy_report.json
    test_loop2_visibility_manifest.py

This is source-bound synthetic evidence. It proves deterministic identity, shape, gather, generation, and rejection behavior for the declared fixture. It does not reproduce a paper’s selector, measure GPU performance, or prove quality under cache reduction.

Loop 3 can now derive the rectangular, per-layer, per-head, metadata, workspace, and temporary-state byte ledgers without guessing what one retained row means.

0x02 Full Baseline, Policy Cost, Traffic, And Accuracy

Loop 2 fixed the identity of one visible K/V row. This loop asks what that row costs to retain, select, publish, and read, and what approximation error follows when rows are omitted. Those are four different ledgers:

persistent state    bytes that survive between decode steps
transaction peak    old state + unpublished successor + selection workspace
traffic             bytes read or written while selecting and attending
accuracy            change in the local attention output, then full-model quality

No single “cache size” number answers all four.

0x02A One Visible-Set Notation

Let

$$V_{b,l,h}(t)$$

be the ordered set of original token positions visible to batch row $b$, layer $l$, and stored K/V head $h$ when query step $t$ executes. Its cardinality is

$$K_{b,l,h}(t)=\left|V_{b,l,h}(t)\right|.$$

The Loop 2 receipt preserves every position in this set. Packing may move a row; it may not erase which original position, layer, or K/V head produced it.

One selected position stores a key row and a value row. If their scalar widths are $b_K$ and $b_V$ bytes, one row instance costs

$$B_{row}=d_h(b_K+b_V).$$

This form is derived, not memorized: each of K and V has $d_h$ scalars, so its byte count is dimension times scalar width; adding the two arrays gives the row-instance payload. The formula intentionally does not multiply by query heads. Under GQA, several query heads consume one stored K/V head.

0x02B Full Persistent Payload From One Row

For a rectangular full cache with $T$ committed positions per sequence, there are $B L H_{kv} T$ row instances. Therefore:

$$M_{full} =B L H_{kv} T B_{row} =B L H_{kv} T d_h(b_K+b_V).$$

With equal K/V widths $b_K=b_V=b$, this is the familiar $B T L 2H_{kv}d_hb$ expression. The factor two is not an unexplained attention constant; it is the sum of one key array and one value array.

Using the running M5 shape:

B = 1, L = 32, H_kv = 8, d_h = 128
b_K = b_V = 2 bytes

B_row                         = 128 * (2 + 2) = 512 bytes
bytes per token, all layers   = 32 * 8 * 512  = 131072 bytes = 128 KiB

Then:

The read column is a logical payload ledger, not a profiler result. M4.5 owns kernel tiling, page traversal, cache behavior, and measured HBM traffic. M5.4 owns which rows the kernel is permitted to read.

0x02C Persistent Policy State Is More Than Payload

A selected-subset policy stores the payload implied by its visible sets:

$$M_{visible} =B_{row}\sum_{b,l,h}\left|V_{b,l,h}(t)\right|.$$

But a runnable policy can also need original-position maps, ragged offsets, validity bits, selection scores, protected-position state, and a versioned receipt:

$$M_{policy} =M_{visible}+M_{position}+M_{offset}+M_{validity} +M_{score}+M_{protected}+M_{receipt}.$$

The terms answer different questions:

Allocator capacity, physical fragmentation, shared-page refcounts, and measured residency remain M5.3 concerns. A host or remote fallback copy remains an M5.5 object. They must not be silently folded into $M_{policy}$.

0x02D Ragged, Padded, And Tombstone Layouts Have Different Costs

Consider a small fixture with two layers, two K/V heads, eight committed positions, and retained row counts:

K[l,h] = [[4, 2],
          [3, 1]]

sum K[l,h] = 10 selected row instances
B_row = d_h * (b_K + b_V) = 4 * (2 + 2) = 16 bytes

A compact ragged representation stores ten rows. A padded representation with $K_{max}=4$ stores $2*2*4=16$ row slots. A tombstone representation keeps the full 32-row payload and adds live-state metadata.

The ragged padding efficiency is

$$\eta_{pad}=\frac{10}{16}=0.625.$$

The toy ragged total is even larger than the bare full payload, 545 B > 512 B, because the fixture deliberately includes a fixed 192-byte receipt and score state for every historical candidate. That is not a paradox and does not show that long-context selection is useless. It shows why small examples and short requests can be metadata-dominated, while the $O(Td_h)$ payload term dominates at long context.

0x02E Score State Can Still Scale With History

Let $C_{b,l,h}(t)$ be the candidate universe a selector can reconsider. It is not necessarily the current visible set. If one score of width $b_s$ is kept per candidate, then

$$M_{score}=b_s\sum_{b,l,h}|C_{b,l,h}(t)|.$$

This yields a design fork:

score only retained rows:
    bounded state, but permanently evicted rows cannot be reconsidered

score all historical rows:
    broader future choice, but score metadata may still grow with history

score recallable external rows:
    state can be reconsidered, but M5.5 must provide lookup and fetch semantics

A claim of “constant K/V” is therefore incomplete unless it also names how the candidate universe and selection evidence are bounded.

0x02F Workspace And Transaction Peak

An update does not teleport from old state to new state. A generation-safe publication can keep the old view live while constructing a successor. If $M_{topk}$ is selector workspace and $M_{backend}$ is additional kernel or runtime workspace, a conservative transaction ledger is

$$M_{peak} =M_{old\ live}+M_{new\ unpublished}+M_{topk}+M_{backend}.$$

In the executable toy:

top-k output workspace             = 80 B
initial full -> ragged peak         = 512 + 545 + 80 = 1137 B
ragged -> ragged update peak        = 545 + 545 + 80 = 1170 B
declared backend workspace          = 0 B

Here zero means “not included in this analytic fixture,” not “the real backend allocates nothing.” A production measurement must report temporary allocation, stream overlap, allocator reuse, and synchronization separately.

0x02G Update Traffic And Write Amplification

A simple publication pass can scan candidate scores, read selected payload, and write a new persistent view:

$$T_{publish} =T_{score\ scan}+T_{selected\ read}+T_{persistent\ write}.$$

For the same fixture:

score scan read             = 128 B
selected K/V payload read   = 160 B
new persistent-state write = 545 B
publication traffic         = 833 B

One newly admitted token contributes 64 B of K/V across the fixture’s layers and K/V heads. Two useful ratios expose different costs:

K/V compaction amplification = (selected payload read + selected payload write)
                               / new-token K/V
                             = (160 + 160) / 64 = 5.0x

publication amplification    = publication traffic / new-token K/V
                             = 833 / 64 = 13.015625x

These are logical byte ratios for a declared algorithm. They are not measured HBM amplification. An in-place ring, stable pages, incremental heap, or fused selector can change the traffic terms without changing the visible-set policy.

0x02H Unique K/V Reads And GQA Query-Expanded Demand

Persistent ownership follows K/V heads. Compute demand follows query heads. For a GQA map [0,0,1,1], two query heads consume each stored K/V head. The toy therefore has:

unique selected K/V payload per query step     = 160 B
query-expanded logical demand                  = 320 B

The second number must not be reported as persistent storage. Whether hardware actually reloads rows for each query head depends on the kernel and cache hierarchy. M4.5 owns that measured execution question.

0x02I Cumulative Decode Traffic Is A Trajectory

A per-step full-cache read grows as the committed prefix grows. If generation starts with $T_0$ visible positions and emits $G$ tokens, with one new visible position per step, the number of row-time instances is

$$\sum_{g=0}^{G-1}(T_0+g) =GT_0+\frac{G(G-1)}{2}.$$

Multiplying by all-layer bytes per token gives the logical cumulative read:

$$T_{full,cumulative} =B L H_{kv}d_h(b_K+b_V) \left(GT_0+\frac{G(G-1)}{2}\right).$$

For $T_0=32768$, $G=512$, and the running 128-KiB-per-token shape:

full-cache logical cumulative read   = 2.015594482421875 TiB
bounded 4096-row logical read         = 256 GiB
analytic reduction ratio              = 8.0623779296875x

The ratio is not a latency prediction. Policy updates, irregular addressing, metadata reads, fallback, and backend efficiency can consume some or all of the logical saving.

0x02J Fallback Is Work And Capacity

An omitted row can be:

irrecoverable:
    quality contract accepts its absence

recomputable:
    source tokens and a valid model path can rebuild it at extra compute cost

recallable:
    exact or approximate state remains in another tier and M5.5 can fetch it

full-cache fallback:
    enough capacity remains to abandon the lossy path for selected requests

Each mode needs reserved capacity and a service policy. M5.4 declares the quality trigger and legal fallback. M5.1 owns cache lifecycle, M5.2 owns admission and service consequences, M5.3 owns local capacity, and M5.5 owns nonlocal state movement.

0x02K Dense And Selected Attention Share One Denominator

For one query, layer, and head, let logits be $z_i$ and values be $v_i$. Dense attention uses

$$Z=\sum_j e^{z_j}, \qquad p_i=\frac{e^{z_i}}{Z}, \qquad o=\sum_i p_i v_i.$$

For selected positions $S$, define retained dense mass

$$M_S=\sum_{i\in S}p_i =\frac{\sum_{i\in S}e^{z_i}}{Z}.$$

Renormalizing only selected logits gives

$$\tilde p_i =\frac{e^{z_i}}{\sum_{j\in S}e^{z_j}} =\frac{p_i}{M_S}, \qquad i\in S,$$

and local selected output

$$\tilde o_S=\sum_{i\in S}\frac{p_i}{M_S}v_i.$$

This derivation explains the final formula: the selected softmax denominator is the dense denominator multiplied by retained mass. The local error oracle reports

$$E_{out}=\|\tilde o_S-o\|_2,$$

alongside retained mass $M_S$, omitted mass $1-M_S$, and oracle top-k recall. None of those is full-model task quality.

0x02L A Cancellation Counterexample

Retained softmax mass and output error can rank two selectors differently. Take

dense probabilities p = [0.4, 0.3, 0.2, 0.1]
scalar values         v = [0,   1,   0,  -3]

The dense output cancels:

$$o=0.4(0)+0.3(1)+0.2(0)+0.1(-3)=0.$$

Now compare two retained sets:

The lower-mass set exactly matches this local output because its retained values are both zero. This does not make it a universally better selector: another query, value direction, layer, token, or downstream nonlinear path can reverse the result. It proves only the required negative claim:

retained attention mass is not an ordering theorem for local output error;
local output error is not an ordering theorem for end-task quality.

Relative error is undefined here because the dense output norm is zero. The oracle records null rather than manufacturing a denominator.

0x02M Executable Oracle And Evidence Boundary

The Loop 3 evidence package recomputes every number above from one fixture:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    cost_accuracy_fixture.json
    invalid_cost_accuracy_fixtures.json
    cost_accuracy_oracle.py
    cost_accuracy_report.json
    test_loop3_cost_accuracy.py

It rejects 23 one-contract mutations covering identity, dimensions, row counts, padding, scalar widths, score state, GQA ownership, workspace, decode trajectory, attention selections, and declared-result drift. The valid report is explicitly not allocator measurement, GPU traffic measurement, latency measurement, paper benchmark reproduction, or full-model task quality:

task_quality = null

Loop 4 can now change the evolution of $V_{b,l,h}(t)$ for full cache, sliding windows, and sink-plus-recent policies while saying exactly which persistent, peak, traffic, and local-error terms changed.

0x03 Exact Full-Cache State Evolution

The static policies need one off-by-one convention before they need an algorithm. Let $F$ be the exclusive committed frontier:

F = 0  -> no committed positions
F = 4  -> committed original positions [0,1,2,3]
F = 8  -> committed original positions [0,1,2,3,4,5,6,7]

The full-cache visible set is therefore

$$V_{full}(F)=\{0,1,\ldots,F-1\}, \qquad |V_{full}(F)|=F.$$

This is a state after publication. While a new token is being evaluated, its K/V row may exist in temporary kernel state. M5.1 commits that row first; only then may M5.4 publish a successor visibility state with frontier $F+1$. A rejected or canceled row must never advance the published frontier.

Full-Cache Transition

Suppose immutable state generation $g$ binds cache generation $c$, frontier $F$, and set $V_{full}(F)$. Accepting one row at original position $F$ creates:

$$V_{full}(F+1)=V_{full}(F)\cup\{F\}.$$

The union is exact because position $F$ is new and all old rows remain legal. The successor must bind the new cache generation and parent state generation:

parent:
    cache_generation = c
    state_generation = g
    frontier = F

successor after M5.1 commit:
    cache_generation = c + 1 or another monotonic generation
    parent_state_generation = g
    frontier = F + 1
    visible = V_full(F) union {F}

For a prompt of $P$ accepted tokens, one prefill publication can advance from $F=0$ to $F=P$ with appended positions [0,...,P-1]. Chunked prefill may publish intermediate frontiers, but the final full-cache set must be identical.

At the Loop 3 running shape, persistent payload at frontier $F$ is

$$M_{full}(F)=F\cdot 131072\ \text{bytes}$$

for one sequence. Full cache changes no visibility semantics, so its update does not need a selector or compaction pass. M5.3 may still page or scatter its rows; those physical changes cannot omit a committed position.

Exact Cache Policy Is Not Exact Model Capability

full_cache_exact means:

the cache policy preserves every checkpoint-compatible committed K/V row
and does not alter the attention domain

It does not mean:

the checkpoint was trained at this position range
the positional scheme extrapolates correctly
the model retrieves every visible fact
the model reasons correctly over arbitrary history

Those remain M3.5 and evaluation concerns. Full cache is the semantic reference because the two policies below can be compared against the same committed source positions.

0x04 Sliding-Window State, Rollover, And Position Identity

For a recent width $W>0$, the published sliding-window set is

$$V_{win}(F;W) =\{\max(0,F-W),\ldots,F-1\}.$$

Its cardinality follows directly from the interval endpoints:

$$|V_{win}(F;W)| =F-\max(0,F-W) =\min(F,W).$$

That last equality explains the two phases:

warmup, F < W:
    every committed row still fits
    visible count grows with F

saturated, F >= W:
    visible count stays W
    every accepted append removes one oldest visible row

Worked Prompt And Decode Trace

The executable fixture uses W=3, a four-token prompt, and accepted decode positions 4..7:

The prompt row at position 0 is not described as “evicted from the prior window” because it never belonged to the published F=4 window. It is an omitted source row at the prompt/decode handoff. That distinction matters for release logs and reclamation: omitted_from_source and removed_from_parent_view answer different questions.

Prompt-To-Decode Handoff Has Two Valid Designs

A retrofitted serving policy can materialize the full prompt first, then publish only the last $W$ rows:

prefill source K/V: [0,1,2,3]
published W=3 view: [1,2,3]
release intent to M5.3: [0]

An architecture trained with local attention can instead avoid materializing a globally visible prompt cache. Both can end with the same visible positions, but they do not have the same prefill graph, temporary peak, training contract, or quality interpretation. A runtime must state which design it implements.

The same closure applies to chunked prefill. If chunks partition one prompt, the final state must equal the set derived from the total committed frontier, not a window restarted independently in each chunk.

Logical Position Is Not Ring Slot

A fixed-capacity implementation may place original position $p$ in a physical ring slot such as

$$r_{slot}=p\bmod W.$$

That equation is an address rule, not a positional-encoding rule:

original position p = 6
physical slot       = 6 mod 3 = 0

slot 0 is not semantic position 0

The Loop 2 receipt must retain original position 6 even after slot 0 is reused. M5.3 owns slot lifetime and overwrite safety. M4.5 owns whether the backend can consume the resulting layout. If keys already contain RoPE at their original absolute positions, relabeling packed slots 0..W-1 is invalid.

Architecture-Local Attention And Retrofitted Eviction Are Different

In a model trained with causal local attention, newer hidden states can carry information that flowed through stacked local layers. The architecture’s receptive field may therefore exceed one layer’s window. That does not make an evicted original K/V row directly visible, and it does not justify applying the same window post hoc to a full-attention checkpoint.

For a retrofit:

fact at position 2
published frontier F = 8
W = 3
visible positions = [5,6,7]

direct attention to position 2 is impossible

The Loop 4 delayed-reference fixture records visibility only. It does not claim that a model must fail or succeed, because newer value rows may encode some earlier information. task_quality remains null until the full checkpoint is evaluated.

Payload And Cached-History Read Trajectory

At saturation:

$$M_{win}=\min(F,W)\cdot B_{token,all\ layers}.$$

For the running shape:

Across cached-history frontiers [4,5,6,7], the fixture reads 12 window-position instances, or 1,572,864 logical bytes. Full cache reads 22 instances, or 2,883,584 bytes. These exclude the current row equally from both policies; a fused kernel that attends to the current row adds the same current-row term. They are logical payload ledgers, not measured HBM traffic or latency.

0x05 Attention Sinks And StreamingLLM State Evolution

Recent-only retention removes the first rows once $F>W$. The StreamingLLM paper reports that initial tokens often receive strong attention even when they are not semantically important and that preserving their K/V rows largely recovers the streaming stability lost by recent-only window attention. It then combines those initial sink rows with a rolling recent suffix.

Source: Xiao et al., Efficient Streaming Language Models with Attention Sinks, Figure 1. Complexity and perplexity annotations are source-reported for the paper’s model and setup. The mechanism-level import is the distinction among full cache, recent-only cache, recomputed recent context, and retained sink-plus-recent K/V.

Sink Plus Recent Is A Union, Not An Unconditional Sum

For $K_s$ initial sink positions and recent width $W$, define

$$S_{sink}(F;K_s)=\{0,\ldots,\min(K_s,F)-1\},$$ $$S_{recent}(F;W)=\{\max(0,F-W),\ldots,F-1\}.$$

The visible set is

$$V_{sink+recent}(F)=S_{sink}(F)\cup S_{recent}(F).$$

Therefore:

$$|V_{sink+recent}(F)| =|S_{sink}|+|S_{recent}|-|S_{sink}\cap S_{recent}|.$$

For contiguous initial sinks and a contiguous recent suffix, the overlap size is

$$I(F)=\max\left(0,\min(K_s,F)-\max(0,F-W)\right),$$

so

$$|V_{sink+recent}(F)| =\min(K_s,F)+\min(W,F)-I(F).$$

The shortcut $K_s+W$ is only the saturated, disjoint upper bound. It is wrong during early overlap.

Early Overlap And First Rollover

With K_s=2 and W=3:

The cache first reaches its five-row upper bound at $F=5$. The first irrecoverable middle omission occurs only at $F=6$. A test that begins after saturation misses both behaviors.

Pinned StartRecentKVCache Tensor Trace

The official StreamingLLM implementation is pinned here at commit 2e5042606d69933d88fbf909bd77907456b9b4dd. Its StartRecentKVCache receives model-specific K and V sequence axes. For a cache longer than start_size + recent_size, it constructs each layer’s successor as:

K_next = cat(
    K_old[..., 0:start_size, :],
    K_old[..., seq_len-recent_size:seq_len, :],
    dim=k_seq_dim,
)

V_next = cat(
    V_old[..., 0:start_size, :],
    V_old[..., seq_len-recent_size:seq_len, :],
    dim=v_seq_dim,
)

This is a shape-level summary of the pinned source, not a universal cache API. Llama, MPT, Falcon, and Pythia do not all place sequence on the same tensor axis. The implementation chooses k_seq_dim and v_seq_dim per model family.

Before a multi-token append, evict_for_space(past_key_values, num_coming) keeps the initial rows and only recent_size - num_coming old recent rows. The incoming rows then refill the recent partition:

before reservation:
    start rows K_s
    recent rows W

reserve n incoming rows:
    start rows K_s
    old recent rows W - n
    free capacity n

after accepted append:
    start rows K_s
    recent rows W

This requires n to fit the declared recent partition or a separately defined chunk/reject path. Reserving prompt_length + max_generation_length, as the official chat example does, is a capacity policy; it should not be hidden inside the attention formula.

Original Identity, RoPE State, And Cache-Local Positions

Three position coordinates must remain separate:

p_original   immutable source-token identity used by policy and rollback
r_packed     current row or physical slot in the retained cache
p_attention  position consumed by the checkpoint-compatible attention path

For ordinary post-RoPE absolute keys:

p_attention = p_original
r_packed must not replace it

The pinned StreamingLLM Llama patch uses another explicit contract. It keeps persistent keys in a repositionable pre-RoPE form, concatenates past and new keys, then applies rotary embedding to keys using cache-local positions arange(kv_seq_len). The query receives the bounded position supplied by the patched streaming path. Values remain unpositioned projections.

That mechanism cannot be reproduced by gathering already absolute post-RoPE keys and pretending their packed rows are positions 0..K-1. A runtime must declare which key stage it stores and install a compatible position-shift attention implementation. M5.4 keeps p_original for visibility identity; M4.5 owns execution admission for the changed key-position path.

Retained Cache Versus Sliding-Window Recomputation

Figure 1 distinguishes two bounded-history baselines:

sliding-window recomputation:
    keep recent token IDs
    re-run the W-token local causal window for each incoming token
    refresh positions and all K/V states in that window

retained sink-plus-recent cache:
    project one accepted new row
    reuse retained sink and recent K/V
    apply the declared position-shift path

For four decode steps and W=3, the synthetic work ledger is:

This is not a FLOP or latency benchmark. It explains why the paper labels the recomputation baseline $O(TW^2)$ while retained StreamingLLM attention is $O(TC)$ for bounded cache size $C=K_s+W$. Source-reported speedups remain tied to the paper’s checkpoint, implementation, hardware, and workload.

Cancellation And Resume Need Versioned State

The Loop 4 fixture stages a candidate at original position 6:

published parent:
    cache_generation 103, state_generation 3, frontier 6
    sink+recent [0,1,3,4,5]

unpublished candidate:
    cache_generation 104, state_generation 4, frontier 7
    sink+recent [0,1,4,5,6]

cancel:
    restore generation 103 / state 3
    position 3 remains visible; candidate row 6 is discarded

resume with replacement content at original position 6:
    cache_generation 105, state_generation 5, frontier 7
    logical positions again [0,1,4,5,6]

The staged and resumed sets contain the same logical positions, but their token content and cache generations are not interchangeable. Publishing the staged receipt after cancellation would bind the right positions to stale K/V.

Stability Anchors Are Not Semantic Summaries

At frontier 8, the delayed-reference fixture asks only whether three positions are visible:

Keeping position 0 repairs an attention-sink failure mode. It does not encode the fact at position 2. The paper and official project explicitly state that StreamingLLM does not expand the context window or add long-term memory; it is suited to streaming workloads that rely primarily on recent interactions.

The paper also studies adding a dedicated sink placeholder during pretraining. That is a model-training variant, not an inference-only cache flag. Likewise, softmax-off-by-one is an alternative attention normalization, not a property obtained by retaining first-token K/V. These design choices must not be folded into the retrofit state machine.

Executable Static-Policy Oracle

The source-pinned synthetic evidence is:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    static_retention_fixture.json
    invalid_static_retention_fixtures.json
    static_retention_oracle.py
    static_retention_report.json
    test_loop4_static_retention.py

It validates full, window, and sink-plus-recent sets; prompt handoff; overlap; rollover; cached-history payload; recomputation work; delayed-reference visibility; candidate cancellation; and generation-safe resume. Twenty-seven one-contract mutations reject. It does not reproduce paper perplexity, measure GPU traffic or latency, implement a production cache, or claim model quality.

Loop 5 can now replace position-only selection with Scissorhands, H2O, and Keyformer score state while preserving the same frontier, transaction, position, recoverability, and accuracy contracts.

0x06 Dynamic Retention: Scissorhands, H2O, And Keyformer

Position-only policies know where a token is. Dynamic policies also maintain evidence about how previous queries used it. That sounds like one incremental improvement over a window. It is actually a new state machine: attention must run before the selector can update, the score row must remain attached to the same original K/V row after compaction, and deletion changes what can ever be scored again.

Owner question. Given a fixed per-owner K/V budget, which statistic is allowed to stand in for unknown future relevance, when is that statistic updated, and how does a successful decision publish K/V rows and score state without losing original token identity?

Scissorhands, H2O, and Keyformer answer differently. Scissorhands periodically drops a batch of rows using a low-attention record over a history window. H2O forms an online one-entry successor from accumulated attention. Keyformer accumulates a Gumbel-softmax selection score with a temperature schedule. A generic cumulative_attention.topk() is therefore a useful baseline, but it is not an implementation of all three papers.

0x06A One Dynamic Phase Order, Three Different Selectors

For one layer and one K/V owner, let the published set before step $t$ be $S_{t-1}$. Its tensors and score state are:

$$K_{t-1},V_{t-1}\in \mathbb{R}^{B\times H_{kv}\times |S_{t-1}|\times d_h}, \qquad s_{t-1}\in\mathbb{R}^{H_{kv}\times |S_{t-1}|}.$$

After projecting the accepted current token, the candidate set is

$$C_t=S_{t-1}\cup\{t\}.$$

The common phase order is:

1. append current K/V as an unpublished candidate row
2. compute current attention over every row in C_t
3. update the method-specific score state
4. protect the method's recent partition
5. select the historical partition
6. gather K, V, score, and original-position metadata with the same indices
7. atomically publish successor cache and policy-state generations

Steps 2 and 3 cannot be reversed: the current query supplies the newest score evidence. Steps 5 and 6 cannot use different permutations: a valid K shape with a shifted V or score row is silent corruption. A candidate that is cancelled before step 7 contributes neither a committed K/V row nor committed importance state.

The papers mostly explain single-head or conventional multi-head attention. For GQA, a deployment must additionally declare whether selection belongs to a K/V head, to every mapped query head, or to a reduction across mapped query heads. That is a runtime integration choice, not an omitted constant in the paper formulas.

0x06B Persistence Of Importance Is A Hypothesis

Scissorhands calls a token pivotal for query position $t$ when its attention exceeds a threshold $\alpha$:

$$P_t(\alpha)=\{j:\ a_{t,j}>\alpha\}.$$

Its persistence-of-importance hypothesis says that tokens which become pivotal at a later step are likely to have appeared in an earlier pivotal set. A useful diagnostic is the later-set coverage:

$$R_{a\rightarrow b} = \frac{|P_b\cap (P_a\cup\cdots\cup P_{b-1})|}{|P_b|}.$$

The paper reports high persistence ratios for its tested checkpoints and sequences and gives a theoretical argument under assumptions on the simplified transformer, residual behavior, and trained attention weights. Those results motivate a cache oracle. They do not prove that every future query preserves the same ranking, that every layer behaves alike, or that a low-attention fact will remain irrelevant after a topic shift.

This distinction matters because Scissorhands’ compression statistic is not the pivotal-set threshold above. Algorithm 2 counts low-score events relative to the uniform baseline $1/t$ over a history window:

$$I_j = \sum_{i=t-w}^{t} \mathbf{1}\!\left[a_{i,j}<\frac{1}{t}\right].$$

Large $I_j$ means that row $j$ was repeatedly below the baseline and is a stronger eviction candidate. Pivotal-set analysis explains the hypothesis; the low-score counter drives the published algorithm.

0x06C Scissorhands Compresses In Batches

Scissorhands reserves memory for at most $B$ rows. Ordinary token generation appends rows until the candidate length $n$ exceeds $B$. A compression event then examines a history width $w$, protects the latest $r$ rows, and drops $m$ rows with the largest low-score counts:

def scissorhands_compress(candidate, attention_history, B, w, r, m, t):
    assert len(candidate.positions) > B
    assert 0 < r <= B and 0 < m <= len(candidate.positions) - B + m

    low_count = {position: 0 for position in candidate.positions}
    for row in attention_history[-w:]:
        for position, weight in row.items():
            low_count[position] += int(weight < 1.0 / t)

    recent = candidate.positions[-r:]
    for position in recent:
        low_count[position] = 0       # intended recent protection

    historical = [p for p in candidate.positions if p not in recent]
    drop = sorted(historical, key=lambda p: (-low_count[p], p))[:m]
    keep = [p for p in candidate.positions if p not in drop]

    return gather_k_v_score_position(candidate, keep)

The batch size $m$ is also a frequency/overhead choice. The paper reports using $r=10$, $w=400$, and $m=0.5B$ in its experiments. A large $m$ amortizes the extra history-score work but creates a larger discontinuity at each compression event. A small $m$ approaches one-at-a-time eviction and performs the selector more often. The paper describes two implementation choices: recompute the importance measurements at compression time, or maintain extra importance state during generation to avoid that burst.

Budget allocation is another paper-level choice. Scissorhands distributes a layer budget evenly across heads, then gives later layers more total budget in response to the source-reported persistence profile. That is not equivalent to the uniform per-layer/per-head contract, and it does not make one head’s score state interchangeable with another’s.

0x06D Scissorhands Algorithm 2 Has A Printed Slice Conflict

Algorithm 2 prints:

I[:-r] <- 0     # Keep cache within the recent window

The slice as printed zeroes the historical prefix, while the adjacent comment and Section 4.1 prose say recent tokens are always kept because their importance is not yet known. Under the same algorithm, large low-score counters are the rows removed by Argsort(I)[:-m]; zeroing the historical prefix would therefore protect old rows rather than the recent window.

The executable teaching fixture follows the stated intent:

I[-r:] <- 0     # protect the last r counters

This is an explicit rewrite interpretation, not a claim that the paper source contains the corrected slice. No official Scissorhands repository is pinned in this evidence set, so the article preserves both the printed line and the consistency correction instead of manufacturing implementation authority.

0x06E H2O Is A One-Entry Dynamic Successor

H2O starts from a stricter successor contract. With fixed budget $k$, the next set has the same cardinality and differs from the previous set by at most one row:

$$|S_t|=k, \qquad |S_t\setminus S_{t-1}|\le 1, \qquad S_t=(S_{t-1}\cup\{t\})\setminus\{u\}.$$

The paper’s generic removal rule chooses the $u$ whose removal maximizes a set score:

$$u= \arg\max_{v\in S_{t-1}\cup\{t\}} F_{score}\!\left((S_{t-1}\cup\{t\})\setminus\{v\}\right).$$

For the intuitive additive instantiation, each retained row accumulates current attention:

$$h_{t,j}=h_{t-1,j}+a_{t,j}, \qquad j\in C_t.$$

Keeping a recent partition plus the largest historical $h_{t,j}$ values is the practical heavy/recent policy. The current query attends to the five-row candidate first; only then may one row disappear:

def h2o_append_score_select_publish(parent, current_kv, attention,
                                    K_heavy, K_recent):
    candidate = append_unpublished(parent, current_kv)
    score = align_score_to_candidate(parent.score, candidate.positions)
    score += attention                 # current query evidence

    recent = latest_original_positions(candidate, K_recent)
    history = positions_not_in(candidate, recent)
    heavy = topk(score, history, K_heavy)
    keep = sorted(heavy) + recent

    successor = gather_k_v_score_position(candidate, keep)
    return atomic_publish(parent, successor)

The official real-drop path is pinned to commit ac75c2a8a9e76832b2a4139b9363373b56336bfb, h2o_hf/utils_real_drop/modify_llama.py. Its H2OKVCache_LayerWise accumulates attn_score_cache into hh_score, selects top heavy scores before the recent suffix, gathers K and V with one boolean mask, and gathers hh_score with the same mask. The pin is valuable source evidence; it is not a universal cache ABI, GQA policy, concurrency protocol, or guarantee that every backend executes a physical gather at the same point.

Source: H2O, Figure 3. The upper steps use retained local statistics; the globally recomputed lower alternative is marked infeasible because deleted rows are no longer available.

0x06F What The Dynamic-Submodular Result Does And Does Not Say

H2O frames online eviction as a dynamic submodular problem. Informally, a set function is submodular when adding an item has diminishing marginal value as the existing set grows:

$$A\subseteq B, x\notin B \quad\Longrightarrow\quad F(A\cup\{x\})-F(A) \ge F(B\cup\{x\})-F(B).$$

The paper extends the set function with evolving source state and analyzes a greedy policy under its stated submodularity and approximation assumptions. The result explains why a local greedy construction can be reasonable; it does not prove that arbitrary transformer attention is always submodular, that the simple cumulative-score implementation globally recovers the unknown future, or that a fixed heavy/recent split preserves every task.

Keep three contracts separate:

H2O’s throughput, latency, and quality numbers remain source-reported for the paper’s models, budgets, hardware, and implementation.

0x06G Keyformer Changes The Score Distribution

Deleting keys changes the softmax denominator. If the original logits are $x_j$ and the retained set is $R$, then:

$$p_j=\frac{e^{x_j}}{\sum_{u\in C}e^{x_u}}, \qquad \widetilde p_j=\frac{e^{x_j}}{\sum_{u\in R}e^{x_u}} = \frac{p_j}{1-\sum_{u\in C\setminus R}p_u}.$$

The surviving probabilities are renormalized upward. Their raw-logit order is unchanged for one query, but a cumulative selector now observes a distribution conditioned on its own earlier deletions. That feedback can reinforce an early mistake.

Keyformer introduces a separate key-token score. For a query/key logit $x_{t,j}=q_tk_j^\top/\sqrt{d_h}$, it adds Gumbel noise $\zeta_{t,j}$ and accumulates a temperature-scaled probability:

$$y_{t,j}=x_{t,j}+\zeta_{t,j},$$ $$f_t(j) = f_{t-1}(j) + \frac{\exp(y_{t,j}/\tau_t)} {\sum_{u\in C_t}\exp(y_{t,u}/\tau_t)}.$$

One way to sample standard Gumbel noise is $\zeta=-\log(-\log U)$ for $U\sim\mathrm{Uniform}(0,1)$. The fixture supplies fixed $\zeta$ values so the equation is reproducible; it does not claim to reproduce PyTorch’s random stream. The official MPT path is pinned at commit f327831056a9909b133aaa14a48f2bd0722acede, models/mpt-keyformer-lib/attention_keyformer_new_bias.py. It computes gumbel_softmax for the cache score while retaining ordinary softmax attention for the model output. The selection distribution is not the model’s token sampling distribution and does not replace the actual attention output.

A25 annotated reconstruction of the Keyformer scoring equations. The Chinese callouts were part of the original Transformer Series asset; the equations correspond to the paper’s Gumbel regularization and temperature-scaled score.

0x06H Temperature Is A Selection-State Schedule

Keyformer schedules temperature over an expected generation length $T$:

$$\Delta\tau=\frac{\tau_{end}-\tau_{init}}{T}, \qquad \tau_t=\tau_{init}+t\Delta\tau.$$

The effect is clearest in what one step adds to the score. Subtracting f_{t-1} from both sides of the update in 0x06G leaves step t’s Gumbel-softmax probability. All candidates share its denominator, so it cancels from the odds of any two rows:

$$\delta_t(j)=f_t(j)-f_{t-1}(j), \qquad \frac{\delta_t(i)}{\delta_t(j)} = \exp\!\left(\frac{y_{t,i}-y_{t,j}}{\tau_t}\right).$$

As $\tau_t$ grows, these per-step odds move toward one and the step’s selection distribution flattens: every candidate’s share approaches 1/|C_t|. The paper motivates this as increasing regularization as more context has been discarded. (The paper defines the per-step score and accumulates it in its Algorithm 1; the odds relation is derived here from that score, not quoted from the paper.)

The relation holds for the increments, not for the cumulative scores. The decode step of the fixture in 0x06I shows the difference. At tau_1 = 1.5, positions 3 and 2 have regularized logits y = 1.5 and y = 0.1. Their step contributions are 0.3104 and 0.1221, and their ratio, 2.543, is exactly exp(1.4 / 1.5). Their cumulative scores are 0.4281 and 0.3590, a ratio of only 1.192, because position 2 entered the step with the larger prompt score (0.2370 against 0.1177, a ratio of 0.497).

In general the cumulative odds are a ratio of two sums:

f_t(i)     f_{t-1}(i) + delta_t(i)
------  =  -----------------------
f_t(j)     f_{t-1}(j) + delta_t(j)

With positive terms, such a ratio lies between the odds carried in, f_{t-1}(i) / f_{t-1}(j), and the current-step odds delta_t(i) / delta_t(j): here between 0.497 and 2.543. No constant of proportionality links the cumulative odds to the exponential either. For positions 3 and 2 they are 0.469 times the exponential; for positions 0 and 2, which entered with equal prompt scores, they are 0.846 times it. The cumulative odds equal the exponential when neither row carries an earlier score, as in the fixture’s single-row prompt step, where the cumulative scores are the step probabilities. Once earlier scores exist, cross-multiplying the ratio of sums shows that a pair matches only if its carried-in odds already equal its step odds. Two rows tied both in their earlier score and in this step’s regularized logit match trivially, with odds of 1 on both sides; positions 0 and 2 share their earlier score but not their logit, so they do not match.

The pinned MPT code departs from the single-row picture in a further way. On every pass, prompt and decode alike, it sums the Gumbel-softmax rows over the batch and over the query positions (gumbel_score.sum(0).sum(1)), keeping one score row per head. The relation then holds for each sequence’s query row, not for that summed increment. Only at decode with batch size 1, where each head’s sum has a single row, does it hold for the code’s own increment.

The temperature therefore shapes each step’s contribution, not the accumulated ranking directly. The gap between two cumulative scores changes by delta_t(i) - delta_t(j), which tends to zero as tau_t grows because every contribution approaches 1/|C_t|. With the fixture’s regularized decode logits y, that change for positions 3 and 2 would be 0.2799 at tau = 1, is 0.1884 at the scheduled tau_1 = 1.5, and would be 0.1414 at tau_end = 2. In the limit a step adds almost the same amount to every row and leaves the accumulated ranking nearly as it was, while the history it inherits keeps its full weight.

The schedule introduces a deployment coordinate: actual generation can stop before $T$ or run beyond it. A runtime must declare whether temperature clamps at $\tau_{end}$, continues increasing, or is recomputed when the generation budget changes.

At prompt processing, Keyformer scores every prompt row, protects the latest $w$ rows, and chooses $k-w$ historical key rows. At decode, one new row enters, the recent window shifts, cumulative score is updated only for surviving rows, and $k-w$ key rows are selected from a candidate key window of size $k+1-w$:

def keyformer_score_select_publish(parent, current_kv, logits, gumbel,
                                   k, w, tau):
    candidate = append_unpublished(parent, current_kv)
    probability = softmax((logits + gumbel) / tau)
    score = align_score_to_candidate(parent.score, candidate.positions)
    score += probability

    recent = latest_original_positions(candidate, w)
    history = positions_not_in(candidate, recent)
    key = topk(score, history, k - w)
    keep = sorted(key) + recent

    return atomic_publish(
        parent,
        gather_k_v_score_position(candidate, keep),
    )

A25 annotated composite based on Keyformer Figure 6. It preserves the original article’s prompt-to-decode explanation and makes the separate ordinary-attention and cache-score paths visible.

0x06I One Fixture, Three Divergent Retained Sets

The Loop 5 fixture has one layer, one K/V head, $d_h=4$, BF16 K/V, budget $k=4$, and recent width two. It is a mechanism oracle, not a quality or runtime benchmark.

Scissorhands sees six candidate rows and three attention-history rows. With $1/t=1/6$, the low-score counters are:

position:       0  1  2  3  4  5
low count I:    3  0  1  2  1  1
protected:                  4  5
drop m=2:       0        3
publish:           1  2     4  5

H2O starts from [0,1,2,3] with score [0.4,0.3,0.2,0.1]:

Keyformer first selects prompt rows at $\tau_0=1$, then appends position 5 at $\tau_1=1.5$:

At decode, position 3’s current regularized probability is approximately 0.3104; its cumulative score becomes 0.4281, overtaking position 2’s 0.3590. This transition is caused by Keyformer’s declared score, not by a generic raw-attention top-k.

0x06J Score State Must Follow Row Identity

The selector adds persistent state; it does not eliminate the Loop 2 receipt:

per retained row:
    original token position
    packed row or physical slot
    K/V owner (layer, kv_head)
    method score
    score-generation provenance

per published successor:
    request generation
    parent and successor cache generations
    parent and successor policy-state generations
    exclusive committed frontier
    selected original-position list

When rows [0,2,3,4,5] are compacted to [0,3,4,5], Keyformer score state must gather indices [0,2,3,4] in the candidate’s packed coordinate system. Gathering score by original position after K/V has already been compacted, or gathering K and V with one mask but score with another, can retain correct shapes and wrong ownership.

For the fixture, each K/V row costs:

$$B_{row}=L\,H_{kv}\,2d_hb=1\cdot1\cdot2\cdot4\cdot2=16\ \text{bytes}.$$

The six-row source is 96 bytes, the four-row published state is 64 bytes, and a five-row append-before-evict candidate peaks at 80 bytes. Four FP32 score values add 16 bytes per owner. These are logical payloads; allocator metadata, workspace, kernels, and physical traffic remain separate ledgers.

0x06K Permanent Deletion Is The Shared Failure

Different selectors do not remove the same rows, but all three make omitted raw K/V unavailable:

The hypothetical weights do not claim that a model would actually assign those values. They expose the causal failure: an evicted row receives no future score because it is absent from the future softmax. Recovery requires recomputing the row, retaining it in a backing tier, or using a different representation. Those mechanisms belong to M5.5 or later representation sections, not to irreversible eviction.

Additional failure surfaces are method-specific:

Scissorhands:
    batch compression can discard several rows at once;
    history width and compression frequency can miss a transition;
    paper/code intent must resolve the recent-slice discrepancy.

H2O:
    older rows have more opportunities to accumulate score;
    one-entry greedy updates cannot recover deleted evidence;
    the theoretical guarantee is conditional on the formulated objective.

Keyformer:
    the score depends on random noise and temperature schedule;
    a generation-length mismatch changes the schedule;
    survivor-only accumulation remains conditioned on earlier deletion.

0x06L Paper And Code Contracts Side By Side

The design alternatives are substantive. Recomputing score at compression time trades burst compute for less persistent metadata. Maintaining score each step trades metadata and update work for a cheaper eviction event. Deterministic attention accumulation is easier to replay than Keyformer’s stochastic score, but fixed noise during testing is not a production randomness policy. Batch drop reduces selection frequency; one-entry replacement reduces state jumps.

None of the papers removes runtime integration work. Continuous batching needs request-local score state. Beam search needs score/cache reordering. Prefix sharing must not share mutable query-dependent scores across requests. Speculative branches need generation-fenced score updates and rollback. Physical row release and compaction remain M5.3 operations, while scheduling admission and fairness remain M5.2.

0x06M Executable Dynamic-Retention Oracle

The source-pinned synthetic evidence is:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    dynamic_retention_fixture.json
    invalid_dynamic_retention_fixtures.json
    dynamic_retention_oracle.py
    dynamic_retention_report.json
    test_loop5_dynamic_retention.py

It validates paper-specific score rules, update order, budget partitions, original-position retention, H2O and Keyformer code pins, temperature, fixed noise, generation progression, payload accounting, and two future-relevance probes. Forty one-contract mutations reject. The oracle does not reproduce paper benchmarks, PyTorch random-number streams, production cache code, GPU traffic, latency, or model quality.

0x07 Prompt Compression: SnapKV

Prompt compression is not decode-time eviction. It makes one lossy decision after the complete prompt has been processed and before the first generated token is decoded.

Owner question. Given a fully materialized prompt cache, how does SnapKV derive one bounded prompt view from the final prompt queries, preserve the source positions of every gathered row, and hand that view to decode without pretending that the omitted prompt can be recovered exactly?

This section follows the SnapKV paper and the official implementation pinned to commit e216ddc84c5bd210378cbdbbba12ba02102aa640. The executable fixture uses a different, explicitly labeled GQA ownership choice so that persistent K/V is not silently duplicated across query heads.

0x07A The Phase Contract Comes Before The Selector

Let the prompt contain T_prompt tokens and let the final W_obs prompt tokens form the observation window. The earlier prefix has length:

L_prefix = T_prompt - W_obs

SnapKV first performs the ordinary full prompt forward pass:

source K/V shape: [B, L, H_kv, T_prompt, d_h]
prompt frontier:   F_prompt = T_prompt       # exclusive

Only then can it read attention from the observation queries and derive a compressed successor. This ordering matters:

full prompt prefill
    -> observation attention exists
    -> score and select prefix rows
    -> gather selected prefix plus complete observation suffix
    -> publish compressed prompt cache
    -> first decode step

The method can lower persistent decode state, but it does not remove the full prompt prefill or its peak working state. The SnapKV paper also states this limitation: the method targets the generation stage and does not make a model capable of understanding a context length that the original prompt pass cannot handle.

If the compressed prompt capacity is C_prompt, then the selected-prefix budget is:

K_select = C_prompt - W_obs

required:
    W_obs > 0
    C_prompt > W_obs
    0 <= K_select <= L_prefix

The last condition is not an extra restriction on which prompts are accepted. Because K_select - L_prefix = C_prompt - T_prompt, it holds exactly when T_prompt >= C_prompt, which is the only case in which selection runs; a shorter prompt keeps its full cache (0x07F).

The observation suffix is protected. It is not part of the top-k competition. The paper’s definitions also express a compression rate p through floor((1-p) * L_prefix); the pinned implementation instead exposes a maximum prompt capacity and selects C_prompt - W_obs prefix rows. These are two ways to parameterize the same selected-prefix count, not two different selectors.

0x07B From Observation Attention To Prefix Votes

For one layer and one query attention head, the observation queries attend to the full prompt:

Q_obs: [W_obs, d_h]
K:     [T_prompt, d_h]

S_obs = Q_obs @ K^T / sqrt(d_h)
A_obs = softmax(mask_causal(S_obs), axis=key_position)

The causal mask is still required inside the observation window. Its first row must not see later observation tokens. After softmax over every key visible to each observation query, SnapKV discards the observation-key columns from the selection score and sums only the earlier prefix columns:

A_prefix = A_obs[:, 0:L_prefix]       # [W_obs, L_prefix]
vote[j] = sum_i A_prefix[i, j]        # [L_prefix]

That order is important. Normalizing only over prefix columns would erase the amount of attention consumed by recent observation tokens and produce a different score. Summation answers a concrete question: across the protected prompt-end queries, how much normalized attention did prefix position j receive?

0x07C Pooling Means Neighborhood Credit, Not K/V Merging

Selecting the largest raw vote picks isolated peaks. SnapKV applies a stride-one pooling operator before top-k so nearby prefix positions can share credit:

pooled[j] = Pool1D(vote)[j]
selected = TopK(pooled, K_select)

The pinned implementation supports average or maximum pooling. With odd kernel width r, padding (r-1)/2, and stride one, the pooled vector keeps the same length as the prefix. The article-local fixture uses average pooling with r=3 and zero padding:

pooled[j] = (vote[j-1] + vote[j] + vote[j+1]) / 3

Out-of-range terms are zero because the pinned PyTorch AvgPool1d call counts padding in the denominator. The implementation detail changes boundary scores; it cannot be omitted from a reproducible selector.

Pooling scores is not semantic clustering and it does not merge K/V vectors. Once positions are selected, their original K and V rows are gathered unchanged. kernel_size=1 removes neighborhood smoothing. Average and maximum pooling are alternative paper variants; neither should be declared universally better without a pinned checkpoint, task, and budget.

0x07D GQA Forces A Selection-Owner Decision

The paper describes head-wise selection. In MHA, each query head owns one K/V head, so the phrase is unambiguous. In GQA, several query heads consume the same persistent K/V head:

q0, q1 -> kv0
q2, q3 -> kv1

There are at least three implementable contracts:

The official Llama integration applies RoPE, calls repeat_kv, and then invokes SnapKV update logic. Its selector therefore works on query-head-expanded K/V. That is a valid pinned implementation choice, not a universal GQA cache ABI.

The local Loop 6 fixture instead uses this owner-preserving teaching rule:

grouped_vote[l, h_kv, :] =
    mean(vote[l, h_q, :] for h_q mapped to h_kv)

This reduction is rewrite interpretation, not a claim about canonical SnapKV. A production implementation must pin one of these contracts before it can state cache bytes, top-k cardinality, or kernel layout.

0x07E A Worked Tensor Trace

Use a deliberately small prompt:

B=1, L=2, H_q=4, H_kv=2, d_h=3
T_prompt=6, W_obs=2, C_prompt=4
prefix positions:      [0, 1, 2, 3]
observation positions: [4, 5]
K_select=2

For layer 0, K/V head 0, query heads 0 and 1 produce the grouped prefix vote:

vote = [0.175, 0.125, 1.000, 0.150]

Average pooling with kernel 3 gives:

pooled[0] = (0     + 0.175 + 0.125) / 3 = 0.100000
pooled[1] = (0.175 + 0.125 + 1.000) / 3 = 0.433333
pooled[2] = (0.125 + 1.000 + 0.150) / 3 = 0.425000
pooled[3] = (1.000 + 0.150 + 0    ) / 3 = 0.383333

Top-2 therefore selects prefix positions [1,2] in descending score order. Appending the complete observation suffix produces packed original positions [1,2,4,5].

The four owner results are:

The packed rows are score-ordered, not sorted by source position. Every row therefore needs an explicit original-position map. Packed row 0 is not token position 0, and M5.3 still owns the physical page, slot, or contiguous-address mapping.

The original paper figure provides the higher-level view: selection is per head, clustered prefix positions are retained, and the observation window is kept in full.

Source: SnapKV: LLM Knows What You are Looking for Before Generation, Figure 1. Orange marks selected prefix neighborhoods; observation-window rows remain in the generation cache. The figure describes the paper’s method and does not guarantee sufficiency for every later query.

0x07F The Pinned Code Path And Its Alternatives

The official implementation’s core sequence is:

1. require prompt K and Q sequence lengths to match
2. if prompt is below the configured capacity, return the full prompt cache
3. form attention from the last W_obs queries to all prompt keys
4. apply the causal observation-window mask
5. softmax over visible prompt keys
6. sum attention to prefix columns across observation queries
7. apply avg-pool or max-pool with stride one
8. take top C_prompt-W_obs prefix positions
9. gather prefix K/V and concatenate the full observation suffix

The short-prompt branch is an algorithmic boundary, not just an optimization:

if T_prompt < C_prompt:
    compressed cache = full prompt cache

The pinned Llama patch tracks the logical sequence length separately from the shortened physical cache length. That separation is required because RoPE and the causal frontier still refer to original sequence progress, not packed row count.

0x07G Materialization Is A Source-Position Gather

For each selected owner (l,h_kv), materialization gathers K and V with the same source indices and then appends the observation suffix:

selected_prefix = topk(pooled_score, K_select)
packed_positions = selected_prefix ++ range(L_prefix, T_prompt)

K_packed = gather(K_source, packed_positions)
V_packed = gather(V_source, packed_positions)

The executable fixture makes wrong-row gathers visible with symbolic source values:

K[l,h,p,c] = 1000*l + 100*h + 10*p + c
V[l,h,p,c] = 5000 + 1000*l + 100*h + 10*p + c

For (layer=0, kv_head=0) and packed positions [1,2,4,5], the gathered K rows must be:

[[10,11,12], [20,21,22], [40,41,42], [50,51,52]]

and V must be:

[[5010,5011,5012], [5020,5021,5022],
 [5040,5041,5042], [5050,5051,5052]]

Using the selected indices for K but a sorted or shifted index list for V would still produce shape-compatible tensors while corrupting attention. The oracle checks every scalar and rejects duplicate selected prefix positions, selected indices inside the observation suffix, a missing observation row, duplicate packed positions, and K/V receipt drift.

0x07H Compression Publishes A New Generation

The source prompt cache and compressed prompt cache are different immutable states. A successful compression receipt records at least:

request_id, request_generation
source_cache_generation
compressed_cache_generation
policy_generation
parent_state_generation
compressed_state_generation
prompt_frontier
selected owner count
rows per owner
original-position map presence
publication status

The Loop 6 fixture publishes:

request:             request-snapkv-a / generation 3
source cache:        generation 200
compressed cache:    generation 201
policy:              generation 60
policy state:        generation 10 -> 11
prompt frontier:     6 exclusive
owners:              2 layers * 2 K/V heads = 4
rows per owner:      4

The receipt also states that the full source prompt was materialized before compression and that exact prefix reuse is false. M5.1 owns request/cache identity and transaction publication. M5.3 owns the physical packed mapping. M5.4 owns which source positions the compressed visibility state contains.

Cancellation before atomic publication must leave generations 200 and 10 visible. Publishing some layers under generation 201 while other layers still read generation 200 is not a partially useful cache; it is an invalid cross-layer state.

0x07I Decode Appends A Tail; It Does Not Re-run Prompt Selection

The pinned integration compresses during prompt processing and then appends generated K/V rows. It does not recompute the observation-window selector at every decode step:

compressed prompt positions for owner (0,0): [1,2,4,5]
first generated original position:            6
after append:                                  [1,2,4,5,6]

All four fixture owners append original position 6; the compressed prompt state generation remains 11 because the prompt selection did not change. The cache frontier advances from 6 to 7 through the ordinary M5.1 append transaction.

After g accepted generated tokens, an append-only SnapKV integration stores:

rows per owner = C_prompt + g

So SnapKV bounds the retained prompt contribution; it does not make the total cache permanently fixed-size. A separate generated-token eviction policy could bound the tail, but that would be a composition of policies and needs its own state, quality, and rollback contract.

0x07J Unique-Owner And Expanded-Head Byte Ledgers

For owner-preserving K/V storage, the logical prompt payload is:

M_unique = B * L * H_kv * T_kept * 2 * d_h * bytes

The factor 2 is K plus V. For the fixture:

full prompt:
    1 * 2 * 2 * 6 * 2 * 3 * 2 = 288 bytes

compressed prompt:
    1 * 2 * 2 * 4 * 2 * 3 * 2 = 192 bytes

logical payload reduction:
    288 / 192 = 1.5x

If the runtime persists query-head-expanded K/V after repeat_kv, replace H_kv=2 by H_q=4: full and compressed prompt payload become 576 and 384 bytes. This is why the GQA selection owner cannot remain implicit.

The fixture also accounts for a 64-byte owner-position map, 20 bytes of owner offsets, and a 48-byte unique-owner K/V append for one generated token. These are logical payload and minimal metadata counts, not allocator residency, HBM traffic, kernel workspace, or latency measurements.

0x07K This Is Not Exact Prefix Reuse

Exact prefix reuse says:

same source tokens + same forward contract
    -> reuse every matching prefix K/V row

SnapKV says:

prompt-end attention observation
    -> predict a subset of prefix rows likely to matter during generation

A compressed cache therefore must not enter M5.3’s exact-prefix index under the source prompt’s ordinary cache identity. It needs policy identity, selection state, source generation, and original-position metadata. Reusing that lossy cache for another request is valid only under an explicitly compatible approximate-reuse contract; ordinary exact prefix matching is insufficient.

0x07L The Accuracy Boundary Is Delayed, Head-Local, And Task-Dependent

In the fixture, original prompt position 3 survives only for layer 1, K/V head 0. It is absent from the other three owners. This proves a visibility fact, not a quality result:

visible in one owner != globally preserved fact
omitted in one owner != measured task failure
task_quality = null

The difficult failures are delayed:

• a later user turn asks about a detail that the original observation window did not emphasize;
• exact-copy generation needs an identifier or delimiter that received little pooled score;
• averaging mapped query heads suppresses a query-head-specific peak;
• max pooling preserves that peak but over-selects a noisy neighborhood;
• top-k ties resolve differently across runtimes unless the tie contract is pinned;
• score-ordered packing loses original position if the map is omitted;
• compressed prompt rows are incorrectly advertised as exact reusable K/V;
• a short prompt is compressed even though the pinned implementation would preserve it in full.

No local selector test can establish downstream model quality. Release evidence must measure the real checkpoint and generation policy across retrieval, long-document QA, exact-copy, summarization, conversation follow-up, and adversarial delayed-reference slices.

0x07M Source-Reported Runtime Results And Local Evidence

The SnapKV paper reports lower generation memory and latency under its models, hardware, prompt lengths, budgets, and Hugging Face integration. Those are source-reported results, not local measurements.

Source: SnapKV, decoding-latency experiment. The measurements use the paper’s A100 80GB setup and named implementations. OOM points are capacity outcomes for that setup; they do not establish the same limit or speedup for another checkpoint, backend, budget, or request mix.

The plot combines two effects: fewer retained prompt rows can reduce per-token decode reads, and lower residency can make a batch/length combination fit. It does not test the delayed-reference failures above.

The article-local evidence is deliberately narrower:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    snapkv_fixture.json
    invalid_snapkv_fixtures.json
    snapkv_oracle.py
    snapkv_report.json
    test_loop6_snapkv.py

The deterministic oracle reproduces observation vote aggregation, GQA owner reduction, average pooling, top-k, source-position gathers, a versioned compression receipt, byte ledgers, and the first decode append. Thirty-one one-contract mutations reject. It does not run the model, reproduce the paper’s benchmark, measure GPU traffic or latency, implement a production cache, or claim task quality.

Loop 5 above owns decode-time Scissorhands, H2O, and Keyformer state evolution; SnapKV’s prompt receipt does not substitute for any of their mutable decode score states. Loop 7 can build on this prompt-compression receipt to make layer and head budgets heterogeneous without reopening SnapKV’s phase, position, or GQA ownership contracts.

0x08 Layer/Head Allocation: From One Budget To A Ragged Runtime

SnapKV answers a selection question after prompt prefill:

given one owner and one budget K, which prompt rows survive?

This section changes the question without changing that selection contract:

allocation:
    how many rows may each layer and persistent K/V head retain?

selection:
    which rows fill each assigned budget?

The distinction is operational. A selector can be identical while the budget schedule changes from uniform to pyramidal, sensitivity-aware, or calibrated per head. Conversely, two methods can assign the same number of rows and still retain different positions. Equal totals do not imply equal visible sets, local error, latency, or task quality.

0x08A One Allocation Manifest, Then One Selection Receipt

Loop 7 starts from Loop 6’s full-prompt observation receipt. It does not score a partially compressed cache and pretend that the result came from the original prompt. The phase order is immutable:

1. materialize the full prompt K/V
2. derive observation scores from that full prompt
3. allocate a prefix budget to each layer
4. allocate each layer budget to persistent K/V-head owners
5. select original prefix rows within every owner
6. append the protected observation suffix outside the prefix budget
7. publish one cross-layer generation-stamped allocation receipt

An allocation manifest therefore needs more than K_total:

request_id, request_generation
source_cache_generation, allocated_cache_generation
policy_generation, parent_state_generation, allocated_state_generation
prompt_frontier and original-position convention
layer count, query-head count, persistent K/V-head count, head dimension
query-head -> K/V-head map
prefix length and protected observation positions
real-valued targets, integer budgets, floors, caps, and tie rule
per-owner selected original positions
ragged lengths and cumulative offsets

The persistent owner is the K/V head, not every expanded query head. For GQA, several query heads can vote for one K/V owner, but charging storage to all of those query heads would count duplicated views rather than unique cache state.

With owner-specific prefix budgets K_lh and protected-window counts W_lh, the logical unique K/V payload is

$$M_{alloc} = B\,2d_hb_s \sum_{l=1}^{L}\sum_{h=1}^{H_{kv}} \left(K_{l,h}+W_{l,h}\right).$$

Renderer-safe form:

M_alloc =
    B * 2 * d_h * scalar_bytes
    * sum_over_layers_and_kv_heads(K_lh + W_lh)

K_lh is the selected prefix budget. W_lh is charged separately because a SnapKV-style observation suffix is protected rather than competing in prefix top-k. Position maps, ragged offsets, allocator metadata, transaction peak, and kernel workspace are separate ledgers.

0x08B PyramidKV: Derive The Layer Schedule

PyramidKV begins with an empirical observation: in the paper’s evaluated models and retrieval-augmented prompts, lower layers often show broader attention while upper layers concentrate more strongly. The observation is a calibration result, not an architectural theorem.

Source: Cai et al., PyramidKV, Figure 2. The figure motivates a layer schedule for the evaluated model and workload. It does not prove that every checkpoint, head, prompt, or later query follows a monotone funnel.

Let m be the number of layers, K_total the budget summed across those layers, and beta > 0 the top-to-bottom shaping parameter. The paper sets the upper layer endpoint to

$$k_{top}=\frac{K_{total}}{\beta m}.$$

For a linear schedule to sum to K_total, its endpoint average must be K_total / m. Therefore

$$\frac{k_{bottom}+k_{top}}{2}=\frac{K_{total}}{m},$$

which gives

$$k_{bottom}=\frac{2K_{total}}{m}-k_{top}.$$

Linearly interpolating from bottom layer 0 to top layer m-1 yields

$$k_l = k_{bottom} - \frac{l}{m-1}\left(k_{bottom}-k_{top}\right).$$

The arithmetic-series identity then verifies conservation:

$$\sum_{l=0}^{m-1}k_l = m\frac{k_{bottom}+k_{top}}{2} =K_{total}.$$

For m=3, K_total=12, and beta=2:

k_top    = 12 / (2 * 3) = 2
k_bottom = 2 * 12 / 3 - 2 = 6
real layer targets = [6, 4, 2]

Source: Cai et al., PyramidKV, Figure 1. Blue cells represent retained positions. The final panel changes retained length by layer; it does not by itself establish accuracy or runtime speed at an arbitrary budget.

The paper formula is real-valued, while a cache stores integer rows. The pinned official implementation at commit 94255b6 uses integer division to form layer capacities in PyramidKVCluster. That is a concrete implementation choice, not proof that every (K_total, m, beta) combination conserves the requested total after flooring.

A deployment needs a declared apportionment rule:

def pyramid_layer_budgets(total, layers, beta, floors, caps):
    top = total / (beta * layers)
    bottom = 2 * total / layers - top
    targets = [
        bottom - l * (bottom - top) / (layers - 1)
        for l in range(layers)
    ]
    return apportion_exact_integer_budget(targets, total, floors, caps)

This article’s oracle floors bounded targets, then assigns residual rows by largest remainder with stable owner-index ties. It checks exact conservation, floors, and caps. The rule is a rewrite implementation contract, not a claim that PyramidKV prescribes this exact integer tie-break.

One more unit distinction matters. The paper and code can apply a layer capacity to each head. The Loop 7 teaching fixture instead treats [6,4,2] as aggregate prefix rows per layer and then distributes each layer total across its two persistent K/V heads. That composition is intentionally explicit; it must not be cited as PyramidKV’s paper algorithm.

0x08C PyramidInfer Is Not Merely Another Fixed Pyramid

PyramidInfer uses a different contract. It identifies Pivotal Context (PvC) from inter-layer consistency, keeps a top proportion of crucial context, and updates the retained context as generation proceeds. Its layer-dependent proportion can decay because later layers are expected to need fewer positions. The method can also remove work during prefill, rather than only publishing a smaller prompt cache after a full prefill.

PyramidKV teaching boundary:
    assign layer capacity -> rank positions -> retain rows

PyramidInfer paper boundary:
    estimate layerwise pivotal context
    -> prune context participating in later computation
    -> update pivotal context during generation

The official PyramidInfer source is pinned here at commit 61ac641. Its model patch is evidence for one research implementation, not a portable cache ABI. Treating PyramidInfer as K_l = decreasing_sequence[l] would erase its score, phase, and compute contracts.

0x08D ZigZagKV: Infer Layer Budgets Instead Of Assuming A Funnel

ZigZagKV asks how much budget each layer appears to need. For each layer it defines a Minimum Budget of Attention (MBA): the smallest retained count needed to preserve a target share of attention mass, 90% in the paper’s analysis. It then averages this signal into a layer quantity, LMBA, and also studies a Minimum Budget of Output (LMBO) based on hidden-output similarity.

Source-local annotated figure from A25, based on ZigZagKV. MBA is attention-mass based; LMBO is output-similarity based. Neither is a full-model task-quality guarantee.

Normalize positive layer sensitivity values:

$$u_l=\frac{LMBA_l}{\sum_i LMBA_i}.$$

If B_mean is the desired mean layer budget across L layers, naive proportional allocation is

$$\widehat B_l=B_{mean}L u_l.$$

This conserves B_mean L, but it can starve a low-scoring layer. Introduce a minimum B_min and allocate only the residual above that floor:

$$B_l = B_{min} + \left(B_{mean}-B_{min}\right)L u_l.$$

The derivation is conservation by construction:

$$\sum_l B_l = LB_{min} + \left(B_{mean}-B_{min}\right)L\sum_lu_l =LB_{mean}.$$

Source-local annotated figure from A25, based on ZigZagKV. The minimum avoids zero-budget layers, while the sensitivity profile redistributes the remaining pool.

For LMBA=[6,4,2], B_mean=4, and B_min=1:

u = [1/2, 1/3, 1/6]
real targets = [5.5, 4.0, 2.5]
integer targets under the declared tie rule = [6, 4, 2]

Unlike a fixed pyramid, calibrated targets need not decrease monotonically. They are profile-, model-, prompt-, and metric-dependent. A retrieval-derived profile may also understate heads or layers needed for reasoning.

0x08E Ada-KV: Why Global Top-B Produces Head Budgets

Ada-KV moves allocation inside a layer. Write the multi-head output as

$$y=\sum_{i=1}^{H}\sum_{j=1}^{T}A_i^j V_i^j W_i^O,$$

where A_i^j is attention from head i to position j, and W_i^O is that head’s output-projection slice. Let I_i^j be one when the row is retained. The compressed output uses only rows selected by I.

Under the paper’s bounded-row-norm assumption, the local L1 output difference has the upper bound

$$\lVert y-\widehat y\rVert_1 \le 2HC - 2C\sum_{i=1}^{H}\sum_{j=1}^{T}I_i^jA_i^j.$$

For fixed H and C, minimizing this upper bound is equivalent to maximizing the retained attention mass. With a layer budget B, concatenate all head-position scores and select the global top B. The number of winners from each head becomes that head’s raw budget.

def adaptive_head_budgets(score_by_kv_head, layer_budget):
    candidates = []
    for kv_head, scores in enumerate(score_by_kv_head):
        for original_position, score in enumerate(scores):
            candidates.append((score, kv_head, original_position))

    winners = sorted(
        candidates,
        key=lambda x: (-x[0], x[1], x[2]),
    )[:layer_budget]

    counts = [0] * len(score_by_kv_head)
    for _, kv_head, _ in winners:
        counts[kv_head] += 1
    return counts

This proves an ordering result for the stated local bound. It does not prove that attention mass orders downstream task quality, that the bounded-norm assumption is tight, or that one observation transfers to later queries.

The paper adds a uniform-allocation safeguard. If B_i* is the global-top-B count, its adjusted real target is

$$\widetilde B_i = \alpha B_i^* + (1-\alpha)\frac{B}{H}.$$

The paper reports alpha=0.2 as its default. The deterministic fixture uses alpha=0.75 only to make redistribution visible. It then imposes a one-row deployment floor and exact integer conservation. Those floor and apportionment steps are article-local runtime policy, not a silent change to the paper.

def apportion_exact_integer_budget(targets, total, floors, caps):
    budget = [
        min(cap, max(floor, int(target // 1)))
        for target, floor, cap in zip(targets, floors, caps)
    ]
    while sum(budget) < total:
        eligible = [i for i in range(len(budget)) if budget[i] < caps[i]]
        i = max(eligible, key=lambda j: (targets[j] - budget[j], -j))
        budget[i] += 1
    while sum(budget) > total:
        eligible = [i for i in range(len(budget)) if budget[i] > floors[i]]
        i = max(eligible, key=lambda j: (budget[j] - targets[j], j))
        budget[i] -= 1
    return budget

The official Ada-KV implementation is pinned at commit 04497ab. Its AdaptiveSnapKVCluster uses rounding and records ragged head lengths and cumulative offsets. Paper equations, research-code rounding, and a production integer reconciliation policy are therefore three distinct contracts.

0x08F Worked Tensor Trace: Three Layers, Two K/V Heads

The executable fixture uses this immutable source:

K, V source shape per layer: [B=1, H_kv=2, T=8, d_h=3]
query heads: 4, mapped [0,0,1,1] -> two persistent K/V heads
prefix original positions: [0,1,2,3,4,5]
protected observation positions: [6,7]
layer prefix totals: [6,4,2]

Ada-style global ranking within each layer yields raw head counts [[2,4],[3,1],[2,0]]. The declared safeguard, one-row floor, and integer reconciliation publish:

integer K_lh = [[2,4], [3,1], [1,1]]
sum K_lh by layer = [6,4,2]
sum all prefix budgets = 12

Each owner selects prefix rows using its own scores, then appends protected positions [6,7]:

The packed lengths are [4,6,5,3,3,3]; cumulative owner offsets are [0,4,10,15,18,21,24]. A row is gathered by (layer, kv_head, original_position) and K/V use the same index. Original positions are not renumbered to packed rows.

def publish_allocated_prompt(source_k, source_v, budgets, scores, protected):
    packed = []
    offsets = [0]
    for layer in range(len(budgets)):
        for kv_head in range(len(budgets[layer])):
            prefix = stable_topk_positions(scores[layer][kv_head],
                                           budgets[layer][kv_head])
            positions = sorted(prefix) + protected
            packed.append({
                "layer": layer,
                "kv_head": kv_head,
                "original_positions": positions,
                "K": gather(source_k[layer][kv_head], positions),
                "V": gather(source_v[layer][kv_head], positions),
            })
            offsets.append(offsets[-1] + len(positions))
    return publish_one_generation(packed, offsets)

At BF16, one unique K/V row is 2 * d_h * 2 = 12 bytes in the toy tensor. The full prompt payload is 3 * 2 * 8 * 12 = 576 bytes. The allocated receipt contains 12 selected prefix rows plus 12 protected rows, or 288 bytes. Its position map is another 24 * 4 = 96 bytes and seven 32-bit cumulative offsets add 28 bytes. These are logical payloads, not allocator, GPU-traffic, latency, or production-kernel measurements.

Article reconstruction. The same 12-row prefix total is first distributed to layers, then K/V heads. Protected observation rows are charged separately. A HeadKV-style calibrated alternative can produce different layer totals under the same global total.

0x08G HeadKV: Allocate Across All Layer/Head Owners

Ada-KV redistributes a fixed layer budget among heads in that layer. HeadKV calibrates head importance across the complete (layer, head) owner set and can therefore move rows between layers as well as between heads. Its importance profiles can use retrieval-only or retrieval-plus-reasoning signals.

For mean per-head budget b, reserve a common base and distribute a pool using normalized importance S_h:

$$B_{pool}=\frac{b}{\beta}LH, \qquad b_h=\left(b-\frac{b}{\beta}\right)+S_hB_{pool}.$$

The sum is bLH because sum_h S_h=1. In the fixture, normalized importances [.05,.35,.25,.10,.20,.05], b=2, and beta=2 produce real targets [1.3,3.1,2.5,1.6,2.2,1.3]. Exact integer apportionment gives [1,3,3,2,2,1], whose layer totals are [4,5,3], not PyramidKV’s [6,4,2].

The official HeadKV source is pinned at commit 0862a09. Its JSON importance profiles and rounding are checkpoint/workload artifacts. They are not universal head importance, and rounding still requires a runtime conservation check.

0x08H FastGen: Choose A Policy Family Per Head

Allocation need not mean only a scalar K_lh. FastGen profiles each head after the prompt and chooses the lowest-memory policy whose retained attention mass meets a recovery threshold. The policy family includes special-token, punctuation, local, frequency/heavy-hitter, and full-cache behavior; hybrid policies can union patterns.

Source-local figure from A25, based on FastGen. The patterns motivate per-head policy selection; they are not permanent semantic labels guaranteed for every request.

def fastgen_profile_head(attention_mass, policy_masks, recovery_threshold):
    candidates = []
    for policy_name, positions in policy_masks.items():
        recovery = sum(attention_mass[p] for p in positions)
        candidates.append((len(positions), policy_name, recovery, positions))
    feasible = [row for row in candidates if row[2] >= recovery_threshold]
    if not feasible:
        full = policy_masks["full"]
        return (len(full), "full", 1.0, full)
    return min(feasible, key=lambda row: row[:2])

The fixture’s four synthetic profiles choose special, punctuation, locality, and full. FastGen assumes that a head’s prompt profile remains useful during generation; topic shifts can break that assumption. A production contract needs a full-cache fallback or an explicit reprofile transition.

The official repository is pinned at commit fb707b6, but that snapshot exposes the paper link and README rather than runnable method source. This article does not invent a source-code trace that is not there.

0x08I Ragged Storage Is Part Of The Method Contract

Heterogeneous K_lh produces ragged owners. There are three broad execution choices:

M5.4 owns the logical selected sets, owner lengths, original-position maps, and allocation receipt. M5.3 owns physical pages, slots, refcounts, sharing, and reclamation. M4.5 owns whether the decode backend can consume ragged K/V efficiently. A beautiful budget curve with a padded kernel can save logical rows while failing to save proportional HBM traffic or latency.

Generated rows form an append-only tail after the compressed prompt receipt. The prompt allocation remains stable unless the policy explicitly declares a recalibration transaction. One layer cannot publish its new allocation while another still exposes the parent generation; the receipt becomes visible only after every owner, position map, length, and offset validates.

0x08J Alternative Contracts And Failure Closure

Failure probes must distinguish arithmetic, state, runtime, and quality:

budget does not sum exactly:
    allocation/rounding failure

selected K and V use different positions:
    tensor identity failure

query-head expansion duplicates persistent rows:
    GQA ownership/accounting failure

one layer publishes before the rest:
    cross-layer generation failure

ragged rows are padded or gathered inefficiently:
    runtime admission/performance failure

later query needs an omitted original position:
    lossy visibility failure

retained attention mass is high but task quality falls:
    proxy-to-quality failure

The deterministic failure probe requests original position 5; none of the six fixture owners retained it. That proves only non-visibility. It does not invent a task score.

0x08K Executable Allocation Oracle

The article-local evidence is:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    adaptive_allocation_fixture.json
    invalid_adaptive_allocation_fixtures.json
    adaptive_allocation_oracle.py
    adaptive_allocation_report.json
    test_loop7_adaptive_allocation.py

The valid fixture derives PyramidKV and ZigZagKV layer targets, Ada-KV raw and safeguarded head budgets, exact integer apportionment, aligned symbolic K/V gathers, HeadKV global owner budgets, FastGen policy choices, one generation- stamped ragged receipt, and payload ledgers. Fifty-eight one-contract mutations reject. The oracle does not run a model, reproduce a paper benchmark, calibrate a checkpoint, implement a production allocator/kernel, or measure GPU traffic, latency, or task quality.

0x09 Boundary: Retention Is Not Retrieval Or Offload

Pure retention asks which K/V rows remain part of the logical history. Retrieval and offload preserve a larger pool elsewhere and decide which rows become resident or visible for one attention step.

retention miss:
    the historical row was permanently omitted from the logical cache

exact-offload miss:
    the exact row exists, but promotion is late or unavailable

retrieval miss:
    the row exists in the larger pool, but the query-dependent selector omitted it

Those cases need different correctness and fallback rules. Exact movement can preserve the full attention set. Query-dependent retrieval changes that set unless it recovers every required row or reruns an exact fallback.

M5.5 - KV Retrieval, Offload, and Hierarchical Memory develops the complete mechanism:

• exact GPU/CPU and multi-tier residency transactions;
• transfer, overlap, and staging-memory derivations;
• omitted-softmax-mass error rather than recall alone;
• InfiniGen’s orthogonal skewing, partial rehearsal, host pool, and prefetch pipeline;
• RetrievalAttention’s attention-aware index and partial-softmax combination;
• scheduler integration, index/pool consistency, exact fallback, and failure injection.

M5.4 retains this boundary because a hybrid can first delete or merge history and then retrieve from what remains. The evaluation must attribute quality loss to the correct stage.

0x0A Alternative Selection Signals And Budget Allocation

The methods above should not be read as a single progression from simple to advanced. They make different choices about three questions:

selection signal:
    recency, cumulative attention, regularized logits, observation-window attention,
    layer/head sensitivity, or query-dependent retrieval

budget location:
    global, per layer, per head, per request, or per decode step

irreversibility:
    permanently evict, merge, offload and fetch, or change the architecture

Scissorhands Versus H2O

Section 0x06 derives the difference from the papers and executable fixture. Scissorhands counts below-baseline attention over a bounded history and drops a batch at a compression event. H2O accumulates retained-row attention and forms a one-entry successor. Their common persistence intuition does not make their score, cadence, recent protection, or rollback state interchangeable.

Keyformer: Correct For Distribution Distortion In The Selection Score

Section 0x06 also derives Keyformer’s denominator distortion, Gumbel-softmax score, linear temperature schedule, prompt selection, and decode update. The important allocation point here is that Keyformer still assigns one key/recent budget per score owner. The next alternatives change that budget unit to a layer, head, cluster, or recall group; they do not redefine Keyformer’s score.

Head Roles Can Change The Budget Unit

Section 0x08 derives this case rather than repeating it here. Ada-KV reallocates within one layer, HeadKV can reallocate across all layer/head owners, and FastGen can assign different policy families rather than only different row counts. Their shared output is a heterogeneous logical visibility contract; 0x08I owns the resulting ragged-layout and backend boundary.

Semantic Clusters Replace Position Pages With Recall Groups

Eviction and merging are not the only ways to organize old K/V. ClusterKV groups cache entries in key/semantic space and recalls relevant clusters for a query. Its index unit is therefore not merely a contiguous token page.

Conceptual state:

cluster centroid / routing key
member token IDs or physical block references
resident members
offloaded or compressed members
recall metadata

At decode:

candidate_clusters = cluster_index.search(q, top_c)
candidate_rows = gather_members(candidate_clusters)
visible_kv = fetch_or_activate(candidate_rows, gpu_budget)
output = attend(q, visible_kv)

This can recover rows that a permanent eviction policy would have lost, but it adds index construction, search, gather, and possible tier-fetch latency. It is also approximate unless the selected clusters provably contain every row needed by the original dense attention.

Cluster granularity creates a different tradeoff from fixed pages:

large clusters:
    cheaper metadata/search, more irrelevant rows fetched

small clusters:
    finer recall, larger index and scattered gathers

The correct telemetry is not only cache hit rate. It includes cluster recall, rows fetched per useful row, search latency, transfer latency, and quality under misses.

Sparse Attention Execution Is Not Automatically Cache Compression

The original source also groups speculative or predicted sparsity with cache optimization. Keep the boundary precise:

store all K/V, score only a predicted subset:
    reduces attention work/read traffic if the kernel supports it
    does not reduce persistent cache capacity

discard non-selected K/V:
    reduces capacity
    changes the future information set and becomes lossy

store off-device and predict what to fetch:
    preserves recoverability
    adds miss and transfer latency

Those designs can share a selection model, but their failure contracts differ. An article must say whether an unselected row remains recoverable.

0x0B Allocation Handoff: Do Not Re-Derive The Budget Here

Section 0x08 owns layer/head allocation, integer conservation, calibration, ragged owner metadata, and paper-by-contract comparison. The remaining policy families change the stored representation rather than only distributing unchanged original rows:

0x08 allocation:
    assign K_lh, then retain unchanged original K/V rows

0x0C merging:
    replace multiple original rows with an approximate representation

M5.5 retrieval/offload:
    preserve a larger recoverable pool and move/select rows at query time

M5.7/M5.8 architecture:
    change which per-token head or latent representation exists at all

That boundary prevents the generic resource-allocation objective from being repeated with different names while hiding a more consequential change to the cache object.

0x0C Representation Change: Merge, Reconstruct, Or Change The Graph

Permanent eviction keeps a subset of original rows. The mechanisms in this section do something stronger: they replace original rows, share a state across layers, or stop producing most layer-local K/V altogether. Calling all three operations “compression” hides the compatibility decision that matters most.

0x0CA Three Compression Axes, Three Different Checkpoints

The immutable representation record therefore needs more than a method name:

KVRepresentationManifest:
    checkpoint_revision
    graph_revision
    representation_kind
    producer_layer_map
    consumer_layer_map
    owner = (layer, kv_head) or shared_layer_pair
    original_position_contract
    merge_or_reconstruction_parameters
    side_metadata_dtype_and_shape
    quantization_contract_id or null
    policy_generation

An eviction receipt can list retained original positions. A replacement receipt must instead list the source span or layer pair represented by every compressed row, the reconstruction rule, and the compatible consumer. It must never advertise the replacement as an exact prefix-cache row.

0x0CB DMC Inference: Derive The Segment State

Dynamic Memory Compression assigns a binary segmentation decision and an importance weight to every token, layer, and head. In the paper notation,

$$\alpha_t = \operatorname{round}(\sigma(k_t[0])) \in\{0,1\}, \qquad \omega_t = \sigma(q_t[0]) \in(0,1).$$

The convention is easy to reverse accidentally:

alpha_t = 0  -> append a new segment
alpha_t = 1  -> accumulate into the trailing segment

The two control neurons are then zeroed before ordinary attention. Let z be the accumulated importance mass of the trailing segment. On accumulation,

$$z_t=z_{t-1}+\omega_t,$$ $$\bar{k}_t = \frac{z_{t-1}\bar{k}_{t-1}+\omega_t k_t}{z_t}, \qquad \bar{v}_t = \frac{z_{t-1}\bar{v}_{t-1}+\omega_t v_t}{z_t}.$$

The denominator is not optional bookkeeping. Expanding the recurrence for one segment S gives

$$\bar{k}_S = \frac{\sum_{i\in S}\omega_i k_i} {\sum_{i\in S}\omega_i}, \qquad \bar{v}_S = \frac{\sum_{i\in S}\omega_i v_i} {\sum_{i\in S}\omega_i}.$$

Thus the update is an importance-weighted mean, not the unnormalized sum, a fixed convex interpolation, or a normalized key with a separately averaged value. If alpha_t=0, the cache appends (k_t,v_t) and resets z_t=omega_t. For n input rows, the compressed length is

$$T_{DMC} = \sum_{t=1}^{n}(1-\alpha_t), \qquad CR=\frac{n}{T_{DMC}}.$$

Source: A25 composite derived from Dynamic Memory Compression Figure 1, Equations 8-10, and Algorithm 1. It is used to expose the paper’s decision convention and running-mass update; it is not a local throughput result.

0x0CC DMC Tensor Trace: Five Rows Become Two Segments

For one layer and one K/V head, take

source positions:  [0, 1, 2, 3, 4]
alpha:             [0, 1, 0, 1, 1]
omega:             [1, 1, 2, 1, 1]

K rows: [[1,0], [3,2], [5,4], [7,6], [9,8]]
V rows: [[10,0], [30,20], [50,40], [70,60], [90,80]]

The two segments and their receipts are:

The anchor is the latest source position in the segment in this teaching receipt. That does not make the merged row equal to the latest token. It says which positional identity the compatible runtime assigns to the replacement. Mixing pre-RoPE rows, post-RoPE rows, or a different anchor convention without checkpoint and kernel agreement changes attention geometry.

def dmc_update(cache, q_t, k_t, v_t):
    alpha = round(sigmoid(control(k_t)))  # 1 means accumulate
    omega = sigmoid(importance(q_t))
    q_t, k_t = zero_control_channels(q_t, k_t)

    if alpha == 0 or cache.empty():
        cache.append(k=k_t, v=v_t, mass=omega,
                     source_span=[position_t, position_t])
    else:
        tail = cache.last()
        mass = tail.mass + omega
        tail.k = (tail.mass * tail.k + omega * k_t) / mass
        tail.v = (tail.mass * tail.v + omega * v_t) / mass
        tail.mass = mass
        tail.source_span[1] = position_t
    return cache

The production cache need not store every segment member, but audit and rollback state still needs enough identity to reject stale generations and to explain which source interval a replacement covers.

0x0CD DMC Training Is Not The Inference Loop Unrolled Naively

The binary decision cannot receive useful gradients through rounding. During continued pretraining, DMC relaxes it with a Gumbel-sigmoid sample,

$$\alpha_t \sim \operatorname{GumbelSigmoid}(k_t[0]-c,\tau) \in[0,1],$$

and extends the recurrence to partial accumulation:

$$z_i=\alpha_i z_{i-1}+\omega_i, \qquad \bar{k}_i = \frac{\alpha_i z_{i-1}\bar{k}_{i-1}+\omega_i k_i}{z_i},$$

with the analogous value update. Training retains the intermediate states so tokens can be processed in parallel, then applies an additive attention mask that suppresses segment states which would no longer exist at inference. This is a different materialized graph from the ragged inference cache.

The one-sided compression objective penalizes too many appended rows:

$$\ell_{CR} = \frac{1}{LHT} \max\!\left( 0, \sum_{l,h,t}(1-\alpha_{lht})-\frac{LHT}{CR_{target}} \right),$$

and the paper optimizes ell_LM + ell_CR while ramping the target compression ratio. The checkpoint can therefore learn unequal compression across heads and layers. A fixed post-hoc alpha rule is an alternative method, not DMC’s paper contract.

The paper uses the first Q/K neuron in its equations. The pinned official Megatron-LM dmc branch at commit d36bf8d5333b60c3d6b871128644c2b87d241d0b uses the last channel, zeros that channel around RoPE, keeps per-head lengths, running weighted windows, and a per-head paged block table. This is an implementation variant, not a reason to rewrite the paper’s notation silently.

0x0CE DMC Runtime State Is Ragged Twice

DMC produces unequal sequence lengths across both layer and head:

$$T_{l,h} = \sum_t(1-\alpha_{l,h,t}).$$

The logical payload is therefore

$$M_{DMC,payload} = B\,b_{kv}\,2d_h \sum_{l=1}^{L}\sum_{h=1}^{H_{kv}}T_{l,h},$$

plus running masses, per-head lengths, block tables, window state if the approximation is enabled, and allocator rounding. Padding every owner to max(T_lh) gives back part of the saved capacity. The pinned implementation uses pages allocated per head; M5.3 still owns the physical allocator, and M4.5 owns whether a backend can consume the resulting ragged state without hidden densification.

0x0CF MiniCache: Merge Across Depth, Not Across Time

MiniCache starts from a different observation: adjacent K/V states become similar in middle-to-deep layers. It pairs layers from a starting layer S and replaces two same-position vectors x^(l-1), x^l with one shared direction plus reconstruction metadata.

Source: A25-preserved MiniCache overview. The diagram distinguishes depth compression from token pruning; the illustrated behavior is paper-reported.

Write the unit directions and angle as

$$u_0=\frac{x^{l-1}}{\lVert x^{l-1}\rVert_2}, \qquad u_1=\frac{x^{l}}{\lVert x^{l}\rVert_2}, \qquad \Omega=\arccos(\operatorname{clip}(u_0^\top u_1,-1,1)).$$

For interpolation coefficient rho, spherical interpolation derives the shared direction

$$e^{l,l-1} = \frac{\sin((1-\rho)\Omega)}{\sin\Omega}u_0 + \frac{\sin(\rho\Omega)}{\sin\Omega}u_1.$$

When Omega -> 0, an implementation needs a normalized-linear fallback; when the pair is nearly antipodal, the shortest interpolation direction is ill-conditioned and should be retained separately or handled by an explicit policy. Restoration preserves each original norm but not its original direction:

$$\hat{x}^{l-1} = \frac{e^{l,l-1}}{\lVert e^{l,l-1}\rVert_2} \lVert x^{l-1}\rVert_2, \qquad \hat{x}^{l} = \frac{e^{l,l-1}}{\lVert e^{l,l-1}\rVert_2} \lVert x^{l}\rVert_2.$$

The directional error remains unless both original directions were equal. A shared row must therefore carry at least the pair identity, shared direction, two magnitudes, and the interpolation contract. Keys and values apply the construction separately.

Source: A25 annotated composite based on MiniCache Figure 3 and Equations 2-3. It exposes the persistent and reconstructed objects rather than claiming exact recovery.

0x0CG Distinct-Pair Retention Is Part Of MiniCache, Not A Footnote

MiniCache measures normalized angular distance d_i=Omega_i/pi and retains layer-specific rows for pairs deemed unsafe to merge. The compressed object is therefore hybrid:

C^(l,l-1) =
    shared directions E
    + norms for layer l-1 and l
    + angle/interpolation metadata
    + original exception rows R^(l-1), R^l
    + exception indices I

On reconstruction, the runtime rescales the shared direction, then overwrites the exception positions with their exact retained layer-local rows.

There is a source-level issue worth preserving rather than normalizing away. The MiniCache prose says highly distinct pairs are retained, but its printed Equation 4 uses a < threshold on angular distance for the set called retained. Those statements point in opposite directions if d is interpreted literally as distance. The official MiniCache repository snapshot f1baccacba6bc83dc499f5db9257576397bb4ed8 contains a project page and figures, not runnable method code that resolves the inequality. The local oracle follows the prose-consistent rule “large angular distance becomes an exception” and labels it rewrite interpretation, not a reproduction of Equation 4.

0x0CH MiniCache Prefill And Decode Materialize Different States

The paper’s runtime starts ordinary attention below S. For each adjacent pair at or above S, prompt prefill materializes both layer caches, merges the pair, stores side metadata and exception rows, and releases the redundant layer-local cache. Decode then reconstructs the layer needed for attention, appends the new token’s layer-local K/V, and re-merges the pair for the next step.

def minicache_pair_prefill(x_lo, x_hi, rho, exception_indices):
    direction, norm_lo, norm_hi, angle = slerp_merge(x_lo, x_hi, rho)
    return SharedDepthCache(
        direction=direction,
        norms=(norm_lo, norm_hi),
        angles=angle,
        exception_indices=exception_indices,
        exception_lo=x_lo[exception_indices],
        exception_hi=x_hi[exception_indices],
    )

def restore_layer(shared, which):
    x_hat = normalize(shared.direction) * shared.norms[which]
    x_hat[shared.exception_indices] = shared.exceptions[which]
    return x_hat

Source: A25-preserved MiniCache inference flow. Fetch, restore, attention, new-token update, and re-merge are separate operations; their latency is not measured locally here.

For one layer pair, sequence length T, head width d_h, exception count R, and one scalar width b, a transparent K-plus-V logical ledger is:

$$M_{full,pair}=4T d_h b,$$ $$M_{MiniCache} \approx 2T d_h b +4Tb_{mag} +2Tb_{angle} +4R d_h b +M_{indices}.$$

The first term stores one shared K and V direction instead of two layers. The remaining terms prove why “two layers become one” is not a 2x byte guarantee. For tiny d_h, metadata can dominate; for real head widths, exception rate and materialization workspace still decide the realized saving.

0x0CI MiniCache And DMC Are Not Interchangeable

PyramidKV and Ada-KV choose which original positions remain. MiniCache can in principle run after a shared position set is established, but merging different position sets is undefined. DMC changes sequence segmentation online, so composing it with a post-hoc selector requires a declared order and a new training/evaluation contract.

0x0D Architecture-Level Condensation And Learned Summaries

0x0DA Layer-Condensed K/V Changes The Producer Graph

Layer-Condensed KV Cache is not a merge operator. In its most condensed graph, queries from every layer consume K/V produced by the top layer from prior tokens. Non-producer layers do not need their own persistent K/V or their W_K,W_V parameters.

Source: A25-preserved Layer-Condensed KV Cache Figure 1. Horizontal edges denote query-to-K/V dependencies. This is a checkpoint graph, not a cache eviction schedule.

Let P be the set of K/V-producing layers. The persistent payload changes from

$$M_{standard} = B T L H_{kv} d_h 2b$$

to

$$M_{LCKV} = B T |P| H_{kv} d_h 2b,$$

before allocator and metadata overhead. A six-layer teaching graph with producer set P={0,5} and consumer map [0,5,5,5,5,5] stores two layer caches, not six. That 3x payload ratio is a graph count, not a local throughput or quality measurement.

0x0DB Why The Graph Creates A Cycle

For token t, a lower-layer query asks for the top-layer K/V of prior tokens. That is legal. If a token also attends to itself, however, its lower-layer query would need its own top-layer K/V before the token has reached the top layer. The paper removes this cycle by masking the attention diagonal for the top-attending path and using zero dummy K/V for the first token. Residual connections still carry the current token upward.

The paper then restores a small set of ordinary warmup layers. With an even count w, its sandwich configuration keeps w/2 bottom and w/2 top layers on standard layer-local attention. The producer/consumer map, diagonal rule, and warmup placement are therefore checkpoint coordinates, not serving flags.

0x0DC Parallel Training And Prompt Encoding Need Iterations

Exact left-to-right training would serialize tokens. The paper first rewrites the graph as n parallel-token iterations: iteration r consumes top-layer K/V from iteration r-1. Token i becomes correct by iteration i, so n iterations recover the sequential graph, but that is still impractical.

The approximation has two truncations:

b backward iterations:
    gradients flow only through the last b iterations
    b=1 cannot train the K/V-producing path used by the last iteration

m forward-only iterations:
    approximate the earlier n-b iterations because K/V empirically converge

The paper uses m=7 and b=2 in its reported setup, and prompt encoding uses m+b iterations before ordinary left-to-right decode. Those values are source-reported choices, not universal convergence constants. The official LCKV repository is pinned at dd21716b9295fb4dbe4af51ec3352a936b2b7df8; its configuration exposes layer_types, forward_passes, and backward_passes, while LayerCache stores only declared producing layers and supports iterative update control.

0x0DD Checkpoint And Composition Gates

ordinary checkpoint + DMC cache updater:
    reject; control channels and compression behavior were not trained

ordinary checkpoint + LCKV consumer map:
    reject; graph, diagonal mask, projections, and training dependencies differ

ordinary checkpoint + MiniCache transform:
    potentially admissible under the paper/runtime contract
    but cache representation and restore kernels still change

representation merge + K/V quantization:
    compose only after declaring whether directions, magnitudes,
    exception rows, and running masses are quantized separately

M8.6 owns the scalar format and calibration of those components. M5.7 owns K/V-head sharing, and M5.8 owns latent-cache identity and absorbed execution. Multiplying theoretical ratios is not enough: each composed stage adds metadata, temporary materialization, kernel constraints, and a new quality surface.

0x0DE Representation-Change Failure Lab And Oracle

The executable evidence is intentionally synthetic:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    representation_change_fixture.json
    invalid_representation_change_fixtures.json
    representation_change_oracle.py
    representation_change_report.json
    test_loop8_representation_change.py

It derives the five-row DMC trace, MiniCache SLERP and exception restoration, payload and reconstruction-error ledgers, the LCKV producer map, prompt iteration count, and incompatible graph/checkpoint failures. It does not train a checkpoint, execute the paper kernels, reproduce paper quality or throughput, measure GPU traffic, or certify a production cache implementation.

Forty-four one-contract mutations reject reversed DMC decisions, inconsistent segment masses, mismatched MiniCache layer pairs, missing exception state, invalid Layer-Condensed producer graphs, incompatible checkpoints, and byte or publication drift. Passing that oracle proves internal contract consistency, not paper-result reproduction.

Learned Segment Summaries Change The Serving Object

AutoCompressor-style memory changes the stored object again. The model receives special learned summary-token embeddings, runs an ordinary causal Transformer, and interprets the final hidden rows at those positions as a compressed state for later segments.

Let segment i contain m_i token embeddings, let kappa be the number of summary tokens emitted per compressed segment, and let sigma_<i concatenate the summaries from all earlier segments. The input order is:

$$X_i = [\,\sigma_{<i};\ E(S_i);\ G\,] \in \mathbb{R}^{B\times((i-1)\kappa+m_i+\kappa)\times d_{model}},$$

where G is the learned block of kappa summary-token embeddings. After the Transformer produces final hidden state H_i, the current summary is the last kappa rows:

$$\sigma_i = H_i[:,\,-\kappa:,:] \in \mathbb{R}^{B\times\kappa\times d_{model}},$$

and summary accumulation updates the persistent state as:

$$\sigma_{<i+1} = [\,\sigma_{<i};\ \sigma_i\,].$$

This derivation exposes two easy mistakes:

summary-token embedding G:
    a learned input that asks the model to produce a summary

summary vector sigma_i:
    the resulting final-layer hidden state at that summary-token position

neither object is:
    an original token's per-layer K/V row

The distinction also separates AutoCompressor from the recurrent-memory baseline it extends:

Summary accumulation is therefore compressed growth, not globally fixed-size memory. With b bytes per scalar, the externally stored summary payload before segment i is:

$$M_{summary}(i) = B(i-1)\kappa d_{model}b.$$

It grows linearly in the number of compressed segments, but at kappa vectors per segment rather than m_i source tokens. The paper explicitly notes that attention over an ever-growing summary prefix still has a quadratic cost. A memory ledger that writes only “50 summary vectors” is incomplete unless it says whether those 50 vectors replace one segment or the whole document.

Training and position semantics are consumed from M3.5

M3.5 - Context Extension owns the AutoCompressor architecture and training derivation: summary-token inputs versus emitted summary vectors, the next-token objective, segment-length randomization, truncated cross-segment gradients, checkpoint-specific LoRA or full-parameter adaptation, and absolute/RoPE position conventions. M5.4 does not repeat that training recipe. It consumes one compatible trained checkpoint and asks what persistent object remains visible under its serving policy.

Runtime has persistent state and working state

Once sigma_i is extracted, an implementation can discard token-level working state for segment S_i if it chooses summaries as the only persistent history. That does not make later attention free. When sigma_<i is prepended to a later call, the Transformer still projects those d_model vectors into per-layer keys and values for that call. If generation caches the resulting prefix, those derived K/V rows become part of the runtime cache.

persistent between calls:
    accumulated summary vectors         [B, (i - 1) * kappa, d_model]

temporary while compressing segment i:
    segment activations and attention state
    current summary-token rows

created when summaries are consumed:
    per-layer K/V derived from summary-prefix embeddings

discarded under summary-only retention:
    token-addressable history for completed source segments

The resulting failure contract is architectural and lossy:

summary bottleneck:
    evidence absent from sigma_i cannot be recovered from discarded source rows

segmentation shift:
    boundaries or lengths unlike training can change what is compressed together

state growth:
    accumulation is smaller than raw context but still grows by kappa per segment

checkpoint mismatch:
    summary vectors are meaningful only to a compatible trained model and position rule

false cache equivalence:
    treating sigma_i as ordinary K/V bypasses the model projections that consume it

M3.5 owns how this learned state is produced. M5.4 owns the serving classification above: summary accumulation is lossy compressed history with a checkpoint-specific consumer, a distinct byte trajectory, and newly projected K/V when the summaries are read. Using precomputed summaries as a retrieval corpus is a separate composition owned by M5.5.

0x0E Runtime Integration: One Generation-Fenced Policy Transaction

The preceding sections derived what each policy retains, merges, or replaces. That does not yet make a policy safe to install in a serving runtime. A service must answer a different question:

how does one logically valid successor become the only cache state that a
later decode step, allocator, retriever, or remote worker is allowed to read?

The answer is not an in-place topk() followed by a free(). Selection, physical materialization, backend admission, publication, and reclamation have different owners. They also fail independently. This section therefore defines a normative Transformer System transaction contract. It is a rewrite-derived systems contract grounded in the immutable state established in Sections 0x01-0x02 and the neighboring M5 ownership boundaries; it is not a claim that StreamingLLM, H2O, SnapKV, DMC, or another paper exposes this exact transaction API.

0x0EA State Machine And Owner Boundaries

One accepted mutation follows this order:

begin
  -> append_committed
  -> score
  -> protect
  -> select
  -> validate
  -> backend_admit
  -> allocator_reserve
  -> pack
  -> publish
  -> release_parent
  -> export_handoffs

The phase names do not claim that every policy recomputes a score after every token. A static sliding window can return a deterministic selection receipt; SnapKV can reuse the prompt-compression receipt during ordinary decode; an architecture-level representation can preserve the checkpoint-defined producer map. The transaction still requires an explicit receipt at every boundary so that a no-op is distinguishable from a skipped validation.

The central invariant is:

$$\operatorname{VisibleState}(r)= \begin{cases} S_g, & \text{before one complete publication of }S_{g+1},\\ S_{g+1}, & \text{after that publication}. \end{cases}$$

There is no legal interval in which layer 0 reads the successor while layer 1 still reads the parent, or in which the new position map names rows whose physical reservation failed.

0x0EB Append Is A Committed M5.1 Input

Loop 5’s dynamic-policy order starts with the current accepted row. Runtime integration sharpens what “current” means. Let the parent logical cache have exclusive frontier $F_g$ and cache generation $c_g$. M5.1 accepts the row at original position $F_g$ and publishes:

$$F_{g+1}=F_g+1, \qquad c_{g+1}=c_g+1.$$

M5.4 may score position $F_g$ only after that append receipt is committed. An unaccepted speculative candidate, cancelled sampling attempt, or stale worker completion cannot enter the source universe merely because its K/V was temporarily materialized.

The Loop 9 fixture starts with:

request                  req-m54-loop9-001, generation 11
policy                   generation 7
parent policy state      generation 30
parent cache             generation 500, frontier [0, 63)

M5.1 accepted append     original position 63
logical successor        cache generation 501, frontier [0, 64)

This creates a two-generation relationship, not one overloaded counter:

cache generation 501:
    the accepted logical source rows changed

policy-state generation 31:
    the legal visible or represented subset of that source changed

Either can advance without the other in a different policy. Keeping both is what makes a stale selector detectable.

0x0EC Score, Protect, Select, Then Validate

The fixture uses two layers and two physical K/V heads. Each owner receives a four-row budget and must preserve original positions 0 and 63:

For every owner $o$, publication requires:

$$\begin{aligned} &V_o=\operatorname{sort}(\operatorname{unique}(V_o)),\\ &V_o\subseteq [0,F_{g+1}),\\ &P_o\subseteq V_o,\\ &|V_o|\le K_o. \end{aligned}$$

These are structural checks, not evidence that the selected set preserves task quality. The selector’s score receipt, decision epoch, policy generation, and source frontier must also match the transaction. A perfectly sorted set from the wrong attention epoch is still stale.

Alternative update strategies have different costs:

The policy manifest must name which strategy is active. They are not equivalent implementations of the same byte ledger.

0x0ED Backend Capability Admission Comes Before Packing

M5.4 can request a view; it cannot declare that a loaded backend consumes it. The Loop 9 request is:

representation: original_kv_rows_ragged_position_map_v1
owner lengths:  [4, 4, 4, 4]
positions:      explicit original-position map
hidden dense materialization: forbidden for this accounting claim

M4.5 returns a capability receipt stating whether the backend supports ragged owner lengths, explicit original positions, and the required mask/position semantics, plus declared workspace. This is admission evidence, not execution evidence. A capability flag does not prove that a kernel launched; that still needs M4.5’s dispatch diagnostic or profiler contract.

The ordering prevents a common accounting fiction:

M5.4 reports 16 retained rows
backend silently pads four owners to source length 64
service actually reads or materializes 256 owner-rows

If the backend cannot consume the requested representation, the legal choices are explicit: reject the policy configuration, fall back to a supported full or padded view and charge it, or select another admitted backend. “Continue and hope” is not a fourth mode.

0x0EE M5.3 Reserves And Packs An Unpublished Successor

After logical and backend validation, M5.3 receives an allocation request bound to:

transaction id
request generation
parent policy-state generation
successor cache generation
owner row lengths
representation schema

M5.3 owns physical blocks, offsets, reference counts, copy-on-write, mapping generation, and release. M5.4 owns the selected original positions. The PackedKVViewReceipt binds the two without merging their authority.

For the worked fixture, one K/V pair row is 256 bytes:

$$M_{\mathrm{row}}=2d_hb=2\cdot64\cdot2=256\ \text{bytes}.$$

The parent has 24 live owner-rows and the successor has 16:

$$M_{\mathrm{parent}}=24\cdot256=6144\ \text{bytes},$$ $$M_{\mathrm{successor}}=16\cdot256=4096\ \text{bytes}.$$

The packed-view digest covers request generation, successor policy and cache generations, manifest hash, and ordered original-position sets. Changing one selected row therefore changes the receipt identity. The digest does not replace the position map or authenticate a tenant; it detects contract drift in this deterministic evidence bundle.

0x0EF Publish Once, Release Later

Allocation success is not visibility. The new mapping remains unpublished until M5.4 can atomically bind all of the following:

parent state generation 30
successor state generation 31
parent cache generation 500
successor cache generation 501
source frontier 64
policy manifest generation 7
packed-view digest
backend capability receipt
score/protection/selection receipt

Conceptually:

$$S_{31}=\operatorname{Publish} \left( S_{30}, C_{501}, V, R_{\mathrm{backend}}, R_{\mathrm{packed}} \right).$$

Publish is conditional on every parent field still matching. It is not a blind pointer swap after asynchronous work. Only after readers can acquire S_31 may M5.3 release the physical rows owned solely by S_30.

wrong:
    free parent rows -> publish successor

correct:
    reserve successor -> pack -> publish successor -> release parent rows

Delayed release can temporarily retain both generations. That is not a leak by definition; it is the cost of making publication safe. A reclamation protocol may use epochs, leases, references, or another mechanism, but M5.3 must prove that no reader can still address the parent before reuse.

0x0EG Rollback Before Publication, Compensation After Publication

“Rollback” means different things on the two sides of the visibility point.

Before publication:

discard score/selection candidate
release any unpublished successor reservation
keep parent mapping, parent state generation, and reader visibility unchanged
record the rejection or fallback reason

After publication:

the successor may already have readers
do not resurrect the parent by mutating the published object
publish a compensating successor or follow an explicit terminal-failure path

This is why the transaction is not “do operations and undo them in reverse.” A published immutable generation cannot be made unpublished for readers that already observed it.

The hard stop condition for Loop 9 is therefore executable:

failure before publication cannot expose a partially compacted cache

0x0EH M5.2 Receives Work, Peak, Readiness, And Fallback Facts

The scheduler needs a forecast before admission and a readiness fact after publication. It does not need M5.4 to reimplement continuous batching or fairness.

The transaction peak is not the successor payload alone. For this fixture:

$$\begin{aligned} M_{\mathrm{txn\ peak}} &=M_{\mathrm{parent\ live}} +M_{\mathrm{successor\ unpublished}} +M_{\mathrm{selector\ workspace}}\\ &\quad +M_{\mathrm{backend\ workspace}} +M_{\mathrm{allocator\ metadata}}\\ &=6144+4096+1024+2048+512\\ &=13824\ \text{bytes}. \end{aligned}$$

The compaction traffic is a different ledger:

$$Q_{\mathrm{compact}}=Q_{\mathrm{read}}+Q_{\mathrm{write}} =4096+4096=8192\ \text{bytes}.$$

Neither number is a measured GPU value. The fixture uses small synthetic rows to make accounting identities auditable. A real runtime must include block rounding, metadata structures, synchronization, allocator behavior, and the actual backend workspace.

M5.4 exports:

persistent successor payload and metadata
transaction peak
selection work and compaction traffic
fallback class
policy version
decode_ready = false before publication
decode_ready = true only after publication and required foreign readiness

M5.2 decides whether and when that work is admitted. A positive estimate is not an allocation receipt, and an admitted transaction is not decode-ready before publication.

0x0EI Recoverability And Transfer Use The Published Identity

The fixture labels omitted rows recallable_nonresident. That label is legal only because a separately identified source history still exists. M5.4 exports:

policy state generation 31
source frontier 64
source-history contract id
retrieval-plan id
omitted-state class

M5.5 owns lookup, promotion, movement, visibility, miss, and fallback. If no source history exists, M5.4 must label the omission irreversible instead of minting a retrieval plan.

M5.6 similarly receives only the published transfer-object identity:

request id and generation
policy state generation 31
cache generation 501
packed-view digest
representation schema
payload bytes

M5.6 owns worker choice, topology, transport, destination publication, and failure recovery. Exporting an unpublished successor would let a remote worker race ahead of local publication. Exporting only request ID would let a stale transfer satisfy a newer request generation.

This preserves the distinction:

M5.4: which logical state is the successor?
M5.5: which preserved state can be recalled and made locally visible?
M5.6: which worker receives a versioned state object, and when is it visible?

0x0EJ Observability Requires Identity And Denominators

A single “compression succeeded” counter cannot diagnose this state machine. At minimum, record by policy version and transaction result:

Useful derived metrics retain their denominators:

$$A_{\mathrm{compact}} =\frac{Q_{\mathrm{compact}}} {M_{\mathrm{newly\ admitted\ K/V}}},$$ $$\operatorname{churn}(V_t,V_{t-1}) =1-\frac{|V_t\cap V_{t-1}|}{|V_t\cup V_{t-1}|}.$$

High compaction amplification can erase the traffic saved by a smaller attention view. High churn can indicate unstable selection and repeated data movement. Neither metric alone proves quality or latency impact.

0x0EK Transaction Pseudocode, Failure Lab, And Executable Oracle

The complete teaching path is:

def mutate_visibility(parent, append_receipt, policy, scheduler_admission):
    txn = begin_transaction(
        expected_request_generation=parent.request_generation,
        expected_cache_generation=parent.cache_generation,
        expected_policy_generation=parent.policy_generation,
        expected_state_generation=parent.state_generation,
    )

    assert append_receipt.parent_cache_generation == parent.cache_generation
    assert append_receipt.committed

    evidence = policy.observe(append_receipt.successor_frontier)
    proposed = policy.select(evidence)
    validate_sorted_unique_in_range(proposed)
    validate_protected_sets_and_budgets(proposed, policy)

    capability = m45_admit_backend(
        representation=policy.representation,
        owner_lengths=lengths(proposed),
        position_contract=policy.position_contract,
    )
    if not capability.admitted:
        return explicit_fallback_or_reject(parent, capability.reason)

    work = estimate_peak_and_compaction(parent, proposed, capability)
    assert scheduler_admission.matches(txn.identity, work)

    reservation = m53_reserve_unpublished(txn.identity, proposed, work)
    try:
        packed = m53_pack_and_validate(reservation, proposed)
        successor = make_immutable_successor(
            parent,
            append_receipt,
            proposed,
            capability,
            packed,
        )
        published = compare_and_publish(parent.identity, successor)
        if not published:
            raise StaleGeneration
    except Exception:
        m53_release_unpublished(reservation)
        assert visible_state(txn.request_id) is parent
        raise

    # Parent release is legal only after publication is observable.
    m53_release_parent_after_publication(parent.physical_receipt, successor)
    export_m52_work_and_readiness(successor, work)
    export_m55_recoverability(successor)
    export_m56_transfer_identity(successor)
    return successor

The checked-in evidence contains one accepted transaction and 109 one-contract-invalid fixtures. The oracle rejects stale identity, malformed selection, unsupported backend, hidden densification, allocation failure, partial publication, parent release before publication, premature scheduler readiness, missing fallback, stale recovery and transfer handoffs, rollback leakage, and inconsistent counters. It validates ordering and accounting only:

docs/numinens-evidence/m5-4-long-context-kv-policies/runtime_transaction_fixture.json
docs/numinens-evidence/m5-4-long-context-kv-policies/invalid_runtime_transaction_fixtures.json
docs/numinens-evidence/m5-4-long-context-kv-policies/runtime_transaction_oracle.py
docs/numinens-evidence/m5-4-long-context-kv-policies/runtime_transaction_report.json
docs/numinens-evidence/m5-4-long-context-kv-policies/test_loop9_runtime_transaction.py

It does not implement a scheduler, allocator, retrieval service, transfer path, backend kernel, or production cache. It records no GPU trace, latency sample, paper benchmark reproduction, or model-quality measurement.

0x0F Unified Policy Reconciliation: One CLI, Typed Coordinates, No Hidden Ranking

Loops 2-9 now have eight independent executable evidence bundles. Independence is useful: a bug in the SnapKV selector should not require the runtime transaction oracle to know how pooling works, and a DMC replacement receipt should not be forced into an original-row visibility schema. Independence also creates a publication problem:

how can one command prove that every checked example is current, identify
which values are actually comparable, and refuse to turn unlike fixtures into
a policy leaderboard?

Loop 10 answers with a report reconciler, not a ninth policy oracle. It consumes the checked outputs of Loops 2-9. It does not import or call their selectors, allocation algorithms, attention calculations, representation transforms, or transaction state machines.

0x0FA The Reconciler Owns Receipts, Not Algorithms

The ownership split is strict:

For report $i$, the normalized receipt is:

$$R_i= \left( \operatorname{id}_i, H_i^{\mathrm{fixture}}, H_i^{\mathrm{report}}, A_i, C_i, S_i, L_i, E_i \right),$$

where $A_i$ is the versioned adapter, $C_i$ is the comparison coordinate, $S_i$ is a sequence of extracted state snapshots, $L_i$ is a ledger sequence, and $E_i$ is the evidence boundary. None of those fields is a new policy decision.

An alternative would be to have the CLI rerun all eight oracles. That sounds stronger but creates a second orchestration implementation, hides which checked report the article cites, and lets environment drift change results at render time. Loop 10 instead verifies immutable outputs. Regenerating an upstream report remains the responsibility of its owning oracle and tests.

0x0FB Raw-Byte Hashes Precede Schema Adaptation

Each fixture and report is pinned by SHA-256 over its checked raw bytes:

$$H_i^{\mathrm{report}}= \operatorname{SHA256}(\operatorname{raw\_bytes}(P_i^{\mathrm{report}})).$$

Parsing before hashing would let whitespace or serialization changes escape the identity used by this release bundle. Canonical JSON is used only inside the negative-test harness when it deliberately constructs an in-memory mutation and refreshes the expected hash. The checked artifacts use raw-file hashes.

The suite digest binds report order, both hashes, and the adapter version:

$$H_{\mathrm{suite}}= \operatorname{SHA256}\!\left( \big\Vert_{i=2}^{9} \operatorname{id}_i:H_i^{\mathrm{fixture}}: H_i^{\mathrm{report}}:A_i \right).$$

For the checked bundle:

suite digest:
0ba2024124155ae0cc5db738eb99aaaf34ec72eddd952d0701858fc9b4c8a24b

A matching hash is necessary, not sufficient. After hashing, the adapter also checks schema version, normalized fixture identity, prior valid-fixture acceptance, prior negative-suite closure, negative-fixture count, and evidence flags. A report that newly claims latency measurement cannot silently pass through a manifest that declares synthetic accounting only.

0x0FC Versioned Adapters Preserve Historical Report Shapes

The report family predates Loop 10, so its shapes are deliberately not rewritten into one large schema. Six reports have named string schemas. Loop 5 uses an integer schema version and Loop 6’s checked report has no top-level schema field. Those two are accepted only through explicit legacy adapters and exact hashes:

The reconciled total is:

$$28+23+27+40+31+58+44+109=360$$

prior one-contract-invalid cases. Loop 10 adds 58 reconciliation-level mutations. A legacy adapter is not permission to guess. Any schema change, even adding a schema string to the schema-less SnapKV report, requires a new adapter or a manifest revision.

0x0FD State Snapshots Are Observations Of Checked Results

The adapters emit one common descriptive shape:

report_id, loop, policy, step, owner
state_generation, cache_generation
selected_original_positions, selected_row_count, published

null is meaningful. Loop 3’s accounting receipt has no publication generation; Loop 8 replaces representations and therefore has no set of unchanged selected original rows. Filling either field with a convenient value would erase the mechanism boundary.

The checked reports yield 61 state snapshots:

Loop 2: four layer/KV-head packed views
Loop 3: one accounting receipt
Loop 4: full, sliding-window, and sink-plus-recent views at every event
Loop 5: H2O and Keyformer decode states plus final Scissorhands state
Loop 6: prompt publication and first-decode append per owner
Loop 7: six allocated layer/KV-head owners
Loop 8: one representation-replacement publication
Loop 9: twelve transaction phases

Loop 10 does not infer a missing state transition. If a checked report omits one, the adapter must expose the absence or the owning loop must publish a new report.

0x0FE Comparison Requires Equality Of The Metric Coordinate

Every report declares:

$$C_i=(c_i,\phi_i,r_i,o_i,b_i,m_i,d_i),$$

for cohort, phase, representation class, owner axis, budget unit, metric scope, and denominator. A metric $q$ is comparable between reports only if its required coordinate keys match:

$$i\sim_q j \iff \forall k\in K_q,\ C_i[k]=C_j[k].$$

Matching units are not enough. Two values both measured in bytes can differ in prompt length, number of owners, scalar width, included metadata, allocator rounding, representation, or phase. Two local attention errors can differ in query, values, selected budget, and normalization. The manifest therefore forbids cross-report comparison of resident bytes, workspace, reads, compaction, churn, local error, compression ratio, latency, and task quality unless a future schema declares one exact common cohort.

Only two cross-report links are currently declared:

The second link is intentionally lineage-only. The same policy manifest does not make Loop 2’s 8-token tensor comparable with Loop 9’s 64-position, four-owner transaction. Across the eight reports there are 28 unordered pairs: 2 have declared lineage and the other 26 are mechanically incompatible for metric comparison.

0x0FF Eleven Legal Within-Report Comparisons

The reconciler extracts 75 ledger rows and groups compatible values only inside their checked fixture. These values reproduce the article’s numeric examples without pretending that one fixture ranks another:

The generated Markdown table is longer because it retains option labels and contract conclusions verbatim. The compact table above is a renderer-facing explanation of the same checked JSON, not an independent calculation.

within_report_comparison_group_count: 11
cross_report_metric_comparisons_executed: 0
policy_algorithms_rerun: false

0x0FG CLI Lifecycle And Deterministic Outputs

The default command reconciles and writes the three checked outputs:

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_cli.py

Publication uses check mode:

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_cli.py --check

The pair explainer states why a tempting comparison is illegal:

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_cli.py \
  --explain loop2_visibility loop9_runtime

Its answer is shared_contract_lineage with metric comparability none, not a ratio. The reconciliation-level negative suite is also directly executable:

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_cli.py \
  --check-invalid docs/numinens-evidence/m5-4-long-context-kv-policies/invalid_policy_reconciliation_fixtures.json

The implementation order is:

load manifest
assert owner and fail-closed contract
for Loop 2..9:
    resolve evidence-root-local paths
    hash raw fixture and report bytes
    assert exact expected hashes
    assert fixture and report schemas
    normalize fixture identity
    assert prior valid and negative-suite closure
    reject undeclared measurement claims
    adapt already-computed snapshots, ledgers, and metrics
validate declared cross-report identity links
classify every report pair by coordinate
emit deterministic JSON, Markdown, and negative-suite report

--check compares exact bytes with the checked outputs. A deterministic reconciler that emits a different ordering is still a release failure because it changes the cited artifact and suite digest contract.

0x0FH Failure Lab And Evidence Boundary

The 58 Loop 10 invalid cases cover manifest ownership, path confinement, hashes, schema, normalized fixture identity, prior report health, evidence scope, adapter version, coordinate completeness, lineage links, and forbidden cross-report metrics. They do not mutate the policy algorithms; the 360 upstream invalid fixtures already own those mechanisms.

Checked artifacts:

docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_manifest.json
docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_cli.py
docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_report.json
docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_tables.md
docs/numinens-evidence/m5-4-long-context-kv-policies/invalid_policy_reconciliation_fixtures.json
docs/numinens-evidence/m5-4-long-context-kv-policies/policy_reconciliation_invalid_report.json
docs/numinens-evidence/m5-4-long-context-kv-policies/test_loop10_policy_reconciliation.py

This is deterministic synthetic reconciliation evidence. It does not execute a model, selector, allocator, scheduler, kernel, retrieval service, or transfer; it records no GPU trace, latency, task quality, paper benchmark, or production service behavior.

Loop 10 closes when:

[x] all Loop 2-9 fixtures and reports are raw-byte hash pinned
[x] every historical report shape has an explicit adapter
[x] all 360 prior invalid cases remain closed
[x] 61 state snapshots and 75 ledger rows have named source reports
[x] all 11 numeric comparison groups stay inside compatible fixtures
[x] lineage identity is distinguished from metric comparability
[x] all 28 report pairs receive a deterministic relation
[x] no cross-report metric comparison or policy ranking executes
[x] the 58 reconciliation mutations fail for their expected reasons
[x] JSON and renderer-safe Markdown outputs are deterministic

0x10 Paper Contracts, Workload Sensitivity, Evaluation Boundaries, And Failure Closure

The mechanism sections deliberately went paper by paper. That is necessary for understanding how each policy works, but it creates one final hazard: once the method names share a table, a cache ratio or speedup can look comparable even when the retained object, checkpoint, workload, baseline, and metric differ.

Loop 11 does not select a winner. It asks a stricter question:

Before two policies or reported results are compared,
are they operating on the same persistent object,
at the same phase and owner granularity,
under the same checkpoint, budget, workload, baseline,
runtime, metric, quality condition, and evidence class?

0x10A A Paper Name Is Not A Cache Contract

For method m, define the policy contract:

$$\mathcal K_m = (O_m,\Phi_m,G_m,S_m,B_m,C_m,U_m,R_m,W_m,E_m,F_m,A_m),$$

where:

O_m   persistent object: original rows, merged state, or producer graph
Phi_m decision phase: prompt handoff, decode update, calibration, or training
G_m   owner granularity: position, layer, head, layer/head, pair, or segment
S_m   selection or transform signal
B_m   budget value, unit, scope, and denominator
C_m   calibration, profiling, checkpoint, or training requirement
U_m   update rule after another generated-input row is committed
R_m   recoverability of omitted source state
W_m   supported workload and task family
E_m   evaluation contract and quality metric
F_m   runtime integration and foreign-owner dependencies
A_m   assumptions and guarantee scope

The tuple is long because each coordinate blocks a real false equivalence. StreamingLLM and H2O both retain original rows, but one protects fixed sinks and recency while the other updates heavy-hitter scores. SnapKV and PyramidKV both compress prompt state, but SnapKV votes from a prompt-end observation window per owner while PyramidKV first changes the budget by layer. DMC and MiniCache both replace original state, but DMC learns temporal segment updates through continued pre-training while MiniCache is a training-free cross-layer reconstruction with magnitudes, interpolation metadata, and exception rows.

The checked registry preserves seventeen complete contracts:

Three related papers stay in the registry as explicit architecture/training boundaries, not additional post-hoc K/V policies:

This distinction also closes a common composition error. A MiniCache result reported with 4-bit state is a depth-transform plus quantization result. A Scissorhands result combined with 4-bit quantization is likewise a composed coordinate. The combined number cannot be attributed to the retention policy alone.

0x10B Evaluation Coordinates Come Before Reported Values

A paper result needs another tuple:

$$\mathcal R = (V,C_s,\Theta,H,W,K,B,M,Q,Y,E),$$

with source version V, source location C_s, model/checkpoint Theta, hardware/runtime H, workload W, compression coordinate K, baseline B, metric and denominator M, quality condition Q, reported value Y, and evidence class E.

Renderer-safe fallback:

R=(source, location, model, hardware/runtime, workload,
   cache/budget coordinate, baseline, metric/denominator,
   quality condition, reported value, evidence class)

Measured comparison is permitted only if every required coordinate agrees:

$$Comparable(r_a,r_b) = \bigwedge_{c\in\mathcal C} c(r_a)=c(r_b).$$

The source-reported registry deliberately retains incompatible headlines:

Every value above is source-reported and unreproduced here. The executable coordinate check finds zero cross-paper result pairs whose complete vectors match. Therefore it produces no speedup ordering. That is the correct audit outcome, not missing analysis.

0x10C Deriving A Shared Byte And Visibility Ledger

The sensitivity suite does not emulate the fourteen methods. It defines a small common ledger that can expose contradictions without claiming paper performance. For L layers and H_kv persistent K/V heads, the number of owners is:

$$N_o=L H_{kv}.$$

One original K/V row stores one key and one value:

$$r_{bytes}=2d_h b.$$

For prompt length S and committed generated-input rows G, full-cache payload is:

$$M_{full}=N_o(S+G)r_{bytes}.$$

The teaching policy keeps K prompt rows per owner and every generated-input row, so:

$$M_{policy}=N_o(K+G)r_{bytes}, \qquad \rho=\frac{M_{full}}{M_{policy}}.$$

This derivation explains why a prompt-compression ratio is workload-dependent. At fixed K, increasing S raises rho; increasing the decode tail G pushes the ratio back toward one because both full and compressed states append the same generated rows.

For a sink-plus-recent-plus-scored policy, the selectable score slots are:

$$K_{score}=K-K_{sink}-K_{recent}.$$

Increasing sinks or recency at fixed K cannot be free: it removes score-based historical slots. For an observation window of O prompt-end queries, a simple teaching score is:

$$s_j^{(O)} = \sum_{q=S-O}^{S-1} a_{qj}.$$

Changing O can change selected identities while bytes remain constant. The result is not a derivative of task quality; it is a visibility counterexample.

The base fixture uses:

L=4, H_kv=2, d_h=128, b=2 bytes
S=32 prompt rows, G=8 generated-input rows
K=10 prompt rows/owner
2 sinks + 4 recent + 4 score-selected rows
uniform layer budgets [20,20,20,20]
uniform head budgets [[10,10], ...]

It derives 8 owners, 512 B per K/V row, 163,840 B full payload, 73,728 B teaching-policy payload, and a 2.222222x full-to-policy ratio. The selected prompt positions are [0,1,5,9,14,20,28,29,30,31]; delayed reference 14 remains visible.

0x10D Eleven One-Control Workload Sensitivities

Each case mutates exactly one declared input. Define the discrete local change:

$$\Delta_x g = g(x')-g(x),$$

while every other control remains immutable. This is a mechanism probe, not a claim about a production response surface.

The budget, recent-window, and sink rows are the central counterexample. All three can advertise the same or lower bytes while exposing different history. The layer/head rows provide a second counterexample: equal global row counts do not imply equal owner visibility.

0x10E The Minimum Evaluation Protocol

A publishable K/V-policy evaluation must declare all of these axes:

checkpoint and positional contract
prompt-length and decode-length distributions
retained object, representation metadata, and recoverability
budget value, unit, owner scope, and denominator
task family and delayed-reference distance
prompt/decode decision phase and update cadence
quality metric and full-cache baseline
hardware, runtime, kernel, batch, and parallelism
prefill, decode, selection, compaction, and transfer timing
failure, fallback, and incompatible-request outcomes
evidence class and local-reproduction status

Quality should be reported as a surface, not one average:

$$Q = Q(\Theta,W,S,G,K,\Phi,G_{owner},C),$$

where checkpoint Theta, workload/task W, prompt/decode lengths S,G, budget K, decision phase Phi, owner granularity, and calibration contract remain visible. Runtime likewise needs its own measured coordinate:

$$T = T(H,R,B_{batch},K_{kernel},S,G,K,\text{compaction cadence}).$$

The two functions cannot be collapsed into one cache ratio. A ratio omits which rows or representations survived; a quality score omits whether the runtime realized any memory-bandwidth benefit; a kernel speedup omits selection and compaction work.

Outcome accounting must also retain incompatible and unmeasured cases:

quality-valid
quality-degraded
runtime-inadmissible
checkpoint-incompatible
backend-incompatible
unmeasured

Dropping incompatible cases makes an architecture-specific method look universally deployable. Dropping degraded cases makes a lossy selector look exact. Reporting only successful runtime cases hides fallback frequency.

0x10F Alternative Methods Follow Different Bottlenecks

The registry supports bounded design branches rather than a winner:

stable streaming with no arbitrary middle recall
    -> sink plus recent retention is relevant

online importance appears persistent
    -> Scissorhands/H2O/Keyformer-style dynamic retention is relevant

one long prompt should be compressed once before generation
    -> SnapKV-style prompt-handoff selection is relevant

layer/head demand is heterogeneous
    -> Pyramid/ZigZag/Ada-KV/HeadKV/FastGen allocation is relevant

checkpoint adaptation is allowed
    -> DMC or layer-condensed architecture choices become admissible

adjacent-layer state is similar and reconstruction kernels exist
    -> MiniCache-style depth representation is relevant

omitted state must remain recallable
    -> this is no longer pure M5.4; M5.5 owns lookup and movement

Composition creates a new contract. Combining Ada-KV with H2O requires both the allocator and base score/update semantics. Combining MiniCache with quantization requires representation metadata, scales, reconstruction order, and compatible kernels. Combining a selector with M5.5 recall changes destructive omission into a timeout/fallback state machine. Paper guarantees do not transfer automatically to any of those compositions.

0x10G Twenty Failure Classes Close Over 418 Prior Rejections

Loop 10 already hash-reconciled 360 invalid fixtures from Loops 2-9 and closed 58 reconciliation mutations. Loop 11 consumes those checked reports by raw hash, giving 418 prior failures without copying their policy logic. Twenty cumulative classes then state where the final errors close:

The remaining classes cover incomplete coordinates, budget-unit mismatch, checkpoint/backend bypass, stale hashes/counts, and architecture-boundary collapse. The executable negative suite adds 48 Loop 11 mutations, so the checked cumulative rejection surface is 418 + 48 = 466 cases.

0x10H Executable Paper, Sensitivity, And Failure Contract

Loop 11 adds:

paper_contract_registry.json
source_reported_evaluation_registry.json
paper_sensitivity_failure_fixture.json
paper_sensitivity_failure_oracle.py
paper_sensitivity_failure_report.json
invalid_paper_sensitivity_failure_fixtures.json
paper_sensitivity_failure_invalid_report.json
test_loop11_paper_sensitivity_failure.py

Run or verify it with:

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/paper_sensitivity_failure_oracle.py --write-report

python3 docs/numinens-evidence/m5-4-long-context-kv-policies/paper_sensitivity_failure_oracle.py --check --check-invalid

The report contains 17 paper/architecture contracts, 14 source-reported result records, 11 one-control cases over 11 axes, 20 failure classes, 8 validator rules, 418 prior invalid cases, zero local reproductions, and zero measured cross-paper comparable pairs.

It proves that the source, comparison, sensitivity, and failure contracts are internally closed. It does not execute the papers, measure task quality, run a GPU, calibrate a selector, reproduce latency/throughput, or establish an optimal K/V policy.

[x] fourteen M5.4 policy contracts retain their distinct objects and phases
[x] three architecture/training papers remain explicit M3.5 boundaries
[x] fourteen source-reported values retain complete named coordinates
[x] zero cross-paper result pairs are ranked
[x] eleven sensitivity cases change exactly one control
[x] equal bytes are shown not to imply equal visibility
[x] checkpoint and backend admission fail closed
[x] all 418 prior invalid cases remain hash-bound
[x] all 48 Loop 11 mutations fail for their expected reasons

0x11 Expanded Mechanism Contracts

The sentence to keep repeating is:

which term changed?

0x12 Accuracy Contracts

Every lossy policy needs a failure-mode box. Without that, a memory article becomes misleading.

Full cache:

Safe with respect to cache visibility.
Not a guarantee of long-context reasoning quality.

Sliding window:

Unsafe for arbitrary old-context recall.
Any evidence older than W is unavailable.

StreamingLLM:

Useful for stable streaming.
Unsafe for arbitrary middle-context recall.
Sink tokens are anchors, not summaries.

H2O and heavy hitters:

Useful when importance persists.
Unsafe when future relevance differs from past attention scores.
Pure eviction cannot recover removed K/V.

SnapKV:

Useful when an observation window predicts future prompt relevance.
Unsafe for later questions about unselected prompt details.
Not exact prefix reuse.

PyramidKV, PyramidInfer, FastGen:

Useful when observed layer/head patterns hold.
Unsafe when the task needs broader context in layers/heads given small budgets.

InfiniGen:

Safer than pure deletion if full state remains offloaded and fallback exists.
Still unsafe for latency if useful entries are not fetched in time.
Approximate if non-fetched entries cannot participate.

The hard warning:

A token's current attention score is an observation under the current query,
layer, head, and decode state. It is not proof that the token will never be
needed again.

Accuracy Evidence Ladder

Section 0x02 derives the dense and selected attention outputs and publishes the article-local executable oracle. The policy-specific sections above must use that evidence ladder consistently:

selection identity and budget
    -> retained and omitted dense probability mass
    -> local attention-output error
    -> layer/token accumulation under the real checkpoint
    -> task metric and service-quality distribution

An early stage can diagnose a later failure, but it cannot replace the later measurement. In particular, the cancellation fixture proves that more retained mass need not mean lower local output error. Neither quantity proves downstream quality. A release claim therefore needs full-model evaluation under the same checkpoint, prompt distribution, generation policy, cache budget, and fallback contract; Loop 3’s task_quality remains explicitly null.

0x13 Pseudocode Shapes

The static policies in Sections 0x03-0x05 derive views from an exclusive committed frontier. They do not mutate a list and then infer what happened:

def derive_static_views(F, W, K_sink, K_recent):
    assert F >= 0 and W > 0 and K_sink >= 0 and K_recent > 0

    source = range(0, F)
    full = source
    window = range(max(0, F - W), F)
    sink = range(0, min(K_sink, F))
    recent = range(max(0, F - K_recent), F)

    return {
        "full": full,
        "window": window,
        "sink_plus_recent": sorted(set(sink) | set(recent)),
        "sink_recent_overlap": sorted(set(sink) & set(recent)),
    }

def publish_static_successor(parent, committed_cache_receipt, widths):
    assert committed_cache_receipt.parent_cache_generation == parent.cache_generation
    assert committed_cache_receipt.frontier > parent.frontier

    views = derive_static_views(committed_cache_receipt.frontier, **widths)
    successor = bind_new_state_generation(parent, committed_cache_receipt, views)
    return atomic_publish(successor)

def cancel_static_candidate(parent, unpublished_candidate):
    discard(unpublished_candidate)
    return parent  # frontier, cache generation, state generation, and views are unchanged

The physical release, compaction, ring-slot reuse, and page reclamation implied by a new view remain M5.3 operations. A StreamingLLM-compatible position-shift path is an additional execution contract, not something this set derivation can silently install.

Dynamic-retention implementations must use the paper-specific contracts from 0x06:

Scissorhands -> scissorhands_compress(...)
H2O         -> h2o_append_score_select_publish(...)
Keyformer   -> keyformer_score_select_publish(...)

A shared cumulative_attention.topk() helper may be an experiment baseline, but it must not be labeled as all three methods.

SnapKV-style prompt compression:

def snapkv_owner_preserving(
    source_k,
    source_v,
    observation_prefix_attention,
    query_to_kv_head,
    observation_window,
    maximum_prompt_capacity,
    pooling,
):
    # source K/V: [L, H_kv, T_prompt, d_h]
    # observation attention: [L, H_q, W_obs, L_prefix]
    L, H_kv, T_prompt, d_h = source_k.shape
    W_obs = observation_window
    L_prefix = T_prompt - W_obs
    K_select = maximum_prompt_capacity - W_obs

    assert source_v.shape == source_k.shape
    assert W_obs > 0 and maximum_prompt_capacity > W_obs

    if T_prompt < maximum_prompt_capacity:
        return full_prompt_receipt(source_k, source_v)

    # Holds here: K_select - L_prefix = maximum_prompt_capacity - T_prompt <= 0.
    assert 0 <= K_select <= L_prefix

    packed_k = {}
    packed_v = {}
    original_positions = {}

    for layer in range(L):
        # Sum normalized prefix attention over observation queries.
        query_head_votes = observation_prefix_attention[layer].sum(axis=-2)

        for kv_head in range(H_kv):
            mapped = [
                q for q, owner in enumerate(query_to_kv_head)
                if owner == kv_head
            ]
            # Teaching contract: preserve one physical K/V-head owner.
            owner_vote = mean(query_head_votes[mapped], axis=0)
            owner_score = pooling(owner_vote)
            selected = stable_topk(
                owner_score,
                K_select,
                tie_break="original_position_ascending",
            )

            positions = selected + list(range(L_prefix, T_prompt))
            assert len(set(positions)) == len(positions)

            packed_k[layer, kv_head] = gather(source_k[layer, kv_head], positions)
            packed_v[layer, kv_head] = gather(source_v[layer, kv_head], positions)
            original_positions[layer, kv_head] = positions

    return publish_compressed_prompt_receipt(
        packed_k,
        packed_v,
        original_positions,
        source_prompt_materialized_in_full=True,
        exact_prefix_reuse=False,
    )

def append_generated_row(compressed_prompt, accepted_row, original_position):
    assert original_position == compressed_prompt.frontier
    # Prompt positions are not reselected during ordinary decode.
    return publish_append_successor(compressed_prompt, accepted_row)

The order of the checks matters. Subtracting the two definitions gives K_select - L_prefix = maximum_prompt_capacity - T_prompt, so a prompt shorter than the capacity has more selection slots than prefix rows. With the fixture’s capacity 4 and observation window 2, a three-token prompt has K_select = 2 but L_prefix = 1; a one-token prompt even has L_prefix = -1. Such prompts must leave through the full-prompt return, as in the pinned implementation, which returns them before it forms any top-k budget. Below that return the budget check restates an invariant and cannot fail; placed above it, the check would raise on every short prompt and the full-prompt branch could never run.

The function above is the owner-preserving GQA alternative used by the local oracle. The pinned official Llama path expands K/V with repeat_kv before selection; changing between those contracts changes the persistent shape and must not be hidden behind the same receipt.

Layer budget:

def apply_layer_budget(cache, K_by_layer, score_by_layer):
    for layer in cache.layers:
        selected = top_positions(score_by_layer[layer], K_by_layer[layer])
        cache[layer].retain(selected)

Retrieval/offload handoff:

def long_context_visibility_contract(cache_policy, retrieval_plan):
    logical_history = cache_policy.retained_positions()
    # M5.5 owns residency, transfer, selection, and fallback for this set.
    return retrieval_plan.make_visible(logical_history)

These are contract sketches, not library APIs. M5.5 owns the packed transfer layouts, asynchronous promotion, retrieval state, and scheduler integration behind make_visible.

0x14 What M5.4 Hands To M5.5-M5.8

M5.4 ends with a changed memory object:

full cache:
    all K/V remains visible

retained cache:
    only selected K/V remains visible

compressed prompt cache:
    prompt K/V was reduced before decode

offloaded cache:
    K/V may exist outside GPU and must be fetched

M5.5 owns retrieval, offload, hierarchical residency, consistency, and miss/fallback behavior. M5.6 asks where prefill and decode run and how exact phase-transition K/V moves between workers. M5.4 matters there because every retention or offload policy changes the transfer problem:

transfer full K/V
transfer retained K/V
transfer compressed prompt state
fetch K/V from host or remote storage

M5.7 and M5.8 ask what the model architecture stores per token. That is a different lever:

M5.4:
    reduce or move visible token positions

M5.7:
    reduce full K/V heads per token position

M5.8:
    replace full K/V rows with latent content plus positional state

Both levers can combine. A GQA model can still use a sliding window. An MLA model can still offload latent KV. But the contracts must stay separate.

0x15 Final Formula, Source, Rendering, And Cross-Article Boundary Audit

Loop 12 adds audit closure without reopening M5.4 mechanisms. It asks whether the finished article can be reconstructed from its equations, source pins, visual provenance, generated page, and neighboring owner contracts without silently changing what any earlier loop claimed.

0x15A Final Formula Closure

The final registry contains 119 ordered display formulas from the historical object taxonomy through the Loop 11 evaluation protocol. Each row records its article heading, normalized LaTeX, SHA-256 digest, formula family, and M5.4 ownership class. Formula order is part of the publication surface: inserting, deleting, or moving an equation changes the generated registry and fails the audit.

The audit does not pretend every symbolic expression is defined on every input. These guards are part of the formula contract:

These are renderer-safe guard descriptions, not substitute equations. The canonical expressions remain the display formulas beside their derivations. formula_registry.json is mechanically regenerated from the Markdown and must match the committed registry exactly.

0x15B Final Source And Figure Closure

The final source surface contains:

17 externally verified primary papers
15 implementation records bound to 12 exact GitHub commits
17 attributed source-figure excerpts
24 local article/evidence sources
15 custom visual units:
    13 responsive desktop/mobile pairs
     2 desktop-only legacy diagrams
45 article-owned content image files in total

All primary-paper endpoints and all twelve unique commit pins were rechecked on 2026-07-31. The audit preserves four different statements:

paper says                         -> source-reported claim
pinned repository contains        -> source-observed implementation detail
local deterministic fixture shows -> synthetic contract evidence
production runtime measured       -> not claimed here

Every source figure records its source URL, figure identity, transformation, rights note, semantic role, and scope limit. Every custom SVG records that it is an original reconstruction, has a title and description, uses no gradient, and is referenced by the article. Four legacy visuals that predated the loop manifests are now registered explicitly; an image may no longer remain in the article merely because its file happens to exist.

0x15C Final Rendering Closure

The source/render audit keeps Markdown correctness separate from browser layout. At source level it checks balanced fences and display delimiters, nonempty expressions, exact formula-registry hashes, local post routes, image targets, alt text, and visual provenance. After npm run build, it requires:

119 display math nodes + 124 inline math nodes = 243 generated math nodes
0 empty generated math nodes
45 article-owned content image targets present in generated HTML
32 nonempty article image fallbacks
all local article routes generated
all four Loop 12 headings present

The responsive browser check remains a separate presentation test. It verifies desktop/mobile source selection, natural dimensions, image loading, MathJax materialization, duplicate IDs, and horizontal overflow. Neither generated HTML inspection nor browser geometry is a GPU, latency, quality, or paper benchmark.

0x15D Final Cross-Article Boundary Closure

M5.4 owns legal historical visibility and the policy state that changes it. It does not absorb the mechanisms needed to create, place, schedule, execute, or encode that state:

The closure is executable:

docs/numinens-evidence/m5-4-long-context-kv-policies/
    formula_registry.json
    final_publication_audit_fixture.json
    final_publication_audit.py
    final_publication_audit_report.json
    render_audit.py
    render_audit_report.json
    test_loop12_final_publication.py

M5.4 is complete after Loop 12. A later edit that changes equation order or semantics, paper attribution, commit pins, visual provenance, generated math, or any ownership edge must rerun the final audit and the cumulative M5.4 test suite.

0x16 Publish Gate

References

Local source articles:

• M3.5 - Context Extension
• M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure
• M5.2 - Continuous Batching and Scheduler Pressure
• M5.3 - KV Memory Layout and Exact Reuse
• M4.5 - Decode Kernels and Backend Boundaries
• M5.5 - KV Retrieval, Offload, and Hierarchical Memory
• M5.6 - Prefill/Decode Disaggregation and KV Placement
• M5.7 - MQA and GQA
• M5.8 - MLA Latent Cache and Absorbed Kernels
• Exploring the Transformer Series (20) - KV Cache
• Exploring the Transformer Series (24) - KV Cache Optimization
• Exploring the Transformer Series (25) - KV Cache Optimization for Handling Long Text Sequences
• Exploring the Transformer Series (26) - KV Cache Optimization: P/D Separation or Merging

Primary and high-quality sources:

• Efficient Streaming Language Models with Attention Sinks
• StreamingLLM official implementation, pinned in Loop 4
• H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models
• H2O official implementation, pinned in Loop 5
• Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time
• Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference
• Keyformer official implementation, pinned in Loop 5
• SnapKV: LLM Knows What You are Looking for Before Generation
• SnapKV official implementation, pinned in Loop 6
• PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
• PyramidKV official implementation, pinned in Loop 7
• PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
• PyramidInfer official implementation, pinned in Loop 7
• ZigZagKV: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty
• Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
• Ada-KV official implementation, pinned in Loop 7
• Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning
• HeadKV official implementation, pinned in Loop 7
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs
• FastGen repository snapshot pinned in Loop 7
• Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
• DMC official Megatron-LM branch, pinned in Loop 8
• MiniCache: KV Cache Compression in Depth Dimension for Large Language Models
• MiniCache official repository snapshot, pinned in Loop 8
• Layer-Condensed KV Cache for Efficient Inference of Large Language Models
• Layer-Condensed KV Cache official implementation, pinned in Loop 8
• Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
• Compressive Transformers for Long-Range Sequence Modelling
• Adapting Language Models to Compress Contexts
• AutoCompressors official implementation
• ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
• InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving