M5.5 - KV Retrieval, Offload, and Hierarchical Memory

M5.5 - KV Retrieval, Offload, and Hierarchical Memory

0x00 Historical Introduction And Owner Contract

The K/V cache began as a computation-saving device. During autoregressive decode, the new query at layer l attends to historical keys and values that were already projected during prefill or earlier decode iterations. Keeping those projections avoids recomputing the complete prefix for every generated token.

That optimization changes the systems problem. The checkpoint weights are fixed-size, but committed K/V grows with request length, live sequences, layers, and stored K/V heads. The first serving design is therefore simple only while the complete live cache fits beside the model on the attention device:

new token
  -> project Q_l, K_l,new, V_l,new
  -> read every legal historical K_l, V_l row from GPU memory
  -> compute full-support attention
  -> append the new committed row

No index decides relevance. No slow-tier read is on the critical path. The logical support, physical residency, and compute site happen to coincide. Long contexts break that coincidence.

M5.5 owns the problem that appears after the legal history is larger than its fastest execution tier. Its owner question is:

Which candidates are fetched, where do nonresident K/V rows live, and what consistency, approximation, fallback, and runtime contracts govern their use?

This is deliberately narrower than “long-context memory.” M5.4 decides which past rows or replacement representations remain legal model context. M5.5 starts from that declared logical pool and asks how a decode step obtains, accesses, or approximates the state it needs.

0x00A From A Resident Cache To A Data-Arrival Problem

Let S_l(t) be the set of historical positions that M5.4 declares legal for layer l at decode step t. Under the dense full-support baseline,

$$S_l(t)=\{0,1,\ldots,t-1\}.$$

The attention row consumes:

Q_l(t)       [B, H_q, 1, d_h]
K_l[S_l(t)]  [B, H_kv, |S_l(t)|, d_h]
V_l[S_l(t)]  [B, H_kv, |S_l(t)|, d_h]

For one request, the persistent K/V payload grows linearly:

$$B_{KV}(t) = t\,L\,2H_{kv}d_hb,$$

where L is the layer count and b is bytes per stored scalar. Moving this payload out of GPU memory can relieve capacity, but it creates a new question: what has to arrive, or what computation has to move, before attention may publish its output?

The design space developed in stages:

1. GPU-resident dense attention. Every legal row is already local. Capacity limits context length or concurrency.
2. Exact hierarchical placement. Full K/V survives in CPU memory, storage, or another tier. Capacity improves, but a move-to-compute design places transfer on the decode path.
3. Overlapped promotion. Layerwise preload and double buffering hide part of that movement. They do not erase bytes that exceed the available overlap window.
4. Compute near data. Instead of returning every row to the GPU, attention heads or partitions execute near the tier holding K/V and return a mergeable reduction state.
5. Query-dependent selection. A proxy, page summary, index, cluster, or predictor chooses a subset for the current query. Traffic falls because the attention support changes, so approximation begins at selection rather than at storage.
6. Recallable context memory. Distant state is organized into pages, semantic clusters, or events that can be recalled later. The state is not permanently discarded, but one decode step still sees only the selected units.
7. Service integration. Every path needs identity, readiness, deadlines, backpressure, cancellation, fallback, isolation, and release. A paper algorithm without those contracts is not yet a serving system.

This progression is not a chronology in which each stage replaces the previous one. A production path may combine exact tiering, query-dependent prefetch, partial computation on another device, and an exact fallback. The contract must say where each operation enters.

0x00B The Owner Objects Are Not One Cache

“The offloaded cache” is too imprecise for implementation. M5.5 must distinguish at least six objects:

logical K/V row:
    model-produced K_l[position], V_l[position] with owner and position identity

tier payload or replica:
    bytes stored in GPU, CPU, storage, or another declared tier

selector proxy:
    dimensions, page extrema, centroids, sketches, or partial K used to score candidates

index entry:
    a searchable representation plus a mapping back to logical K/V identity

gathered attention set:
    full K/V rows materialized for this query after candidate resolution

partial reduction:
    numerically mergeable attention state computed where some K/V already resides

These objects may differ in dtype, layout, lifetime, update cadence, and size. A product-quantized search code is not the full key row. A page summary is not an attention input. A remote online-softmax state is not a K/V replica. Their ledgers and failure modes cannot be merged.

Every path also carries five control planes:

identity:  model, adapter, tenant, request, generation, layer, head, position, RoPE state
residency: authoritative tier, replicas, leases, publication state, deletion state
selection: full support, candidate rule, index/proxy version, recall and fallback class
execution: move-to-compute, compute-near-data, gather layout, reduction and dtype
service:   deadline, queue, credit, cancellation, retry, disable and rollback policy

Loop 2 turns these fields into an immutable manifest. Loop 1 freezes why they exist and which article owns them.

0x00C Four Mechanism Lanes Before Paper Names

The central split is not “offload versus retrieval.” There are four lanes:

The article’s invariant follows directly:

moving compatible exact K/V or an exact mergeable reduction can preserve semantics;
omitting legal rows, changing their representation without an exact decoder,
or changing their position/mask contract changes the attention operation.

“Exact” here means exact with respect to the declared model, logical support, representation, positional state, mask, and numerical-tolerance contract. It does not promise bitwise equality across arbitrary devices, reduction orders, or dtypes.

0x00D One Decode Row, Four Different Traces

The same query tensor can drive four systems with different data paths:

baseline input
    Q_l                 [B, H_q, 1, d_h]
    full legal K/V      [B, H_kv, |S_l(t)|, d_h] each

A. exact move-to-compute
    required block ids  [B, N_blocks]
    promoted K/V        [B, H_kv, |S_l(t)|, d_h]
    attention output    [B, H_q, 1, d_h]

B. exact compute-near-data
    query partition     [B, H_q, 1, d_h]
    remote partial      (m_remote, ell_remote, o_remote)
    local partial       (m_local,  ell_local,  o_local)
    merged output       [B, H_q, 1, d_h]

C. query-dependent selection
    selector input      method-specific proxy or index query
    selected ids        [B, H_kv or H_q, K]          # often ragged
    gathered full K/V   [B, H_kv, K, d_h]
    approximate output  [B, H_q, 1, d_h]

D. recallable memory
    lookup units        pages, clusters, or events
    recalled identities ragged unit and token ids
    memory-path output  method-specific

The triple (m, ell, o) denotes a running maximum, an exponential denominator, and an unnormalized value numerator. M4.3 owns the online-softmax algebra. M5.5 later uses that algebra only to state what a heterogeneous tier must return for an exact merge.

This trace exposes two common category errors:

selected ids are not K/V rows;
a partial attention state is not a selected-row list.

0x00E Capacity Relief Can Become Transfer Or Compute Placement

Suppose B_remote,l(t) bytes required by layer l are not resident on its attention device. A serialized move-to-compute lower-bound sketch is:

$$T_{move,l} \gtrsim T_{queue,l} + \frac{B_{remote,l}(t)}{BW_{effective,l}} + T_{sync,l}.$$

Prefetch can overlap part of this time with earlier work, but the residual stall cannot be smaller than the transfer that remains unfinished at the attention deadline. Therefore a capacity win can become a bandwidth or queueing loss.

A compute-near-data path changes the byte equation rather than merely starting the same copy earlier:

move-to-compute traffic:
    remote K/V payload -> attention device

compute-near-data traffic:
    query or compact query state -> K/V tier
    mergeable partial attention state -> output device

That second path is exact only if every legal row is consumed once and the partial reductions can be merged under one numerical contract. Loop 5 derives this boundary. It is not permission to hide CPU, storage, or network compute under the word “offload.”

The same distinction explains why query-dependent prefetch is qualitatively different. It reduces B_remote,l(t) by predicting which rows matter. If it does not fetch all of S_l(t), the resulting speedup is coupled to selector error and fallback policy.

0x00F Source Lineage: What Each Step Added

Only after fixing the four contracts is it useful to attach paper names:

All benchmark claims for these systems remain source-reported and locally unreproduced unless a later loop explicitly records a local measurement coordinate.

The following source figure from InfiniGen makes the historical pressure concrete. Full GPU residency minimizes arrival delay but limits capacity. CPU residency removes that capacity limit while exposing transfer. Conventional prefetch overlaps only the part that fits under prior block compute. Critical-row prefetch reduces bytes by introducing query-dependent support selection.

Source: InfiniGen, Figure 3, preserved through A25. The timing paths are the paper’s conceptual comparison, not a local latency measurement. Panel (d) belongs to Lane C because it fetches selected critical rows; it is not the exact Lane A control.

0x00G Retrieval Is Not Ordinary Eviction

A token that appears unimportant now can become important for a later query. Permanent removal in M5.4 destroys the row’s recoverability. M5.5 instead permits a larger logical pool to remain in CPU memory, storage, or another declared tier while the active step materializes only what its contract requires.

But recoverable is not visible. A row can occupy one of four states:

resident and visible:
    the current attention path may consume it

preserved but nonresident:
    exact state exists, but it must be promoted or consumed near its tier

selected but not ready:
    the row is wanted for this query, but its transfer or remote compute missed readiness

recoverable but absent from selected support:
    state still exists, but the current approximate attention step behaves as if it were absent

Selection therefore needs more than a top-k list. It needs a versioned candidate receipt, a mapping to full row identity, readiness, a miss/error contract, and a fallback whose exactness class is explicit.

0x00H Retrieval Has Three Different Injection Points

The word retrieval hides three different model interfaces. They can all use nearest-neighbor search, but they do not retrieve the same object, inject it at the same stage, or preserve the same semantics.

The shared word does not create a shared cache contract. To classify a design, ask three questions in order:

what tensor or record is the query?
what exact object does the index return?
where does that returned object enter the model computation?

Attention-State Retrieval

M5.5 owns this interface. In the direct K/V form, the query for layer l selects historical positions and gathers their cached rows before attention:

Q_l                    [B, H_q, 1, d_h]
selected_position_ids  [B, H_kv, K]
gather K_l, V_l        [B, H_kv, K, d_h]
attention output       [B, H_q, 1, d_h]

With GQA or MQA, query heads share or expand the selected K/V-head rows according to the model’s head mapping. The retrieved object still participates in the attention numerator and denominator. Missing a row can therefore change both the value contribution and the softmax normalization, which is why later sections track omitted attention mass rather than retrieval recall alone.

Memorizing Transformers is an architecture-level relative of this interface: it stores recent internal key/value pairs in a non-differentiable memory and performs approximate k-nearest-neighbor lookup to augment attention. That memory should not automatically be identified with a serving engine’s byte-exact paged K/V cache. InfiniGen and RetrievalAttention, which this article studies in depth, operate closer to the long-context K/V serving boundary.

Output-Distribution Retrieval

kNN-LM uses a different interface. Its datastore associates a context representation k_i with the observed next-token label v_i. For the current hidden representation h_t, nearest neighbors produce a vocabulary distribution:

$$p_{\mathrm{kNN}}(y\mid h_t) = \frac{1}{Z(h_t)} \sum_{i\in\mathcal{N}_K(h_t)} \mathbf{1}[v_i=y] \exp\!\left(-\frac{d(h_t,k_i)}{\tau}\right).$$

The language-model and neighbor distributions are then interpolated:

$$p(y\mid x_{<t}) = (1-\lambda)p_{\mathrm{LM}}(y\mid x_{<t}) + \lambda p_{\mathrm{kNN}}(y\mid h_t).$$

The dataflow is:

h_t                         [B, d_model]
nearest-neighbor ids        [B, K]
neighbor next-token labels  [B, K]
distance-weighted p_kNN     [B, |V|]
interpolate with p_LM       [B, |V|]

Here, the datastore’s words key and value do not mean Transformer attention K_l and V_l. The values are token labels, and the retrieved result modifies the output distribution. kNN-LM does not recover evicted attention rows, reduce the attention-cache payload, or make historical K/V visible to a layer.

Document And Prompt Retrieval

Document retrieval moves the injection point earlier. A retriever returns passages or records; the system then supplies that evidence to the generator. A common prompt-oriented path is:

retrieval query embedding  [B, d_retrieval]
retrieved document ids     [B, K_doc]
tokenize and pack evidence [B, T + R]   # ragged in practice
prefill packed input       -> ordinary per-layer K/V for those input tokens

The original Retrieval-Augmented Generation paper is more specific than simple prompt concatenation: it couples a parametric sequence-to-sequence model to a dense passage index and marginalizes retrieved documents as latent variables. RAG-Sequence keeps one document choice for the generated sequence, whereas RAG-Token can use a different latent document per generated token. Modern prompt RAG often materializes retrieved text directly in the context instead. Neither contract is K/V offload: the retrieved evidence causes new model computation and therefore produces ordinary K/V during prefill.

The implementation consequence is strict:

attention-state retrieval:
    evaluate selected-row recall, omitted attention mass, K/V identity,
    transfer visibility, and exact fallback

output-distribution retrieval:
    evaluate datastore coverage, neighbor quality, interpolation calibration,
    and vocabulary-distribution quality

document/prompt retrieval:
    evaluate evidence recall, ranking, packing, citation/grounding quality,
    and the added prefill/context cost

These systems may coexist, but their recall metrics, cache identities, fallback rules, and correctness tests are not interchangeable.

0x00I Owner Contract: Consumes, Owns, Defers, And Forbids

M5.2 owns iteration selection, admission, fairness, and preemption intent. M5.5 exports versioned promotion/readiness/failure facts to that scheduler; it does not choose the next token rows or debit scheduler service.

M5.5 consumes facts from neighboring owners:

This article owns:

tier capability, residency, authority, and replica declarations
exact promotion, demotion, spill, restore, and inactive-session lifecycle
move-to-compute versus compute-near-data inside one decode service boundary
query-dependent selection at dimension, token, page, cluster, event, and index granularity
selector/index representation, build, update, lookup, gather, calibration, and garbage collection
partial attention state returned across tiers and its valid merge contract
versioned reserve-copy-or-compute-validate-publish transactions
fetch deadlines, coalescing, credits, backpressure, fallback, and retrieval-aware admission
capacity, transfer, compute, approximation, quality, and service ledgers as separate coordinates
tenant isolation, deletion, release, disable, and rollback for external K/V state

It defers:

sparse/ragged/heterogeneous kernel implementation  -> M4.5
prefix hashing, blocks, allocation, exact reuse     -> M5.3
irreversible deletion, retention, and temporal merge-> M5.4
P/D worker choice and inter-worker transport       -> M5.6
MQA/GQA architecture and head sharing              -> M5.7
MLA latent representation and absorbed kernels     -> M5.8
retrieval-backed drafting and verification         -> M7
K/V quantizer and codec derivation                  -> M8.6
output-distribution retrieval and kNN-LM            -> generation/output track
document/prompt RAG and evidence construction       -> retrieval/context construction track

The following collapses are forbidden throughout the article:

exact tier movement == approximate retrieval
recoverable nonresidency == permanent retention
prefix-cache hit rate == selector recall
candidate recall == attention-output or task-quality guarantee
same payload bytes == same model identity or position contract
offload == all K/V must return to GPU
selector/index bytes == full K/V bytes == transfer-codec bytes
uncommitted or cancelled rows are legal shared-index entries
paper benchmark == production capacity result
runtime tier orchestration == M5.6 worker placement or M4.5 kernel implementation

The handoff to Loop 2 is now precise: it will encode the logical row, tier replica, selector/index, gathered set, partial reduction, generation fence, deadline, and fallback as one immutable manifest. It will not yet choose an offload policy or a retrieval algorithm.

0x01 Immutable Retrieval, Residency, And Index Manifest

Loop 1 separated four mechanism lanes. They now need one source-to-tensor ABI. Without it, an exact offload path can return stale bytes, an approximate selector can query an index built for another checkpoint, and a correct set of positions can still be gathered under the wrong RoPE or K/V-head contract.

This section is a normative teaching contract. It synthesizes the logical cache generation from M5.1, the physical mapping and prefix identity from M5.3, and the recoverability decision from M5.4. InfiniGen and RetrievalAttention motivate the proxy/index/gather split, but neither paper is claimed to expose this exact manifest.

0x01A Why Shape Is Not Identity

Suppose two buffers both have shape [16, 8, 128] and dtype float16. They are not interchangeable merely because those fields match. The rows may have been produced by different:

model artifacts or adapters
layer and tensor-parallel K/V-head shards
original positions or RoPE scaling rules
committed cache generations
visibility policies
scalar codecs or layout transforms
tenant and cache-isolation namespaces

The first rule is therefore:

shape compatibility is necessary for a kernel call;
semantic compatibility is necessary for a legal cache read

The semantic identity of one logical row can be written as:

$$I_{row} = H\!\left( I_{model}, I_{adapter}, I_{cache\ ABI}, l,h_{kv},p,I_{position},G_{cache},G_{publication},I_{tenant} \right).$$

H means a canonical collision-resistant digest over typed fields, not string concatenation. The row identity does not include a GPU pointer: moving the same row creates another physical representation of the same logical state.

0x01B Three Identities, Not One Giant Key

One undifferentiated cache key hides where a failure entered. M5.5 keeps three related identities:

For a chunk replica:

$$I_{replica} = H\!\left( I_{chunk},I_{tier},G_{allocation},G_{replica},G_{authority}, I_{codec},I_{layout},H(payload) \right).$$

This factorization permits a codec-preserving promotion to change physical identity without changing semantic identity. It also exposes a codec mismatch without pretending that the source tokens changed.

0x01C Seven Objects Cross The Boundary

The objects form a provenance chain, not seven aliases for “the cache”:

logical row -> physical block -> tier replica
       |                           |
       +-> selector proxy -> index entry -> candidate ids
                                           |
                         validated replicas + candidate ids
                                           v
                                      gather rows
                                           |
                               optional partial reduction

The non-aliasing invariant is explicit:

candidate id != K/V row != tier replica != query-specific partial state

0x01D Freeze The Source Before Deriving Metadata

The manifest binds upstream identity, tensor geometry, and committed source state into one source digest:

$$I_{source} = H\!\left( I_{upstream},I_{tensor},G_{cache},G_{publication},[p_0,F_c) \right),$$

where F_c is the committed frontier. A proxy or index derived from that source must retain I_source; otherwise a same-shape stale index is indistinguishable from a current one.

@dataclass(frozen=True)
class RetrievalSourceBinding:
    logical_cache_abi_sha256: str
    cache_layout_manifest_sha256: str
    visibility_policy_manifest_sha256: str
    model_artifact_sha256: str
    checkpoint_generation: str
    adapter_set_sha256: str
    position_contract_sha256: str
    rope_config_sha256: str
    tenant_namespace: str
    namespace_generation: int
    request_generation: int
    cache_generation: int
    publication_generation: int
    committed_frontier: int

request_generation prevents late work from a cancelled/reused request. cache_generation identifies the logical history. publication_generation identifies the committed snapshot within that history. An allocator generation belongs to a different namespace and cannot substitute for any of them.

0x01E Tensor Contract And GQA Ownership

For one synthetic source with L=2, H_q=4, H_kv=2, T=8, and d_h=4:

K_source, V_source: [layer=2, kv_head=2, position=8, channel=4]
Q_decode:            [query_head=4, query_row=1, channel=4]
query -> KV head:    [0, 0, 1, 1]

Query head 2 therefore consumes K/V head 1. A selector result labelled only with head=2 is ambiguous: it may mean a query head or a K/V head. Every receipt must name both when H_q != H_kv.

The full teaching payload is:

$$M_{full} = L H_{kv}T d_h(b_K+b_V) = 2\cdot2\cdot8\cdot4\cdot(2+2) =512\ \text{bytes}.$$

At realistic scale, each token contributes:

$$M_{token}=L H_{kv}d_h(b_K+b_V).$$

For L=32, H_kv=8, d_h=128, and BF16 keys and values, this is 128 KiB/token; a 32K-token request is 4 GiB of logical payload before blocks, proxies, indexes, descriptors, checksums, leases, and staging.

0x01F Tier Capability Is A Manifest, Not A Tier Name

“CPU tier” does not establish what an operation may do. A capability record must state:

tier and address-space identity
kernel readability
DMA source and destination support
supported codecs and layouts
whether partial attention may execute there
persistence scope
trust and tenant boundary
alignment, checksum, and visibility requirements

The same bytes can therefore have different legal paths:

This table declares possibility, not policy. Loop 3 derives movement cost; Loop 5 derives compute-near-data reductions; Loop 10 maps version-pinned runtimes.

0x01G Authority, Replicas, And Leases

The semantic truth remains the committed M5.1 cache generation. M5.5’s authoritative replica has a narrower meaning: it is the validated physical copy whose survival currently prevents data loss or forced recomputation.

directory_generation:
    version of the replica directory

authority_epoch:
    fence for changing the recovery-authoritative replica

authoritative_replica_id:
    recovery responsibility, not semantic authorship

replica_generation:
    publication lifetime of one tier copy

allocation_generation:
    ABA fence for the underlying physical allocation

lease_generation and readers:
    reclamation fence for in-flight copies or kernels

The minimum survival condition is:

$$\sum_{r\in\mathcal R} \mathbf 1[valid(r)\land epoch(r)=G_{authority}] \ge R_{min}.$$

For exact demotion with R_min=1, the old replica cannot be released until a new replica validates under the same logical chunk, payload digest, codec, layout, tenant, and authority epoch. A selector proxy or index entry never counts toward R_min: neither is sufficient to reconstruct full K/V.

0x01H Selector And Index Publication Are Separate

A selector proxy is a derived tensor. An index is a search structure built over that proxy. They may update at different cadences:

proxy snapshot:
    source binding, coverage, basis or projection identity,
    proxy dtype/shape, derivation revision, exactness class

index snapshot:
    selector id/generation, source cache generation/frontier,
    coordinate namespace, entry ids, tenant, publication state

The version chain is:

$$G_{index} \rightarrow (I_{selector},G_{selector}) \rightarrow (G_{cache},F_c,I_{source}).$$

An appendable index may cover an older frontier F_i < F_c only if its contract explicitly adds another exact resident set or fallback for [F_i,F_c). This Loop 2 fixture chooses the stricter immutable snapshot: F_i=F_c=8.

The distinction matches the mechanisms without collapsing them. InfiniGen uses a reduced rehearsal representation to predict rows that should be prefetched; RetrievalAttention builds an attention-aware vector index over a CPU-resident pool. Those different algorithms still need source version, coverage, coordinate, and candidate-only semantics.

0x01I Query Operation, Deadline, And Fallback

The immutable manifest is long-lived; a lookup is query-specific. One decode operation binds:

request/cache/manifest generations
decode position, layer, query head, and owning K/V head
query and mask-contract digests
deadline measured from an explicit clock origin
fallback-contract id

The fallback is part of correctness because timeout behavior changes the executed support:

broaden_search: approximate, but spends more lookup/fetch work
exact_promote: exact if all legal rows become visible
recompute: exact under the same model/cache ABI
reject: no model output is published

Silently continuing with whatever rows arrived before the deadline is not a fallback contract. It is an undeclared approximation.

0x01J Source-To-Tensor Trace

The executable fixture traces one query in eight publication steps:

1. PREFILL PRODUCES SOURCE
   K,V each [2 layers, 2 KV heads, 8 positions, 4 channels]
   cache_generation=23, publication_generation=11, committed_frontier=8

2. M5.3 PACKS PHYSICAL STORAGE
   logical chunk covers l=[0,2), h=[0,2), p=[0,8)
   payload=512 bytes under the declared FP16 layout

3. M5.5 PUBLISHES REPLICAS
   host replica generation 17 is recovery-authoritative
   GPU replica generation 18 is stream-visible
   both bind the same logical chunk and payload digest

4. SELECTOR PROXY IS DERIVED
   synthetic two-channel proxy [2,2,8,2]
   source_binding binds ABI, checkpoint, RoPE, tenant, and frontier

5. INDEX SNAPSHOT IS PUBLISHED
   index generation 6 binds selector generation 4 and positions [0,8)

6. DECODE LOOKUP
   query head 2 -> KV head 1 at decode position 8
   candidate original positions = [0,3,7]

7. VALIDATE AND GATHER FULL ROWS
   K_gather,V_gather each [3,4]
   source replicas = [GPU generation 18, host generation 17, host generation 17]
   destination allocation generation 85 becomes visible at event generation 31

8. OPTIONAL PARTIAL ATTENTION
   receipt binds query, mask, layer/head, support=[0,3,7],
   FP32 online-softmax state (m,l,o), and merge revision

The sentinel for packed row 1 proves original-position preservation:

packed row:        1
original position: 3
layer / KV head:   1 / 1
K row:             [110030, 110031, 110032, 110033]
V row:             [610030, 610031, 610032, 610033]

The numbers encode (layer, KV head, original position, channel) and are not model activations. They make a swap, dense reindex, or wrong-head gather detectable without claiming a GPU or paper reproduction.

0x01K Lookup Receipt Is Not A Gather Receipt

A lookup publishes identifiers and scores:

LookupReceipt(
    query_sha256=query_hash,
    index_generation=6,
    selector_generation=4,
    source_cache_generation=23,
    source_committed_frontier=8,
    candidate_original_positions=[0, 3, 7],
    semantics="candidate_ids_only",
)

Only after validating candidate range, replica generation, codec/layout, row digest, destination allocation, and visibility event may a gather publish:

def validate_and_gather(manifest, operation, lookup, directory):
    require(lookup.query_sha256 == operation.query_sha256)
    require(lookup.index_generation == manifest.index.generation)
    require(lookup.source_cache_generation == manifest.cache_generation)
    require(all(p < manifest.committed_frontier for p in lookup.positions))

    rows = []
    for position in lookup.positions:
        replica = directory.choose_valid_replica(position)
        require(replica.authority_epoch == directory.authority_epoch)
        require(replica.tenant == manifest.tenant)
        rows.append(read_full_kv_row(replica, position))

    staged = normalize_layout_and_codec(rows)
    event = publish_to_attention_stream(staged)
    return GatherReceipt(
        original_positions=lookup.positions,
        source_replica_generations=[row.replica_generation for row in rows],
        destination_allocation_generation=staged.allocation_generation,
        visibility_event_generation=event.generation,
    )

candidate_ids_only forbids treating an ANN posting, page id, cluster id, or proxy row as if it were full K/V.

0x01L Partial Attention Receipt Is Query-Specific

Compute-near-data may return an online-softmax partition state rather than rows. Its immutable receipt is:

$$R_j= \left( H(q),H(mask),l,h_q,h_{kv},H(\mathcal S_j), m_j,\ell_j,o_j,dtype,I_{merge} \right).$$

S_j is the exact original-position support evaluated by partition j. m_j, ell_j, and o_j are query-specific reduction state. They cannot be indexed as reusable K/V because another query changes the scores and therefore the reduction. M4.3 owns the merge algebra; M5.5 owns the cross-tier receipt carrying its inputs.

Even a mathematically valid merge over [0,3,7] remains approximate relative to full support [0,8) unless another partition or fallback covers the omitted positions.

0x01M Design Alternatives Must Change Explicit Fields

The manifest does not choose among these designs. It makes the choice visible to later cost and algorithm loops.

0x01N One-Field Invalid Fixtures

The valid fixture is accompanied by twenty isolated failures. Each mutation is expected to fail with one stable reason before output publication:

Generation counters are not interchangeable. A current index generation does not repair a stale cache generation; a current allocation generation does not repair a stale tenant or RoPE binding.

0x01O Executable Contract And Claim Status

The article-local oracle publishes:

valid source K,V shape:       [2,2,8,4] each
selector proxy shape:         [2,2,8,2]
gathered K,V shape:           [3,4] each
partial state shapes:         m scalar, l scalar, o [4]
valid recovery replicas:      2
one-field invalid cases:      20 / 20 rejected as expected

Reproduce it with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/retrieval_residency_manifest_oracle.py

The handoff to Loop 3 is now bounded: derive exact offload bytes, transfer latency, overlap, staging peak, and transfer-bound regimes without changing the Loop 2 identity ABI or introducing selector approximation.

0x02 Exact Offload Is The Baseline Control

Before asking a selector to omit anything, establish a control that changes placement only. Suppose layer l is allowed to attend to committed support S_l(t). A move-to-compute control may keep those rows outside GPU memory between uses, but the layer cannot start until every legal row has a valid read path.

That sentence contains three different exactness claims:

For the first two axes, the contract is:

$$\widehat K_l[S_l(t)] = K_l[S_l(t)], \qquad \widehat V_l[S_l(t)] = V_l[S_l(t)].$$

The resulting output should satisfy:

$$O_{\text{offload}}(Q_l,S_l(t)) \simeq O_{\text{resident}}(Q_l,S_l(t)),$$

where \simeq permits only the numerical tolerance declared for the same dtype and supported kernel family. This is not permission to call a lossy transport codec exact. A lossless codec may change wire bytes; a lossy codec changes the representation contract and must be evaluated separately.

Merely retaining all rows in host memory does not establish exactness. If the kernel sees only the GPU-resident subset, the system has silently changed logical support.

0x02A A Versioned Residency Transaction

One useful state machine is:

GPU_VISIBLE
    -> DEMOTE_RESERVED
    -> COPY_OUT
    -> HOST_VALID
    -> GPU_RELEASED

HOST_VALID
    -> PROMOTE_RESERVED
    -> COPY_IN
    -> GPU_STAGED
    -> GPU_VISIBLE

COPY_IN completion is not publication. The event must refer to the same logical chunk, cache generation, allocation generation, owner, representation, and destination slot that were reserved when the copy began.

def complete_promotion(txn, copy_event):
    chunk = table.lookup(txn.logical_chunk_id)

    if chunk.cache_generation != txn.cache_generation:
        release_staging(txn.gpu_slot)
        return STALE_CACHE_GENERATION

    if allocator.generation(txn.gpu_slot) != txn.allocation_generation:
        release_staging_if_owned(txn)
        return RECYCLED_DESTINATION

    if chunk.owner_key != txn.owner_key:
        release_staging_if_owned(txn)
        return IDENTITY_MISMATCH

    if copy_event.bytes_completed != txn.payload_bytes:
        release_staging_if_owned(txn)
        return SHORT_COPY

    attention_stream.wait_event(copy_event)
    publish_visible_replica(chunk, txn.gpu_slot, txn.publication_generation)
    return READY

The pseudocode is a reference contract, not a claim about one framework’s classes. Loop 4 will extend the lifecycle through pause, save, restore, and resume; this loop owns the movement and schedule once a valid host replica already exists.

0x02B From Logical Rows To Physical Transfer Blocks

The Loop 2 ABI names logical rows by layer, K/V head, original position, and channel. DMA engines do not move abstract positions. They move physical byte ranges, usually at block or chunk granularity.

The executable Loop 3 fixture uses:

logical K,V per layer: [H_kv=8, T=4096, d_h=128]
K scalar bytes:        2
V scalar bytes:        2
transfer block:        64 token positions
layers:                4
transport codec:       identity FP16

One token position for one layer occupies:

$$B_{\text{row},l} = H_{kv}d_h(b_K+b_V) = 8\cdot128\cdot(2+2) =4096\ \text{bytes}.$$

A 64-token transfer block is therefore 256 KiB. The full four-layer request is 4096 * 4 * 4096 = 64 MiB of K/V payload.

The fixture deliberately mixes resident hits and cold blocks:

Every legal position still participates. The resident-hit vector changes traffic, not support. A block-based runtime may also transfer rows not requested by the logical operation. That is exact overfetch, not approximate attention.

0x02C Exact Alternatives Are Different Schedules

There is no single mandatory exact-offload implementation:

These are design choices, not steps of one universal system. This loop chooses layerwise move-to-compute as the executable control because it exposes both the transfer bottleneck and the memory needed to hide it.

0x02D FlexGen And vLLM Optimize Different Contracts

FlexGen begins from a broader placement problem. Its exact inference formulation jointly chooses a block schedule and fractions of weights, activations, and K/V assigned to GPU, CPU, and disk. It then estimates per-layer prefill and generation time from the maximum of CPU-to-GPU, GPU-to-CPU, disk-to-CPU, CPU-to-disk, and compute terms under a perfect-overlap assumption, and searches policy variables subject to memory constraints. The paper targets latency-insensitive, high-throughput inference; its zig-zag schedule amortizes weight I/O over a block of GPU batches. Its quantized-cache extension is not part of this exact representation control.

vLLM v0.26.0 documentation describes a different surface: completed K/V blocks are copied to a pinned CPU primary tier and promoted on demand; optional secondary tiers stage through that CPU tier because only the primary tier has direct GPU access. Its offload block size must be a multiple of the GPU block size. The documented CPU budget is total across workers, and the per-request max_offload_tokens control is explicitly experimental.

The implementation trace is pinned to vLLM tag v0.26.0, commit 568afb3a. At that commit, gpu_worker.py constructs separate CPU-to-GPU and GPU-to-CPU handlers, uses pinned descriptor buffers, records transfer events and byte counts, orders jobs within each direction, and gates GPU-to-CPU reads on the model-compute stream. Those facts ground the need for direction-specific queues, pinned-memory state, events, and measured byte receipts. Loop 10 will trace scheduler/manager integration; this loop does not claim that vLLM implements the teaching schedule below.

0x03 Exact Traffic, Latency, And Pipeline Derivation

0x03A Derive Logical Payload Before Physical Traffic

Let R_l be the legal row positions needed by layer l, G_l the subset already visible on GPU, and \mathcal{B}(R_l \setminus G_l) the physical transfer blocks covering the cold rows. For layer-specific geometry:

$$B_{\text{row},l} = H_{kv,l}d_{h,l}(b_{K,l}+b_{V,l}).$$

The logical cold payload is:

$$M_{\text{logical},l} = |R_l\setminus G_l|B_{\text{row},l}.$$

The H2D payload is instead determined by physical blocks:

$$M_{\text{H2D},l} = \sum_{c\in\mathcal{B}(R_l\setminus G_l)} B_{\text{payload}}(c).$$

Their ratio is read amplification:

$$A_{\text{read},l} = \frac{M_{\text{H2D},l}}{M_{\text{logical},l}} \ge 1.$$

The valid fixture is block-aligned, so A_read = 1. That does not make the distinction optional. A request for one row from a 64-token block can preserve exact support while moving 64 rows.

Per request and scheduler iteration:

$$M_{\text{H2D,request}} = \sum_{l=0}^{L-1}M_{\text{H2D},l}, \qquad M_{\text{H2D,iteration}} = \sum_{r\in\mathcal{A}}\sum_l M_{\text{H2D},r,l},$$

where \mathcal A is the set of admitted requests in that iteration. Three fixture-equivalent requests therefore move 108 MiB, not 36 MiB.

At the earlier 32-layer running shape (H_kv=8, d_h=128, BF16 K/V), the same formula gives:

This is why “fetch only a small subset” is not a byte contract. The selector must eventually be translated into row, block, request, and iteration traffic.

0x03B Promotion, Demotion, And Metadata Are Separate Ledgers

Promotion is not the only traffic. If T_write newly committed token positions are demoted after prefill or decode, the representation payload is:

$$M_{\text{D2H,payload}} = T_{write}\sum_l B_{\text{row},l},$$

before block rounding. The Loop 3 fixture writes one 64-token block for each of four layers, so D2H payload is 1 MiB.

Descriptors, checksums, events, and directory updates are not K/V payload:

$$M_{\text{traffic}} = M_{\text{payload}} +M_{\text{descriptor}} +M_{\text{protocol}}.$$

The fixture charges 64 B of descriptor state per physical block:

Do not add H2D and D2H bandwidth as if every link were automatically full-duplex. Conversely, do not serialize them if the measured platform has independent copy engines and traffic does not contend in host memory. The resource graph, not the direction label, decides overlap.

0x03C Transfer Time Is A Service Chain

payload / peak_bandwidth is a lower bound, not a latency model. For one promotion job j:

$$\begin{aligned} T_{j}={}&T_{lookup,j}+T_{source\ queue,j}+T_{source\ read,j}\\ &+T_{decode,j}+T_{link\ queue,j}+T_{submit,j}+T_{register,j}\\ &+\frac{M_{wire,j}}{B_{eff,j}}+T_{layout,j}+T_{publish,j}. \end{aligned}$$

The terms mean different things:

B_eff must use one byte convention consistently. The fixture uses binary GiB/s; published hardware links often use decimal GB/s. Mixing them creates a silent 7.4% error before any queueing is considered.

For a secondary tier staged through host, an unchunked path has at least:

$$T_{secondary\to GPU} = T_{secondary\to host} +T_{host\ transform} +T_{host\to GPU} +T_{publish}.$$

With n equal chunks and independent pipeline stages of service time S_h, the idealized stage pipeline becomes:

$$T_{chunked} = \sum_h S_h +(n-1)\max_h S_h.$$

This identity needs fixed chunk sizes, bounded queues, independent resources, and no head-of-line blocking. Without those conditions, use an event schedule, not the formula.

0x03D The Layer Pipeline Has An Exact Recurrence

Let F_l be the complete service time that makes layer l’s cold blocks visible and C_l its GPU compute time. A one-layer-ahead schedule uses one H2D resource and two staging slots. Slot l mod 2 cannot be reused until layer l-2 has finished consuming it. Let a_l denote when the serialized H2D resource becomes available after considering layer l; let e^F_l denote when layer l’s data is ready. These are deliberately separate: a resident layer is ready immediately, but it does not erase an earlier in-flight copy.

For a cold layer:

$$s^F_l = \max(a_{l-1}, e^C_{l-2}), \qquad e^F_l=s^F_l+F_l, \qquad a_l=e^F_l.$$

For an already-visible layer, set e^F_l=0 while retaining a_l=a_{l-1}. Compute obeys both the layer dependency and the data-ready event:

$$s^C_l = \max(e^C_{l-1}, e^F_l), \qquad e^C_l=s^C_l+C_l.$$

The exact service time is e^C_{L-1}. The uncovered transfer stall before a layer is:

$$T_{stall,l}=s^C_l-e^C_{l-1}\ge0.$$

Only in the special constant-time, unlimited-progress case does this reduce to:

$$T_{pipeline}=F+C+(L-1)\max(F,C).$$

That is the derivation behind the familiar warmup plus steady-state plus drain sketch. It fails when jobs have variable sizes, a slot is unavailable, copy and compute contend, a queue delays launch, or the next layer is not known early enough.

Equivalently, iterate only over cold transfer jobs in layer order and keep the last copy-engine completion unchanged when a layer is resident. Assigning a_l=0 at a resident layer would permit a later cold fetch to overlap an older copy on a supposedly serialized H2D resource.

def exact_layer_pipeline(layers, slots=(slot0, slot1)):
    fetch_end = 0.0
    compute_end = [0.0] * len(layers)

    for l, layer in enumerate(layers):
        slot = slots[l % 2]

        if layer.cold_blocks:
            slot_release = compute_end[l - 2] if l >= 2 else 0.0
            fetch_start = max(fetch_end, slot_release)
            receipt = promote_exact_async(
                blocks=layer.cold_blocks,
                destination=slot,
                expected_generation=layer.cache_generation,
                earliest_start=fetch_start,
            )
            fetch_end = receipt.end_ms
            data_ready = receipt.end_ms
        else:
            data_ready = 0.0

        previous = compute_end[l - 1] if l else 0.0
        compute_start = max(previous, data_ready)
        assert_exact_visible_support(layer, compute_start)
        compute_end[l] = compute_start + run_layer(layer, slot)

    return compute_end[-1]

The code-level invariant is stronger than copy_async(): the compute stream must wait for a receipt that binds the expected generation, bytes, and destination slot.

0x03E Worked Timeline: What Overlap Actually Hides

The fixture uses 24 GiB/s sustained H2D bandwidth. Its fixed service terms include lookup, queue, submit, layout, and publication. Compute and transfer times are synthetic; they are not vLLM or paper measurements.

The reconciliation is:

sum(fetch service) + sum(compute):  4.595 ms
two-slot pipeline service:          3.469 ms
transfer hidden by compute:         1.126 ms
uncovered transfer stall:           0.719 ms
scheduler queue outside service:     1.400 ms
request-visible contribution:        4.869 ms

Capacity, traffic, service time, and request-visible latency are four different answers. Reporting only 3.469 ms would hide the scheduler queue. Reporting only 36 MiB / 24 GiB/s = 1.465 ms would hide fixed costs and the compute dependency.

0x03F Double Buffering Buys Overlap With Memory

For variable-size staging allocations, the one-layer-ahead peak is:

$$M_{stage,variable} = \max_l\left(M_{H2D,l}+M_{H2D,l+1}\right).$$

For two fixed slots sized to the largest layer transfer:

$$M_{stage,fixed} = 2\max_l M_{H2D,l}.$$

The fixture gives 28 MiB variable staging versus 32 MiB reserved by two fixed 16 MiB slots. Peak GPU memory must then include every simultaneously live class:

$$\begin{aligned} M_{GPU,peak}={}&M_{persistent\ hits}+M_{stage}+M_{new\ rows}\\ &+M_{descriptors}+M_{kernel\ workspace}. \end{aligned}$$

Its reconciled peaks are:

Host capacity needs a different ledger: authoritative K/V payload, pinned DMA regions, filesystem/object-store read buffers, codec scratch, outstanding writeback, and metadata. Counting host K/V capacity while omitting pinned staging can still produce an out-of-memory or registration failure.

def can_admit_exact_offload(req, budgets):
    return (
        req.full_support_is_recoverable
        and req.peak_gpu_bytes <= budgets.gpu_free_bytes
        and req.peak_host_bytes <= budgets.host_free_bytes
        and req.pinned_bytes <= budgets.pinned_free_bytes
        and req.h2d_bytes_per_second <= budgets.h2d_available
        and req.d2h_bytes_per_second <= budgets.d2h_available
        and req.transfer_jobs <= budgets.queue_credits
    )

Admission based only on host capacity can accept a request whose decode or re-prefill rate collapses at the link.

0x03G Derive The Cached-To-Prefill Critical Ratio

A recent primary preprint, Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading, models re-prefill with K cached tokens restored from host and T new tokens computed on GPU. Let:

B_KV   K/V bytes per cached token across all cache-producing layers
F_pf   GPU FLOPs per new prefill token under the declared model
BW     sustained H2D bytes/s
C_eff  sustained prefill FLOPs/s

Ignoring fixed costs first:

$$t_{xfer}=\frac{K B_{KV}}{BW}, \qquad t_{compute}=\frac{T F_{pf}}{C_{eff}}.$$

Transfer dominates when t_xfer > t_compute:

$$\frac{K B_{KV}}{BW} > \frac{T F_{pf}}{C_{eff}}.$$

Divide by T, multiply by BW/B_KV, and define \kappa_{ratio}=K/T:

$$\kappa_{ratio} > \underbrace{\frac{F_{pf}}{B_{KV}} \frac{BW}{C_{eff}}}_{\kappa_{crit}}.$$

The units cancel:

(FLOP/token) / (byte/token) * (byte/s) / (FLOP/s) = 1

This is a regime test, not a latency prediction. Fixed lookup, queue, launch, storage, and publication costs can dominate small transfers. Partial overlap is often summarized as:

$$T=t_{xfer}+t_{compute}-\eta\min(t_{xfer},t_{compute}), \qquad 0\le\eta\le1,$$

but \eta must be measured. The event recurrence in 0x03D is preferable when the layer schedule is known.

For the fixture:

B_KV:                       16 KiB/token
F_pf:                       12.890625 GFLOP/token
BW:                         24 GiB/s
C_eff:                      300 TFLOP/s
kappa_crit:                 67.584
full cached/new ratio:      4096 / 64 = 64
cold-equivalent/new ratio:  2304 / 64 = 36

The ideal full-miss transfer is 2.604 ms; ideal compute is 2.750 ms. At the measured cold fraction, the fixture is compute-bound by this coarse test, even though startup still exposes 0.719 ms of transfer stall.

0x03H One-Control Sensitivity

The oracle changes one coordinate at a time:

The last row is a boundary sensitivity, not a free M5.5 optimization. M5.7 owns K/V-head sharing, M5.8 owns latent-cache representations, and M8.6 owns codec derivation and quality. M5.5 consumes their resulting B_KV only after the representation ABI is explicit.

0x03I Exactness And Performance Fail Independently

Executable assertions should therefore cover identity and schedule:

assert required_positions == committed_legal_support
assert transport_codec.exactness == "lossless_identity"
assert all(receipt.generation == cache_generation for receipt in receipts)
assert sum(receipt.payload_bytes for receipt in receipts) == h2d_payload_bytes
assert every_compute_start_is_after_its_data_ready_event
assert no_staging_slot_is_reused_before_consumer_completion
assert reported_ttft >= scheduler_queue_ms + pipeline_service_ms

0x03J Executable Reconciliation And Claim Status

Reproduce the deterministic trace with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/exact_offload_pipeline_oracle.py

It publishes:

full request payload:                64 MiB
H2D exact cold payload:              36 MiB
D2H newly produced payload:           1 MiB
three-request H2D iteration:        108 MiB
pipeline / no-overlap service:       3.469 / 4.595 ms
variable / fixed-slot GPU peak:     64.26 / 68.26 MiB
critical ratio:                     67.584
one-field invalid cases:            20 / 20 rejected as expected

Loop 4 now receives a bounded question: given this exact movement and cost model, when should an inactive or interrupted session save, preserve, restore, truncate, recompute, or discard K/V? It must not change the Loop 2 identity ABI or relabel selector approximation as exact offload.

0x04 Hierarchical Residency Is A Transaction Protocol

Once K/V can exist in more than two places, “the cache” is no longer one allocation. It is a versioned set of replicas with a legal read path.

GPU tier:
    smallest capacity, kernel-visible, highest value per resident byte

CPU tier:
    larger capacity, promotion source, often the serialization and indexing tier

local storage tier:
    persistent capacity, much higher access latency, useful for reusable prefixes

remote tier:
    shared capacity across workers, but network and failure domains enter the read path

The hierarchy does not imply that every chunk should visit every tier. A short-lived decode request may use GPU and CPU only. A shared system-prompt prefix may be worth writing to local or remote storage. A cold request tail may be cheaper to recompute than to persist.

0x04A Authoritative Copy, Replicas, And Leases

For each committed chunk, define:

owner_key       semantic cache identity
generation      monotonically increasing physical-lifetime version
authoritative   tier whose valid copy prevents data loss
replicas        other tiers containing byte-compatible copies
read_leases     in-flight kernels or transfers that prevent reclamation
write_txn       at most one publication transaction for this generation

An exact demotion should not free the last valid GPU copy until the slower-tier write is complete and validated:

def demote_exact(chunk, destination):
    assert chunk.state == GPU_VISIBLE

    txn = reserve_destination(
        key=chunk.owner_key,
        generation=chunk.generation,
        nbytes=chunk.nbytes,
    )
    event = copy_async(destination=txn.buffer, source=chunk.gpu_block)
    chunk.state = COPY_OUT

    event.wait()
    if not validate_identity_and_checksum(txn, chunk):
        abort(txn)
        chunk.state = GPU_VISIBLE
        return DEMOTION_FAILED

    publish_replica(txn)
    chunk.authoritative = destination
    chunk.state = HOST_VALID if destination == CPU else SECONDARY_VALID

    wait_for_read_leases(chunk.gpu_block)
    release_gpu_block(chunk.gpu_block)
    chunk.state = GPU_RELEASED
    return OK

This ordering costs temporary double residency. Releasing first and copying later saves peak memory but turns a copy failure into data loss or recomputation.

0x04B Promotion Is Reserve, Copy, Validate, Publish

A promotion is not simply cudaMemcpyAsync:

1. locate a compatible source replica;
2. reserve a destination block without exposing it;
3. transfer into that block;
4. validate generation, owner, length, and layout;
5. establish stream visibility;
6. publish the block-table entry atomically;
7. release the old mapping only after readers finish.

def promote(chunk_key, target_stream):
    source = directory.find_compatible(chunk_key)
    if source is None:
        return MISS

    handle = gpu_allocator.reserve(source.nbytes)
    txn = PromotionTxn(
        owner_key=chunk_key,
        generation=source.generation,
        source=source,
        gpu_block=handle,
    )

    copy_event = dma_copy_async(handle.ptr, source.ptr, source.nbytes)
    return finalize_after(copy_event, lambda: complete_promotion(txn, copy_event))

The attention batch may reference only published handles. A block-table entry that points at a still-copying buffer is a visibility bug even when every byte eventually arrives.

0x04C Cancellation And ABA Reuse

Suppose request R1 starts a host-to-GPU transfer into physical block 42, then is cancelled. The allocator gives block 42 to R2. If the old completion callback publishes using only the block number, it can overwrite R2’s mapping. This is an ABA problem: the physical identifier looks the same while ownership changed twice.

The completion must compare at least:

(physical block, allocation generation, owner key, transfer generation)

Cancellation should mark the transaction dead, but it cannot assume the DMA engine instantly stops. The stale completion still needs a safe cleanup path.

0x04D Promotion Coalescing And Request Fan-In

Many requests can ask for the same reusable prefix chunk. Starting one transfer per request wastes bandwidth and GPU blocks. A directory can coalesce compatible promotions:

def get_or_start_promotion(key):
    with directory.lock(key):
        if directory.has_gpu_visible(key):
            return READY(directory.gpu_handle(key))

        if directory.has_inflight_promotion(key):
            return WAIT(directory.inflight_future(key))

        future = start_promotion(key)
        directory.record_inflight(key, future)
        return WAIT(future)

Coalescing is legal only when identity includes every model-state component that changes K/V. A shared token prefix under two different adapters is not automatically a shared K/V prefix.

0x04E Tier Choice Is An Expected-Cost Decision

Let chunk c have size M_c, estimated reuse probability p_c, recomputation cost T_recompute(c), and promotion latency from tier j, T_promote(c,j). A simple expected-value heuristic is:

$$V(c,j) = p_c\left[T_{recompute}(c)-T_{promote}(c,j)\right] -C_{capacity}(M_c,j) -C_{write}(c,j).$$

This is not an optimal policy; it exposes the trade. Persisting a chunk is valuable when likely reuse avoids more work than storage, transfer, and metadata cost. FIFO, LRU, LFU/counters, learned reuse predictors, and admission filters estimate different parts of this value.

0x04F Two Pauses That Must Not Be Collapsed

An inactive conversation and an interrupted request both stop consuming decode iterations, but they do not create the same object.

inactive multi-turn session:
    the turn is complete;
    a future request may extend the committed conversation;
    the idle interval can be seconds, hours, or permanent;
    persistence competes with later recomputation.

interrupted request:
    the request is still logically in flight;
    a tool, human, or external service must return before decoding continues;
    sampling state, original arrival time, and the external call identity survive;
    preserve, swap, and discard/recompute compete during the pause.

The first lane is represented by CachedAttention. The second is represented by InferCept. Their paper mechanisms are useful together only after their different lifecycle semantics are kept visible.

M5.3 still owns the exact prefix key and reusable-prefix result. M5.5 consumes that identity and owns where the resulting K/V object resides, how it moves, when a saved generation becomes authoritative, and whether restoration is cheaper than recomputation. M5.2 owns the global scheduling policy; this section derives the costs and readiness receipts that policy consumes.

0x04G A Pause Is A Manifested State Transition

A session name is not enough to resume. The durable pause manifest needs to bind control state to the exact committed tensor frontier:

The snapshot frontier must satisfy

$$T_{snapshot}\le T_{committed},$$

and it must exclude speculative or otherwise uncommitted rows. This is a control-plane invariant, not a best-effort optimization.

cache_generation and lifecycle_epoch fence different objects. The former changes when physical cache authority is replaced or republished. The latter orders pause and resume transitions for one manifested request. Pause and resume therefore use the same counter namespace and an accepted resume is the immediate successor of the pause it consumes:

$$E_{resume}=E_{pause}+1.$$

An inequality between unrelated pause_epoch and resume_generation counters would not establish this ordering. Retries for the same manifested resume retain E_{resume}; a later pause/resume cycle advances the lifecycle counter again.

The two legal state families are:

completed turn:
    RUNNING -> QUIESCING -> SAVING -> PAUSED_HOST
            -> RESTORING -> RESUME_READY
            -> PREFILL_NEW_TURN -> RUNNING

interrupted request, preserve:
    RUNNING -> QUIESCING -> PAUSED_GPU -> RESUME_READY
            -> PREFILL_EXTERNAL_RESULT -> RUNNING

interrupted request, swap:
    RUNNING -> QUIESCING -> SWAPPING_OUT -> PAUSED_HOST
            -> SWAPPING_IN -> RESUME_READY
            -> PREFILL_EXTERNAL_RESULT -> RUNNING

interrupted request, discard:
    RUNNING -> QUIESCING -> TOKEN_ONLY -> RECOMPUTING
            -> RESUME_READY -> PREFILL_EXTERNAL_RESULT -> RUNNING

QUIESCING closes new decode work and waits for the committed frontier. It does not mean that an already-issued DMA operation has stopped. A save becomes publishable only after the copied generation, owner, byte length, position contract, and checksum validate. The last valid GPU copy cannot be released before another authoritative copy exists.

0x04H Why Multi-Turn History Becomes A Persistence Problem

In a conventional multi-turn path, turn 2 reconstructs the K/V for turn 1, turn 3 reconstructs turns 1 and 2, and so on. The handwritten A25 article uses the following source figure to introduce the repeated-work problem. The lower path stores completed historical K/V in AttentionStore and prefills only the new turn after restoration.

Source: CachedAttention conceptual figure, preserved through A25. The figure explains the mechanism; its latency benefit is not a local measurement.

Let H_r be the historical tokens already committed before turn r, N_r the new user tokens, and F_{pf} the model work per prefill token under a declared convention. Without a reusable snapshot, repeated model work across R turns is approximately

$$F_{recompute} \approx F_{pf}\sum_{r=1}^{R}(H_r+N_r).$$

With exact restored history, model work falls to

$$F_{restore} \approx F_{pf}\sum_{r=1}^{R}N_r.$$

This does not erase the separate save/load ledger. Persisted bytes, tier capacity-time, writes, reads, queueing, staging, synchronization, and validation must be priced independently. Persistence wins only when that complete service path is cheaper than the avoided model work.

0x04I CachedAttention: Layerwise Preload And Asynchronous Save

A bulk restore places every historical layer on the critical path before layer 0 can run. A layerwise schedule needs only layer l ready before layer l consumes it, while the copy engine fetches a later layer. For historical K/V payload M_{hist,l}, effective H2D bandwidth BW_{H2D}, and fixed per-layer cost tau_l, the load service is

$$t_{load,l} = \frac{M_{hist,l}}{BW_{H2D}}+\tau_l.$$

CachedAttention expresses the same pressure per token. Let T_load be K/V load time per historical token, T_pref new-token prefill time per token, L_hist historical length, L_new new-turn length, and B transfer bandwidth. Loading outruns available overlap when

$$T_{load}L_{hist}>T_{pref}L_{new}.$$

The paper’s larger read-buffer construction covers the time gap with

$$S_{buf} = B\max\left(0, T_{load}L_{hist}-T_{pref}L_{new} \right).$$

This formula is not a universal memory reservation. It assumes the paper’s per-token timing abstraction and enough lookahead to issue those reads. The event recurrence from Loop 3 remains the safer implementation model when layer sizes, fixed costs, or shared copy engines differ.

The deterministic Loop 4 fixture has four layers and 3,072 historical tokens. At 16 KiB of model-wide K/V per token, history occupies 48 MiB, or 12 MiB per layer. With a 24 GiB/s H2D path plus 0.1 ms fixed cost, each layer needs 0.58828125 ms to arrive. Its new-turn prefill work is 0.45 ms per layer:

The load/suffix pipeline is 2.803125 ms; lookup and scheduler queueing raise request-visible host resume to 3.483125 ms. The same object behind a disk penalty of 12 ms reaches 15.483125 ms, so this fixture restores from host but recomputes rather than taking the disk path.

Saving is the reverse dependency, not a mirror-image blocking copy. During prefill, a layer can produce many new K/V rows at once; during decode, it produces one row per active sequence. CachedAttention saves layer outputs asynchronously and reserves a write buffer for rows whose D2H write has not finished when the request releases its execution allocation.

def save_completed_turn(snapshot, write_slots):
    assert snapshot.frontier == snapshot.committed_frontier
    assert not snapshot.contains_uncommitted_branch_rows

    txn = reserve_snapshot_generation(snapshot.identity, snapshot.nbytes)
    for layer in cache_layers:
        await snapshot.layer_read_leases[layer].closed()
        slot = await write_slots.acquire(layer)
        copy_d2h_async(slot, snapshot.layer_view(layer), txn.generation)

    await txn.all_copies_complete()
    validate_identity_layout_positions_and_checksum(txn)
    publish_authoritative_snapshot(txn)
    release_gpu_rows_after_authority_exists(snapshot)

For the fixture, 12 MiB at 20 GiB/s plus 0.06 ms per layer gives 0.6459375 ms/layer and 2.58375 ms total D2H service. A declared 3 ms overlap window hides that service, so visible save stall is zero. This result does not mean saving is free: the copy engine, host capacity, pinned slots, and writeback queue remain occupied.

0x04J Scheduler Hints Place Sessions; They Do Not Validate Them

CachedAttention uses the waiting-job queue to prefetch sessions from disk into host memory and to avoid evicting sessions likely to run soon. Under its equal-size approximation, host prefetch capacity C_mem and average session K/V size S_kv give a lookahead window

$$L_{pw}=\left\lfloor\frac{C_{mem}}{S_{kv}}\right\rfloor.$$

Including disk capacity gives the paper’s eviction lookahead scale

$$L_{evict} = \left\lfloor \frac{C_{mem}+C_{disk}}{S_{kv}} \right\rfloor.$$

Conversation sessions are the paper’s fetch/eviction unit because a resumed turn consumes the session history together. A serving implementation with variable context lengths cannot safely turn those count formulas into byte admission. Its prefix of prefetched jobs must instead satisfy

$$\sum_{r=1}^{n}S_{session(r)} \le C_{mem,reserved\ for\ prefetch},$$

while preserving write-buffer, active-runtime, and emergency capacity. This is one reason a scheduler hint is not a cache hit: M5.3 must still validate identity, and the M5.5 transaction must still validate the replica before publication.

A small expected-cost model makes the save-versus-discard choice auditable. For reuse probability p, idle duration D, history size M, host resume latency T_resume, suffix GPU time G_suffix, full recompute service/GPU time T_full,G_full, and explicit cost weights:

$$\begin{aligned} J_{persist}={}&p(c_tT_{resume}+c_gG_{suffix}+c_xM) +c_tT_{save,visible}+c_xM+c_hMD,\\ J_{discard}={}&p(c_tT_{full}+c_gG_{full}). \end{aligned}$$

The fixture’s synthetic weights yield J_persist=3.03785 and J_discard=7.868, so host persistence wins. These numbers are a deterministic policy example, not a trained predictor or a production pricing claim.

0x04K Truncation Is A Representation Contract

Truncating old tokens can change the surviving tokens’ logical positions. If position information has already been embedded in cached keys, simply slicing the tensor does not necessarily produce the K/V that a fresh run would create at shifted positions.

CachedAttention addresses this in its paper by storing K/V before applying its relative positional transform, truncating that stored representation, and applying positions after load. That is a paper-specific alternative design, not permission to relabel arbitrary cached tensors.

The current Loop 2 fixture declares:

representation: full-kv-fp16-post-rope-v1
position rule: immutable logical positions; no shift

Therefore its exact behavior is:

same surviving rows, same logical positions:
    restore is eligible after all identity checks

truncation shifts surviving positions:
    reject this snapshot for exact shifted reuse
    recompute, or use a separately declared pre-position representation

M5.4-selected sparse or merged support:
    consume M5.4's representation and recoverability receipt
    do not call it full-support exact restoration

Moving the positional transform changes the cached ABI, kernels, save/load path, and compatibility key. It cannot be introduced as a hidden migration of an existing post-position snapshot.

0x04L InferCept: Preserve, Swap, Or Recompute

An intercepted request pauses while an external operation runs. Keeping its K/V on GPU avoids recomputation but strands scarce capacity. Discarding it releases capacity but later repeats forward work. Swapping avoids both long retention and full recomputation, but consumes link time, host capacity, and copy scheduling.

InferCept compares these paths through GPU-memory waste. Let:

C_i^j      request i context tokens at interception j
C_other    context tokens of other running requests
C_batch    C_i^j + C_other in the simple batch ledger
M          K/V bytes per context token
T_fwd(C)   forward/recompute time for C tokens
T_INT^j    interruption duration
T_swap(C)  one-direction swap time for C tokens

Discard wastes the request’s own K/V capacity during recomputation and delays other resident requests for the same added iteration time:

$$\begin{aligned} W_{discard,i}^{j} ={}&T_{fwd}(C_i^j)C_i^jM +T_{fwd}(C_i^j)C_{other}M\\ ={}&T_{fwd}(C_i^j)(C_i^j+C_{other})M. \end{aligned}$$

Preserve holds the interrupted context for the pause duration:

$$W_{preserve,i}^{j}=T_{INT}^{j}C_i^jM.$$

A naive synchronous round-trip swap pays out and in, while blocking the batch in this paper-level ledger:

$$W_{swap,i}^{j} = 2T_{swap}(C_i^j)C_{batch}M.$$

These quantities have units byte-ms. They are not transfer bytes and not latency by themselves. They provide one common objective for capacity stranded over time.

InferCept then divides recomputation into n chunks. Its paper model halves the request’s average self-occupancy during reconstruction and adds the delay experienced by other contexts during each chunk:

$$W_{chunkD,i}^{j} = \frac{T_{fwd}(C_i^j)C_i^jM}{2} +nT_{fwd}\!\left(\frac{C_i^j}{n}\right)C_{other}M.$$

The preserve-versus-chunked-discard crossing follows by setting their wastes equal and dividing by C_i^jM:

$$T_{INT}^{*} = \frac{W_{chunkD,i}^{j}}{C_i^jM}.$$

Below this duration, preserve strands fewer byte-milliseconds under the model; above it, chunked recomputation has lower potential waste unless swap budget intervenes.

0x04M Hiding Swap And Bounding Recomputation

InferCept pipelines swap at layer granularity: while the runtime launches the copy for layer l+2, it moves layer l+1, and foreground work can use memory freed by layer l. It also chunks a context across scheduler iterations so a single long swap does not monopolize the link.

In a simplified bandwidth model, choose the per-iteration swap limit N_i so that swap service fits under foreground forwarding:

$$T_{swap}(N_i)\le T_{fwd}(B_i).$$

With sustained bandwidth BW, per-token K/V bytes M, foreground interval T_fg, and no fixed cost in the hidden portion:

$$N_i = \left\lfloor\frac{T_{fg}BW}{M}\right\rfloor.$$

The fixture uses T_fg=0.8 ms, BW=20 GiB/s, and M=16 KiB/token, giving N_i=1,048 tokens/iteration. A 2,048-token context therefore needs two swap iterations. Fixed copy costs and paged physical fragmentation still consume budget; a production solver must use measured T_swap, as the paper does.

Chunked recomputation has a different limit. InferCept profiles a GPU saturation point S, the number of query tokens that fills useful compute. For a running group of size B_running, a bounded recompute chunk is

$$C_{chunk}\le\max(0,S-B_{running}).$$

The fixture’s S=6,144 and B_running=4,096 leave room for at most 2,048 recompute tokens in that iteration. This is a scheduling input, not a new M5.2 fairness policy.

0x04N Multiple Interrupted Requests Need A Joint Decision

The action cannot be chosen independently for every request when swap bandwidth and host capacity are shared. InferCept defines each request’s potential waste before swap as

$$W_i^j=\min(W_{preserve,i}^j,W_{chunkD,i}^j),$$

sorts requests by this value, and spends bounded swap-out capacity on the highest-potential requests. Requests that do not receive swap budget fall back to preserve or chunked discard according to the lower modeled waste.

For the deterministic fixture:

The crossing is 8.7 ms. The naive round-trip swap ledger is 334,705,459.2 byte-ms; InferCept’s chunking and overlap are precisely why the runtime should not implement that synchronous baseline.

The paper dynamically estimates an unknown interruption duration as

$$\widehat{T}_{INT}=t_{now}-t_{call},$$

so a request can move from an initially reasonable preserve choice toward swap or discard as the pause ages. That estimate is censored elapsed time, not a forecast of when the external result will arrive. The decision must therefore remain revisable and budget-aware.

The scheduling handoff must preserve the request’s original arrival time. Resetting it at resume makes every interruption look like a new request and can induce starvation. InferCept separates running, swap, and waiting queues; M5.2 owns how such age, slack, and fairness signals compose with the rest of the service.

0x04O Resume Is Validate, Acquire, Repair, Then Publish

The complete control path needs to handle early return, duplicate external results, partial saves, cancellation, and generation changes:

def pause_for_external_call(request, call_id, action):
    quiesce_at_committed_frontier(request)
    manifest = build_pause_manifest(
        identity=request.exact_cache_identity,
        frontier=request.committed_frontier,
        sampler=request.sampling_state,
        call_id=call_id,
        original_arrival=request.original_arrival,
        contains_uncommitted_branch_rows=False,
    )

    if action == PRESERVE:
        hold_gpu_leases(manifest)
    elif action == SWAP:
        manifest.save_txn = start_versioned_chunked_save(manifest)
    elif action == DISCARD:
        retain_tokens_and_recompute_inputs(manifest)
        release_gpu_only_after_recovery_inputs_are_durable(manifest)

    return manifest


def resume_after_external_result(manifest, result):
    assert result.call_id == manifest.external_call_id
    assert manifest.resume_lifecycle_epoch == manifest.pause_lifecycle_epoch + 1

    # Durable CAS is unique on (call_id, pause_lifecycle_epoch). The first
    # reservation binds result_id and its digest inside that receipt.
    receipt = reserve_result_application(
        scope=(manifest.external_call_id, manifest.pause_lifecycle_epoch),
        proposed_result_id=result.id,
        proposed_result_digest=digest(result.tokens),
    )
    if receipt.result_id != result.id:
        raise CompetingExternalResult(receipt.result_id, result.id)
    if receipt.state == APPLIED:
        return IDEMPOTENT_NOOP
    if receipt.state == RESULT_APPEND_COMMITTED:
        mark_receipt_applied(receipt)
        return IDEMPOTENT_NOOP

    try:
        if manifest.save_txn is still IN_FLIGHT:
            # Coalesce with the existing authority; never restore a partial save.
            await_or_cancel_without_losing_last_authority(manifest.save_txn)

        if manifest.state == PAUSED_GPU:
            cache = validate_gpu_generation(manifest)
        elif manifest.state == PAUSED_HOST:
            cache = restore_validate_and_publish(manifest)
        elif manifest.state == TOKEN_ONLY:
            cache = chunk_recompute_to_committed_frontier(manifest)
        else:
            raise InvalidLifecycleState(manifest.state)

        validate_identity_positions_frontier_checksum(cache, manifest)
        advance_receipt(receipt, RESERVED, CACHE_READY_AND_VALIDATED)

        # Build the result suffix privately. Publication of its token/K/V
        # frontier and the durable receipt transition is one transaction.
        extension = prefill_result_into_private_extension(cache, result.tokens)
        commit_result_extension_and_receipt_atomically(
            cache=cache,
            extension=extension,
            receipt=receipt,
            expected_state=CACHE_READY_AND_VALIDATED,
            next_state=RESULT_APPEND_COMMITTED,
        )
    except Exception:
        # A pre-commit failure leaves no published result suffix. The receipt
        # remains retryable instead of consuming the exactly-once key.
        mark_receipt_retryable_if_append_not_committed(receipt)
        raise

    mark_receipt_applied(receipt)
    enqueue_with_original_arrival(manifest.original_arrival)

An inactive completed turn uses the same restore validation but prefills the new user-turn suffix instead of an external result. If only part of a snapshot is valid, the runtime must either combine that exact prefix with a declared recompute suffix or reject exact restoration. It cannot publish the partial object as full support.

The result receipt is a state machine, not an eager Boolean claim:

UNSEEN -> RESERVED -> CACHE_READY_AND_VALIDATED
       -> RESULT_APPEND_COMMITTED -> APPLIED

restore or validation failure:
RESERVED -> RETRYABLE -> RESERVED -> ... -> APPLIED

same result after RESULT_APPEND_COMMITTED:
finalize APPLIED without appending again

same result after APPLIED:
idempotent no-op

different result for the reserved call:
reject as a competing result

RESULT_APPEND_COMMITTED means the result-token suffix, its K/V rows, its new committed frontier, and that receipt state became durable atomically. A crash after that transaction but before APPLIED is acknowledged can finalize the receipt; it must not run the result prefill a second time. This transaction protocol is a rewrite interpretation for runtime correctness, not a claim that CachedAttention or InferCept specifies this exact receipt schema.

0x04P Failure Lab And Required Telemetry

At minimum, observe:

paused sessions and bytes by state/tier/action
save/load bytes, service time, visible stall, and queue time
resume source: GPU / host / disk / recompute
exact-prefix miss versus replica/position/checksum rejection
time paused, time since external call, and original scheduling age
swap/recompute chunks per iteration and credits consumed
preserve/swap/discard decisions plus decision revisions
duplicate, stale-generation, cross-tenant, and partial-publication rejects

0x04Q Executable Reconciliation And Claim Status

The Loop 4 fixture and oracle are reproducible with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/inactive_interrupted_lifecycle_oracle.py

They reconcile:

inactive history:                         48 MiB
layerwise host resume pipeline:           2.803125 ms
host / disk request-visible resume:       3.483125 / 15.483125 ms
host decision / disk decision:            restore / recompute
short / long interrupted action:          preserve / swap if budgeted
swap limit:                               1,048 tokens per iteration
invalid one-field cases:                  33 / 33 rejected as expected

Loop 5 receives a bounded next question: if moving all exact K/V back to the GPU is too expensive, can full-support attention run near the tier that owns those rows and return a numerically mergeable partial state? Inactive-session policy, interrupted-request scheduling, and position-compatible snapshot publication are closed here rather than being silently delegated to that compute-placement loop.

0x04R Compute-Near-Data Is A Different Placement Contract

“Offloaded attention” can name three materially different schedules:

move-to-compute:
    K/V history -> GPU staging buffer -> GPU attention

whole-request compute-near-data:
    current Q/K/V -> CPU or storage-side engine
    attention output -> GPU continuation

support-partitioned attention:
    each tier scans a disjoint part of the same logical history
    mergeable online-attention states -> one exact output

The first schedule changes the working-set granularity but not the device that evaluates attention. The second changes the device for a whole request. The third divides one head’s support across devices and therefore needs an exact reduction contract. These distinctions prevent three common errors:

HeadInfer head-wise staging       != CPU attention
NEO CPU-request placement         != splitting one request across CPU and GPU
InstInfer in-storage execution    != exact dense attention under SparF

The owner boundary is equally important. M5.5 owns the tier, transport, receipt, deadline, and fallback contract. M4.3 derives why the online state is mergeable, while M4.5 owns a production sparse, split-K, indirect, or heterogeneous kernel. M5.6 owns routing between distinct prefill and decode workers. This section does not silently absorb any of those implementations.

0x04S Derive The Traffic Before Choosing The Device

For one layer with batch B, history length T, H_kv K/V heads, head width d_h, and b_kv bytes per stored scalar, full historical K/V occupies:

$$M_{KV} = 2BTH_{kv}d_hb_{kv}.$$

Moving the full cache to the GPU for one decode row transfers approximately:

$$M_{move} = M_{KV}+M_{append}, \qquad M_{append}=2BH_{kv}d_hb_{kv}.$$

Now suppose the tier that owns history computes attention. For ordinary MHA with H_q query heads and activation width b_a, moving current projections to that tier and returning the layer attention output transfers:

$$M_{near} = M_Q+M_{append}+M_O,$$ $$M_Q=BH_qd_hb_a, \qquad M_O=BH_qd_hb_a.$$

The synthetic Loop 5 fixture uses T=4096, H_q=H_kv=8, d_h=128, and BF16-sized payloads. It therefore reconciles:

full historical K/V moved to GPU:       16,777,216 B   (16 MiB)
one head-sized HeadInfer staging group:  2,097,152 B    (2 MiB)
all eight staged head groups per layer: 16,777,216 B   (16 MiB)
Q + current K/V + returned O:                 8,192 B    (8 KiB)

Head-wise staging reduces peak GPU residency from a layer-sized K/V buffer to a group-sized buffer. It does not by itself reduce the total historical bytes read for exact dense attention. Compute-near-data removes the history from the interconnect path, but adds tier-side compute, synchronization, and a small-tensor transport path whose fixed overhead can dominate its byte count.

An exact support-partition path returns one online-softmax state per producer, not one state for the whole request. With P producers and FP32 state scalars:

$$M_{state,one}=BH_q(2+d_v)b_s, \qquad M_{state,return}=P\,M_{state,one}.$$

The production-shaped traffic fixture has P=2, H_q=8, d_v=128, and b_s=4, so each receipt is 4,160 B and the aggregate return is 8,320 B. Counting only 4,160 B would silently omit one producer. The later four-row numerical merge is a separate reduced oracle query: it proves the algebra and receipt checks without pretending that four toy rows cover this 4096-token traffic request.

For a send and return with direction-specific fixed costs:

$$t_{near,xfer} = \tau_{send}+\frac{M_Q+M_{append}}{BW_{send}} + \tau_{return}+\frac{M_O}{BW_{return}}.$$

At a synthetic 24 GB/s in both directions and 0.02 ms fixed cost per direction, the fixture obtains 0.040341 ms. That number is not a CPU-attention latency. It only proves that the payload ledger is small; the tier-side scan still has to meet the iteration deadline.

0x04T The Papers Change Different Axes

This table is a paper-by-contract comparison, not a performance ranking. The systems target different hardware, workloads, and execution objects.

0x04U HeadInfer: Smaller Exact Staging, Not Tier-Side Attention

For one head h:

$$K^{(h)},V^{(h)}\in\mathbb{R}^{B\times T\times d_h}, \qquad O^{(h)} = \operatorname{softmax} \left(\frac{Q^{(h)}K^{(h)\top}}{\sqrt{d_h}}\right)V^{(h)}.$$

Heads are independent until their outputs are concatenated. HeadInfer exploits that axis by preallocating CPU history plus ping-pong GPU buffers and executing:

for layer in layers:
    for group_id, heads in enumerate(adaptive_head_groups):
        wait(prefetch_done[group_id])
        kv = gpu_ping_pong[group_id % 2]

        async_prefetch(
            cpu_kv[layer, next_group(group_id)],
            gpu_ping_pong[(group_id + 1) % 2],
        )
        head_outputs[heads] = gpu_attention(q[heads], kv, full_support=True)
        async_append_new_kv(cpu_kv[layer, heads], new_kv[heads])

    layer_output = concatenate_heads_in_model_order(head_outputs)

The ordering requirement is head identity, not an online-softmax merge: every head owns a different output slice. Adaptive grouping trades launch count and staging bytes. With g K/V heads per group:

$$M_{stage}(g)=2BTgd_hb_{kv}, \qquad N_{groups}=\left\lceil\frac{H_{kv}}{g}\right\rceil.$$

Smaller g lowers peak staging but increases launches and synchronization. Larger g approaches layer-wise offload. The HeadInfer paper’s grouping thresholds and million-token results are source-reported for its evaluated models and hardware; they are not universal scheduler constants.

Source figure: HeadInfer Figure 4. The figure establishes that head-sized K/V still crosses PCIe and attention still executes on GPU; it is not evidence for CPU-side attention.

0x04V NEO: Place Requests, Then Build An Asymmetric Pipeline

NEO makes a different decision. After prefill, a request’s complete cache is classified as a GPU-cache or CPU-cache resident object. A CPU-request sends its current projections to a CPU attention kernel and returns the attention output, while GPU linear work for other requests overlaps that scan.

The paper’s two-sub-batch transformer-layer estimate is:

$$T_{tr} \approx L\left[ \max\left(T_{l0},T_{ca1}\right) + \max\left(T_{l1}+T_{ga0},T_{ca0}\right) \right],$$

where T_lx is linear work, T_cax CPU attention, and T_ga0 GPU attention for sub-batch x. The maxima appear because the corresponding CPU and GPU stages overlap; adding all stages would count hidden work as exposed latency. NEO attempts to maintain:

$$T_{ca1}\le T_{l0}, \qquad T_{ca0}\le T_{l1}+T_{ga0}.$$

The synthetic four-layer schedule uses:

T_l0=0.65 ms, T_ca1=0.55 ms
T_l1=0.22 ms, T_ga0=0.28 ms, T_ca0=0.45 ms

per layer = max(0.65,0.55) + max(0.22+0.28,0.45)
          = 1.15 ms
four layers = 4.60 ms

Those numbers only reconcile the equation. NEO estimates stage times from offline profiles and makes an iteration-level choice between a GPU-only and a heterogeneous schedule. M5.2 still owns fairness, admission, and global queue policy; M5.5 emits tier feasibility and predicted service receipts.

Source figure: NEO Figure 5. Batch 0 and batch 1 are intentionally asymmetric; the diagram does not depict one request’s historical support split between CPU and GPU.

0x04W An Exact Support Split Must Return A Mergeable State

Some heterogeneous systems may divide one head’s allowed history into pairwise-disjoint sets S_1,...,S_P. This is a general exact design contract, not a claim that HeadInfer or NEO uses that cross-device partition.

For partition p, compute scores s_i=qk_i^T/sqrt(d_h) and return:

$$m_p=\max_{i\in S_p}s_i,$$ $$l_p=\sum_{i\in S_p}\exp(s_i-m_p),$$ $$u_p=\sum_{i\in S_p}\exp(s_i-m_p)v_i.$$

For two receipts a and b, choose the common maximum:

$$m=\max(m_a,m_b),$$

then rescale both local coordinate systems:

$$l=e^{m_a-m}l_a+e^{m_b-m}l_b,$$ $$u=e^{m_a-m}u_a+e^{m_b-m}u_b, \qquad o=\frac{u}{l}.$$

M4.3 proves closure and merge-order equivalence in exact arithmetic. M5.5 adds the transport receipt that a remote partition must publish:

tenant, request_id, request_generation, checkpoint_generation
layer, query_head, kv_head, query_to_kv_mapping_id, query_position
support_descriptor and support_digest
mask_id, position_schema_id, score_scale
state_schema = fp32-m-l-u-v1
m: fp32 scalar
l: fp32 scalar
u: fp32[d_v]
producer_generation, checksum, completion_event, deadline

The coordinator must reject overlapping support, missing support, mismatched query identity, an output-only receipt without normalization mass, or a late receipt whose fallback has already published. In finite precision, legal merge orders can differ slightly; the numerical tolerance is part of the backend contract rather than proof that arbitrary dtypes are interchangeable.

The executable receipt repeats the tenant, request, checkpoint, layer, query head, K/V head, query-to-K/V mapping, query position, mask, positional schema, score scale, state schema, support descriptor, and SHA-256 support digest. This is deliberate duplication across a trust boundary: a coordinator validates the returned state against the query it is reducing rather than inferring identity from arrival order or producer placement.

0x04X InstInfer Moves Compute Near Flash, But SparF Is Approximate

InstInfer places decode attention and the large cache in computational storage. The GPU retains projections, output projection, and FFN. During decode it sends current q,k,v, appends new K/V through peer-to-peer DMA, and receives the attention output. This is genuine compute-near-data placement.

However, the paper’s SparF path is not the full-support state merge above. It:

1. selects r salient query channels;
2. reads hidden-dimension-indexed K pages;
3. estimates token importance;
4. selects k token positions;
5. reads token-indexed full K/V pages;
6. computes attention on selected rows;
7. applies a value-mean correction to omitted mass.

Flash page granularity creates two-stage filtering: a page is fetched when it contains at least one selected unit, then near-flash filters remove unwanted units. This trades duplicated/indexed K layout and approximate support for lower flash traffic. Calling it “exact because compute stayed beside K/V” would confuse placement with semantics.

Source figure: InstInfer Figure 7. The architecture demonstrates in-storage decode attention. SparF’s selected support remains an approximation and is derived with selector methods in the following loops.

0x04Y Runtime Selection Needs Deadlines And A Semantic Fallback

def choose_attention_placement(req, layer, manifest, profile, deadline):
    candidates = []

    if manifest.full_kv_h2d_feasible(layer, deadline):
        candidates.append(plan_exact_move_to_gpu(req, layer))

    if manifest.head_group_staging_supported(layer):
        candidates.append(plan_exact_headwise_gpu(req, layer))

    if manifest.cpu_full_request_supported(req, layer):
        candidates.append(plan_exact_cpu_request(req, layer))

    if manifest.exact_partition_receipts_supported(layer):
        candidates.append(plan_exact_support_partition(req, layer))

    if req.approximation_contract.allows("sparf"):
        candidates.append(plan_approximate_in_storage(req, layer))

    feasible = [p for p in candidates if p.predicted_finish <= deadline]
    chosen = min(feasible, key=lambda p: p.declared_cost) if feasible else None

    if chosen is None:
        return req.fallback  # delay, exact reroute, recompute, or explicit reject
    return reserve_validate_then_publish(chosen, manifest.generation)

The exact path cannot silently fall through to SparF or another selector. The request must opt into approximation, and telemetry must retain both attempted and published plans. Required observations include:

placement candidate and rejection reason
history bytes, Q/K/V bytes, returned-state bytes, physical read bytes
tier queue, send, attention, reduction, return, and uncovered-stall time
support digest, receipt count, duplicate/overlap/gap rejects
dtype and merge-order tolerance
deadline miss, fallback action, and publication generation

0x04Z Executable Reconciliation And Loop 6 Handoff

The Loop 5 fixture and oracle are reproducible with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/heterogeneous_attention_oracle.py

They reconcile the 16 MiB full-history transfer, 2 MiB K/V-head staging window, 8 KiB near-data request payload, two 4,160 B partial-state receipts, NEO’s overlap equation, and dense attention against sequential (m,l,u) merges. A separate asymmetric GQA sensitivity case proves that staging-group count follows H_kv, not H_q. Invalid one-field fixtures exercise support gaps, overlap, generation, complete receipt identity, dtype, traffic aggregation, deadline, publication, and paper-classification failures.

The next selector loop receives a sharper question: when exact full-support movement or tier-side scanning misses the budget, how do SparQ, Quest, and InfiniGen derive different proxy scores from query dimensions, page extrema, and adjacent-layer rehearsal? Loop 6 must derive each geometry rather than replacing all three with generic top-k pseudocode.

0x05 Retrieval Changes The Attention Set

Exact offload preserves the set of historical rows. Retrieval attention chooses a subset.

For one head, full attention over historical set S is:

$$o = \sum_{i\in S}p_i v_i, \qquad p_i = \frac{\exp(s_i)}{\sum_{j\in S}\exp(s_j)}, \qquad s_i=\frac{qk_i^\top}{\sqrt{d_h}}.$$

Let retrieval return R subseteq S, and let O=S\R be omitted rows. Running softmax only over R gives:

$$o_R = \sum_{i\in R} \frac{p_i}{1-\delta}v_i, \qquad \delta=\sum_{i\in O}p_i.$$

delta is omitted attention mass, not omitted-token fraction.

0x05A Why Recall Alone Is Not The Error Contract

Define the full distribution’s conditional output over omitted rows as:

$$o_O = \sum_{i\in O}\frac{p_i}{\delta}v_i.$$

Then:

$$o=(1-\delta)o_R+\delta o_O,$$

so:

$$o-o_R=\delta(o_O-o_R).$$

If every value vector has norm at most V_max, both conditional averages have norm at most V_max, and:

$$\lVert o-o_R\rVert \le 2\delta V_{max}.$$

This derivation explains why these metrics are not interchangeable:

A retrieval system can miss many low-mass rows and remain accurate, or miss one high-mass row and fail badly. Report recall and mass/error metrics together.

0x05B Thresholds Convert Score Margins Into Work

Suppose a system keeps rows whose score is within alpha of the maximum:

$$s_i\ge s_{max}-\alpha.$$

For an omitted row at the boundary, its unnormalized weight is at most exp(-alpha) times the maximum row. At alpha=5, that ratio is about 0.0067. But many omitted rows can accumulate nontrivial total mass, so this relative bound is not by itself a global error guarantee.

The threshold also creates variable work:

sharp query distribution  -> few retrieved rows -> low traffic
flat query distribution   -> many retrieved rows -> high traffic

A fixed threshold therefore needs a maximum candidate budget and an explicit fallback when the budget is exceeded.

0x05C Retrieval Fallbacks Have Different Semantics

increase K and rerun retrieval:
    still approximate, but may reduce omitted mass

read full exact history and rerun attention:
    restores the exact attention-set contract at added latency

recompute K/V then rerun full attention:
    exact under compatible model state, usually more expensive

continue with the incomplete subset:
    accepts an approximation and must be evaluated as such

reject or degrade the request:
    preserves a declared service contract rather than silently changing output

An exact-offload miss is primarily a latency, admission, or availability failure when a valid exact fallback exists. A retrieval miss is a model-semantic change unless the attention step is rerun over the full required history.

0x05D One Word, Three Proxy Axes

“Query-aware top-k” is not an algorithm. A selector must say which axis it reduces before it ranks anything:

The comparison also separates metadata exactness from support exactness. Quest’s page score is a valid upper bound. InfiniGen’s shared orthogonal transform preserves a full dot product. Neither fact proves that a finite selected support equals dense attention.

0x05E SparQ: Select Dimensions Before Selecting Tokens

SparQ starts from two empirical observations: attention mass is often concentrated over sequence positions, and the magnitude distribution inside a decode query is often heavy-tailed. The first observation motivates reading only selected value rows. The second is needed to avoid reading every key row merely to discover those positions.

For one query q in R^(d_h), choose the indices of its r largest-magnitude components:

$$I_r=\operatorname{TopR}\left(|q|,r\right).$$

Only those dimensions are read across the historical key matrix:

$$K_{I_r}=K[:,I_r] \in\mathbb{R}^{T\times r}.$$

This is a column gather over K. The later full-row K/V fetch is a row gather. A layout that is efficient for one need not be efficient for the other, which is why implementation storage belongs in the contract rather than in a kernel footnote.

0x05F Derive The Approximate Distribution And Temperature

SparQ forms proxy logits from the selected query dimensions:

$$\widehat z_i = \frac{q_{I_r}k_{i,I_r}^{\top}}{\tau}, \qquad \widehat s=\operatorname{softmax}(\widehat z).$$

Simply replacing sqrt(d_h) with sqrt(r) assumes randomly selected channels with comparable energy. Top-magnitude selection violates that assumption. If the selected coordinates retain almost all query mass, scaling as though only r random coordinates survived makes the proxy too sharp. SparQ interpolates between those regimes with its L1-coverage temperature:

$$\tau = \sqrt{ d_h \frac{\lVert q_{I_r}\rVert_1}{\lVert q\rVert_1} }.$$

The limits make the design readable:

selected L1 coverage near 1:
    tau approaches sqrt(d_h)

uniform energy and r of d_h dimensions retained:
    coverage is about r / d_h
    tau approaches sqrt(r)

For grouped-query attention, multiple query heads share one K/V head. The paper does not independently choose incompatible K/V rows for each query head. It aggregates query magnitude within the group before selecting dimensions, then aggregates proxy scores before selecting shared token positions:

$$I_r^{(g)} = \operatorname{TopR}_j \left( \sum_{h\in g}|q_{h,j}|, r \right).$$

0x05G The Second Gather Recomputes Full Scores

Let R contain the k largest proxy scores, optionally unioned with a local window of ell recent positions. SparQ then fetches full-dimensional rows:

$$K_R,V_R\in\mathbb{R}^{k\times d_h},$$

and recomputes attention scores with all channels:

$$s_R = \operatorname{softmax} \left( \frac{qK_R^\top}{\sqrt{d_h}} \right), \qquad y_R=s_RV_R.$$

The recomputation is exact over R. Candidate selection remains approximate relative to the complete history.

The paper’s local-window rule is part of support construction, not a second attention result:

proxy = softmax(proxy_logits)
recent = {T - ell, ..., T - 1}
candidate_score = proxy.copy()
candidate_score[list(recent)] += 1.0
selected = topk(candidate_score, k)  # requires ell <= k

Because each proxy probability lies in [0,1], adding one forces legal recent positions ahead of non-local positions. A production implementation should form an explicit union or use an exact mask contract rather than relying on an undocumented magic constant.

0x05H Mean-Value Reallocation Estimates Omitted Mass

Sparse renormalization assigns all probability to fetched rows. SparQ instead uses the proxy distribution to estimate how much full-distribution mass belongs to R:

$$\widehat\alpha = \sum_{i\in R}\widehat s_i.$$

Maintain the running mean value:

$$\bar v_T=\frac{1}{T}\sum_{i=1}^{T}v_i, \qquad \bar v_{T+1} = \frac{T\bar v_T+v_{T+1}}{T+1}.$$

Then interpolate:

$$y_{\text{SparQ}} = \widehat\alpha y_R + (1-\widehat\alpha)\bar v_T.$$

This is not an exact correction distribution. It assumes the omitted values are usefully represented by their running mean and that the proxy mass estimate is calibrated. The Loop 6 toy deliberately misses one exact top token. Its uncorrected attention-output maximum error is about 0.3023; mean reallocation reduces that toy error to about 0.1967, but does not recover dense attention.

Source: SparQ Figure 5 and Algorithm 1. The source figure is an algorithm contract, not evidence that a particular runtime has implemented every step.

0x05I Transfer, Layout, And Paper-To-Runtime Variants

For one head, the SparQ paper counts scalar-element traffic as:

$$M_{\text{dense}}=2Td_h+2d_h,$$ $$M_{\text{SparQ}}=Tr+2kd_h+4d_h.$$

The terms are structurally different:

T r:
    dimension-sliced historical K read for proxy scoring

2 k d_h:
    selected full K and V rows

4 d_h:
    append current K/V plus read/update the running value mean

With T=4096, d_h=128, r=32, and k=128, the oracle obtains 1,048,832 dense elements versus 164,352 SparQ elements, a ratio of about 0.1567. On its five-token teaching fixture, SparQ traffic is slightly larger than dense traffic because fixed overhead dominates. Sparsity therefore has a sequence-length break-even point.

The pinned Graphcore llama.cpp fork exposes another important boundary:

This is not a defect in either source. It is a paper-variant comparison: the tutorial/runtime fork implements a simpler two-stage method and explicitly omits the correction discussed by the paper.

0x05J Quest: Bound A Page Without Reading Its Rows

Quest chooses pages rather than individual tokens. Let page p contain keys k_i and maintain coordinate extrema:

$$m_{p,j}=\min_{i\in p} k_{i,j}, \qquad M_{p,j}=\max_{i\in p} k_{i,j}.$$

For current query q, define:

$$U_p(q) = \sum_{j=1}^{d_h} \max(q_jm_{p,j},q_jM_{p,j}).$$

This expression is not merely a learned importance score. It is an upper bound on every unscaled query-key dot product in that page.

0x05K Prove The Coordinate-Box Upper Bound

For any k_i inside page p:

$$m_{p,j}\le k_{i,j}\le M_{p,j}.$$

If q_j >= 0, multiplication preserves order and:

$$q_jk_{i,j}\le q_jM_{p,j}.$$

If q_j < 0, multiplication reverses order and:

$$q_jk_{i,j}\le q_jm_{p,j}.$$

Therefore, in both cases:

$$q_jk_{i,j} \le \max(q_jm_{p,j},q_jM_{p,j}).$$

Summing coordinates proves:

$$qk_i^\top\le U_p(q) \qquad\text{for every }i\in p.$$

The bound is equivalent to sign-normalizing the query. Define a_j=|q_j| and choose e_{p,j}=M_{p,j} for nonnegative q_j, otherwise e_{p,j}=-m_{p,j}, the page maximum of the sign-flipped coordinate -k_{i,j}, so that a_j e_{p,j}=q_j m_{p,j}. Then U_p(q)=a e_p^T. This is the form used by the official Quest reference tests.

0x05L Append And Decode Are Different Kernel Paths

Page metadata changes on every committed K append:

def append_key(page_meta, key):
    page_meta.minimum = minimum(page_meta.minimum, key)
    page_meta.maximum = maximum(page_meta.maximum, key)

def quest_decode(query, page_table, page_meta, page_budget):
    bounds = [
        sum(max(q * low, q * high)
            for q, low, high in zip(query, meta.minimum, meta.maximum))
        for meta in page_meta
    ]
    selected_pages = topk(bounds, page_budget)
    return paged_attention(query, page_table.gather(selected_pages))

The official Quest repository keeps the physical K/V pool and the min/max metadata pool as separate paged objects. Its checked decode controller excludes the mutable last page from top-k and reserves one page-budget slot for it. That is a runtime policy, not a term in the mathematical upper-bound proof.

The paper also applies dense attention in its first two layers because its measured sparsity was lower there. Treat that as a source-reported evaluation policy. A different checkpoint needs its own layer-sensitivity audit.

Source: Quest Figure 5. The green checks and red cross illustrate the paper’s page ranking example; they do not prove that a finite page budget always contains the true highest-scoring token.

0x05M A Valid Bound Can Still Overfetch And Miss

The extrema describe a coordinate box. Its corner maximizing q may not be an actual token because each coordinate maximum can come from a different row. Thus:

$$\max_{i\in p}qk_i^\top \le U_p(q),$$

but the gap can be large. Loose bounds create page overfetch. More subtly, top-K by upper bound can omit the page with the largest actual token score if K other pages have looser bounds.

The oracle makes that failure concrete:

Every bound is valid, yet a one-page budget misses page 2. Raising the budget to two pages includes it. This is why “upper bound” and “no false negatives at fixed budget” are different claims.

If every full K or V vector occupies M bytes, there are P pages of B_p tokens, and K_p pages are selected, the paper’s normalized read fraction is:

$$\rho_{\text{Quest}} \approx \frac{1}{B_p} + \frac{K_p}{P}.$$

The first term reads two reduced vectors per page; the second reads full K/V for selected pages. Smaller pages tighten bounds and reduce overfetch, but increase page count, metadata, top-k work, and address traversal. M5.3 owns the allocator and page table; Quest consumes that physical granularity.

0x05N Loop 6 Executable Reconciliation

The local selector fixture can be reproduced with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/selector_geometry_oracle.py

It checks:

SparQ:
    selected dimensions, coverage temperature, local-token union,
    candidate mismatch, mean correction, and transfer break-even

Quest:
    per-coordinate extrema, bound validity, looseness,
    fixed-budget miss, and metadata plus selected-page traffic

InfiniGen:
    two-dimensional SVD/right-singular basis reconstruction,
    prefill column-evidence selection, orthogonal full-score invariance,
    adjacent-layer partial score, threshold/cap selection, and a miss

The three selector cases intentionally use different synthetic support lengths; they share checkpoint/layer metadata but are not presented as one request with three incompatible committed frontiers. All 31 one-field-invalid fixtures are rejected before selector state is used. The oracle reproduces no paper benchmark and runs no paper kernel. It tests the mathematical and object-lifecycle contracts used in this article.

Loop 6 ends at proxy geometry and full-row/page gathering. Product-quantized indexes, attention-aware ANN search, centroid hierarchies, and LSH sampling do not become interchangeable merely because they also return candidate IDs. Loop 7 owns their build, update, query, estimator, and stale-index contracts.

0x06 InfiniGen: Predict, Prefetch, Then Compute Full-Dimension Attention

A25 introduces InfiniGen as a concrete answer to the offload problem. Its system keeps a large K/V pool in host memory and attempts to predict, one layer early, which rows the next layer will attend to. Only those rows are prefetched to the GPU, where attention uses the normal full-dimensional K/V rows.

The mechanism has four distinct stages:

offline transform:
    rotate query/key channels into a basis with concentrated query energy

partial rehearsal:
    use a subset of rotated channels to estimate next-layer attention scores

selection and prefetch:
    choose token positions, gather their full K/V rows from host, issue transfer

full attention:
    compute the actual layer attention over the prefetched candidate set

Calling the first two stages “compressed attention” would be misleading. The partial channels are used to predict the set. The selected rows’ full K/V representations are still needed for the actual attention operation.

0x06A Orthogonal Skewing Preserves Full Scores

Let A be orthogonal, so:

$$AA^\top=I.$$

Rotate queries and keys with the same matrix:

$$\widetilde Q=QA, \qquad \widetilde K=KA.$$

Then:

$$\widetilde Q\widetilde K^\top = QAA^\top K^\top = QK^\top.$$

The paper folds the rotation into both projection weights once, offline:

$$W_Q'=W_QA, \qquad W_K'=W_KA.$$

Runtime projection then directly produces skewed Q and K; it need not materialize a second q @ A operation on every decode step. The rotation alone is exact. Approximation begins when the rehearsal retains only a subset D of rotated dimensions:

$$\widehat s_i = \widetilde q_D\widetilde k_{i,D}^\top.$$

The omitted score term is:

$$s_i-\widehat s_i = \widetilde q_{\bar D}\widetilde k_{i,\bar D}^\top.$$

By Cauchy-Schwarz:

$$|s_i-\widehat s_i| \le \lVert\widetilde q_{\bar D}\rVert_2 \lVert\widetilde k_{i,\bar D}\rVert_2.$$

The transform is useful when the query energy outside D is small for the workload used to choose the basis. This is the derivation behind the design: concentrate query energy, then make a low-dimensional rehearsal informative.

0x06B Why SVD Produces The Skewing Basis

Collect representative query rows for one layer/head into sample matrix Q_s and compute:

$$Q_s=U\Sigma V^\top.$$

Choose:

$$A=V.$$

Because V is orthogonal:

$$Q_sA=U\Sigma.$$

The singular values rank directions by energy in the calibration queries. Selecting leading columns of that rotated representation retains directions that explain the largest sampled query variation. It does not prove that every future query has the same concentration; calibration coverage remains part of the accuracy contract.

SVD chooses a basis, but it does not by itself choose the runtime partial columns. During prefill, InfiniGen takes absolute skewed query and key values, combines their column evidence, sums down the sampled token rows, and chooses one shared column set:

$$c_j = \sum_t \left( |\widetilde Q_{t,j}| + |\widetilde K_{t,j}| \right), \qquad D=\operatorname{TopR}(c,d_r).$$

Using the same D for the partial query weight and transposed partial key cache is mandatory: a query column multiplies the corresponding key column in the dot product. Independent query/key top-r sets would not define the intended rehearsal. The paper’s 30% partial-weight ratio is a source-reported setting, not an implementation law.

0x06C Tensor Trace Across Two Adjacent Layers

Consider one request at decode step t, one local K/V head, and adjacent layers l-1 and l. The critical temporal point is that the predictor does not wait for the actual layer-l attention input. It uses the layer-l-1 input as an approximation, together with partial layer-l state:

x_t^(l-1)                                      [1, d_model]
partial next-layer projection
    x_t^(l-1) @ W_Q'^(l)[:, D_l]                -> q_hat_t^(l)      [1, d_r]

host partial next-layer keys
    K'^(l)[:, D_l]                              -> K_hat^(l)        [T, d_r]
speculated next-layer scores
    q_hat_t^(l) @ K_hat^(l)^T                   -> s_hat^(l)        [1, T]
threshold and cap                              -> R_t^(l)          [K_fetch]

host full keys              K_host[R_t^(l)]         [K_fetch, d_h]
host full values            V_host[R_t^(l)]         [K_fetch, d_h]
H2D prefetch                -> GPU staging buffers
layer-l attention           -> output                [1, d_h]

The predictor can run while preceding work executes because layer adjacency provides a partial future query signal. The schedule is only beneficial if selection plus gather plus transfer complete before layer l needs the rows.

def speculate_next_layer(previous_layer_input, next_layer, host_pool, transfer_stream):
    dims = next_layer.partial_column_ids
    q_rehearsal = linear_selected_outputs(
        previous_layer_input,
        next_layer.skewed_q_weight,
        output_columns=dims,
    )
    k_rehearsal = host_pool.partial_keys(
        layer=next_layer.index,
        columns=dims,
    )

    approximate_scores = q_rehearsal @ k_rehearsal.transpose(-1, -2)
    threshold = approximate_scores.max(dim=-1) - next_layer.alpha
    selected_positions = threshold_and_cap(
        approximate_scores,
        threshold=threshold,
        max_candidates=next_layer.max_candidates,
    )

    transfer = transfer_stream.prefetch_full_kv_async(
        host_pool,
        layer=next_layer.index,
        positions=selected_positions,
        generation=host_pool.generation,
    )
    return selected_positions, transfer

The code trace deliberately separates partial_keys from full_kv. Accidentally feeding reduced rehearsal keys into the final attention kernel changes both shape and semantics.

The paper selects tokens above max(score)-alpha. It also equalizes candidate counts across heads by averaging how many positions fall above each head’s threshold, because parallel heads otherwise request different transfer sizes. Its evaluation caps transfers at 20% of the total K/V when more candidates pass. Both policies change work and potentially support, so they belong in a receipt:

layer, head group, partial-column generation,
alpha, uncapped candidate count, equalized count, cap,
selected token IDs, pool generation, prefetch completion event

InfiniGen starts speculation from layer 1 in the paper because the useful outlier structure emerges during layer 0. This is a paper-specific schedule, not proof that all models can skip prediction for exactly one layer.

0x06D Indexing, Gather, And Pool Management

Selected token IDs are not useful until they map to physical host rows. A practical host pool needs:

logical token/layer/head ID -> host chunk + row offset
validity bitmap or generation
rehearsal-key layout
full-key/full-value layout
gather descriptors
pinned/stageable destination policy
eviction/admission metadata

The original InfiniGen design discusses counter-, FIFO-, and LRU-style pool policies and selects a counter-based policy for its implementation trade-offs. These policies answer a different question than attention retrieval:

retrieval policy:
    which preserved rows should this query read?

pool policy:
    which rows should remain available in the finite host pool at all?

If the pool evicts a row, the retrieval stage cannot select it later. Pool recall is therefore an upper bound on query-time retrieval recall.

0x06E Source Figure Reading

Figures 8-10 expose three states that a single “predict important tokens” box would hide. The orthogonal weight transform is offline. The shared partial column IDs and partial key cache are produced during prefill. Decode then uses the preceding layer input with the next layer’s partial query weight before issuing the full-row prefetch. These are separate validity generations.

The system figure is valuable because it makes the temporal dependency visible: a partial computation derived from the preceding layer launches a prefetch for the next layer. Read the arrows as a pipeline, not merely as a memory hierarchy.

The latency breakdown should be read as source-reported evidence for the paper’s evaluated models, hardware, sequence lengths, and implementation. It is not a transferable latency promise. The reusable lesson is the decomposition: prediction, CPU-side selection/gather, transfer, exposed stall, and GPU attention must be measured separately.

0x07 RetrievalAttention: Index The Large Pool, Keep A Persistent GPU Set

RetrievalAttention starts from a related observation: for long contexts, only a sparse subset of K/V rows may receive substantial attention for a query, but finding that subset through a generic nearest-neighbor index is difficult because keys and queries do not necessarily share the same distribution.

Its architecture can be understood as two sets:

G_persistent:
    K/V rows kept on GPU because they are repeatedly or predictably important

C_dynamic(q):
    query-dependent rows retrieved from the CPU-side index

The attention step uses their union, with duplicate positions removed. The index is attention-aware rather than assumed to be an ordinary symmetric key/query vector-search problem.

0x07A Partial Softmax Results Can Be Combined Exactly Over Their Union

Partition a selected attention set into disjoint subsets A and B. For each subset, compute the numerically stable online-softmax state:

$$m_A=\max_{i\in A}s_i, \quad \ell_A=\sum_{i\in A}e^{s_i-m_A}, \quad u_A=\sum_{i\in A}e^{s_i-m_A}v_i,$$

and similarly (m_B, ell_B, u_B). Define:

$$m=\max(m_A,m_B).$$

Rescale both partitions into the shared exponent frame:

$$\ell = e^{m_A-m}\ell_A +e^{m_B-m}\ell_B,$$ $$u = e^{m_A-m}u_A +e^{m_B-m}u_B.$$

Then:

$$o=\frac{u}{\ell}$$

is exactly the softmax-weighted output over A union B, subject to ordinary floating-point differences. This is the same associativity principle used by tiled online softmax.

The important qualifier is:

combination can be exact over the selected union;
selection can still be approximate relative to the complete history.

0x07B CPU And GPU Work Need A Common Position Namespace

def retrieval_attention(q, gpu_set, cpu_index, budget):
    gpu_positions, gpu_kv = gpu_set.visible_rows(q.request_key)
    cpu_positions = cpu_index.search(q, budget=budget)

    cpu_positions = remove_duplicates(cpu_positions, gpu_positions)
    cpu_partial = cpu_attention_summary(q, cpu_positions)
    gpu_partial = gpu_attention_summary(q, gpu_kv)

    return merge_online_softmax_states(gpu_partial, cpu_partial)

Both partial outputs must use the same score scaling, masking rules, positional transformation, head identity, and logical position IDs. Combining two internally valid summaries built under different RoPE positions or masks is not valid attention.

0x07C Index Lifecycle Is Part Of Cache Lifecycle

An index entry may outlive or lag its K/V row. Updates need a publication order:

append token K/V to authoritative pool
validate row and position metadata
publish searchable index entry

on eviction:
    make index entry unreachable
    wait for search/read leases
    reclaim row storage

Publishing the index first creates a dangling retrieval result. Reclaiming storage before index readers finish can return a different token’s row under the old ID.

Index rebuilds also need versioning. A request should not combine candidate IDs from index generation g with row mappings from incompatible generation g+1.

0x07D InfiniGen And RetrievalAttention Make Different Design Choices

Neither method dominates by definition. The right choice depends on model geometry, context distribution, link bandwidth, CPU resources, concurrency, and the accepted semantic error envelope.

0x07E Indexed Retrieval Is A Pipeline, Not A topk Call

The previous sections established the exact reduction over a selected union. They did not establish how that union is found. Loop 7 now owns that missing pipeline:

authoritative full K/V rows
    -> build a queryable proxy or index
    -> publish one compatible index generation
    -> produce candidates for one query
    -> validate logical IDs and remove duplicates
    -> fetch or address authoritative full rows
    -> compute selected-support attention or a sampling estimator
    -> record error, bytes, and fallback separately

The generation fence in this reconstruction is as important as the search algorithm. Candidate IDs are usable only after the matching proxy generation and row map are published, and they become attention inputs only after they resolve to authoritative full rows or to a method-declared estimator state.

The four methods below alter different objects. Calling all of them “ANN attention” erases the design choices that determine correctness and cost.

This gives four separate approximation stages:

index representation error: proxy scores differ from full q @ k
candidate-support error: important logical rows are absent
payload representation error: fetched K/V differs from authoritative K/V
reduction/estimator error: selected rows are combined differently

PQCache, RetrievalAttention, and SqueezedAttention can leave payload representation error at zero by fetching full rows. MagicPIG deliberately changes the reduction into an estimator. None of those facts makes candidate support equal to dense support.

PQCache Figure 5 makes the publication order visible: authoritative K/V reaches host memory, the CPU builds codes and centroids, the GPU searches those proxy objects, and only then are selected full K/V rows fetched. The figure is source-reported architecture, not evidence that every stage overlaps on the hardware used by this article’s synthetic oracle.

0x07F PQCache: Product Quantization Is The Search Index, Not The K/V Payload

PQCache interprets each head’s key matrix as an embedding database. For one layer and K/V head, let:

Read the source figure as a dependency graph. “Centroids and Codes” selects IDs; the uncompressed K/V cache remains the authoritative payload below it.

K_full:       [T, d_h]       authoritative keys in host memory
V_full:       [T, d_h]       authoritative values in host memory
m:            number of disjoint key subvectors
d_m:          d_h / m
C:            [m, 2^b, d_m]  codebooks/centroids
codes:        [T, m]          one b-bit centroid ID per subvector

0x07F1 Derive The PQ Score From Dot-Product Linearity

Split key k_i and query q into m matching subvectors:

$$k_i=[k_i^{(1)},\ldots,k_i^{(m)}], \qquad q=[q^{(1)},\ldots,q^{(m)}].$$

K-means replaces subvector k_i^(j) with centroid C[j, code[i,j]]. The reconstructed key is:

$$\widehat{k}_i = [C_{1,c_{i,1}},\ldots,C_{m,c_{i,m}}].$$

Because the partitions are disjoint, its approximate inner product is a sum of table lookups:

$$\widehat{s}_i =q^\top\widehat{k}_i =\sum_{j=1}^{m} \left(q^{(j)}\right)^\top C_{j,c_{i,j}}.$$

For one query, first build the lookup table:

$$L[j,c]=\left(q^{(j)}\right)^\top C_{j,c}, \qquad L\in\mathbb{R}^{m\times 2^b}.$$

Then each token score costs m indexed reads and additions rather than a d_h-element dot product. The approximation error is explicit:

$$s_i-\widehat{s}_i =q^\top(k_i-\widehat{k}_i).$$

Small key reconstruction error does not guarantee preserved top-k order. Two nearby exact scores can swap under a much smaller perturbation than the norm of either key.

0x07F2 Build, Append, Search, Then Fetch Full Rows

The paper’s prefill path clusters each layer/head independently. Its reported shape contract is:

partitioned keys: [m, T, d_m]
centroids:        [m, 2^b, d_m]
codes:            [T, m]

New decode rows begin in the local GPU window. When one leaves that window, its m nearest-centroid IDs are appended to the code stream. The code stream is therefore mutable even when the original prompt codebooks are fixed.

def pqcache_decode(q, index, host_pool, local_rows, budget):
    assert index.pool_generation == host_pool.generation

    # [m, 2**b]: exact query-to-centroid products.
    lookup = einsum("md,mcd->mc", split(q, index.m), index.codebooks)

    # [T]: approximate MIPS scores from compact codes.
    approx = gather_sum(lookup, index.codes)
    history_ids = stable_topk(approx, budget.history_rows)

    selected_ids = unique(history_ids + index.initial_ids + local_rows.ids)
    full_kv = host_pool.gather_full_rows(selected_ids)

    # PQ reconstructions never enter final attention.
    return exact_attention_on_selected_rows(q, full_kv)

The final gather must return both the full key and full value under the same logical row ID. Feeding reconstructed PQ keys to the attention kernel would introduce payload error that the paper’s pipeline does not require.

0x07F3 Index Bytes, Payload Bytes, And Overlap Are Different Ledgers

Ignoring padding and metadata, PQ code storage for one layer/K/V head is:

$$M_{\text{codes}}=Tmb/8.$$

Codebook storage in b_C-byte elements is:

$$M_{\text{centroids}}=m2^bd_mb_C=2^bd_hb_C.$$

The authoritative full K/V remains:

$$M_{\text{full K/V}}=2Td_hb_{KV}.$$

Fetching k full rows costs approximately:

$$M_{\text{gather}}=2kd_hb_{KV},$$

before alignment and descriptor overhead. Thus M_codes is not a compressed replacement for M_full K/V; it is an additional search structure intended to reduce M_gather.

The paper overlaps K/V offload, CPU K-means, and code movement with model computation, keeps small centroids on GPU, and uses block-level caching for repeatedly selected rows. The non-overlappable boundary is still the query-dependent full-row fetch needed before attention. Publish a timeline, not only a sum:

build latency | hidden build latency | code-prefetch latency
PQ search | top-k | cache hits | host gather | H2D | exposed stall

0x07F4 Adaptive Construction And Three Cache Regions

The prefill schedule does not merely call K-means after the model finishes. For layer l, full K/V is copied to CPU while the GPU continues attention and FFN work. CPU workers then build H_kv * m independent clustering jobs. The paper profiles clustering and layer-compute time and caps K-means iterations so construction is intended to remain hidden:

$$t_{cluster}=\alpha_1+\beta_1 T n,$$ $$t_{layer}=\alpha_2+\beta_2 n+\gamma_2 n^2,$$

so their fitted crossing gives:

$$T_{max} = \frac{\gamma_2 n^2+\beta_2 n+\alpha_2-\alpha_1} {\beta_1 n}.$$

This is a source-reported scheduling heuristic, not a universal complexity guarantee. It requires a model/device-specific profile and clipping bounds.

Decode then maintains three regions:

InitKV:   fixed initial rows, always in selected attention
MidKV:    host-resident full rows with mutable PQ-code stream
LocalKV:  recent GPU rows; eviction appends one code and offloads full K/V

For each layer, next-layer codes may be prefetched while the current layer runs. After approximate MIPS, PQCache checks a block-granular GPU cache before reading misses from CPU; the paper compares LRU and LFU policies. The block cache changes physical traffic, not logical selected support. A cache hit must still bind the same row-map and pool generation as the PQ result.

0x07F5 K-Means Is Scheduled Work, Not A Free Prefill Side Effect

For one layer, H_kv stored heads, m subspaces, 2^b centroids, sequence length T, and I K-means iterations, the paper gives the construction order:

$$O\!\left(T H_{kv}m d_m 2^b I\right).$$

Because m d_m=d_h, the work is linear in prompt length for fixed PQ shape, but it is multiplied by heads, clusters, and iterations. PQCache profiles a device-specific overlap model:

$$t_{cluster}=\alpha_1+\beta_1TI,$$ $$t_{prefill}=\alpha_2+\beta_2T+\gamma_2T^2,$$

and chooses an iteration cap near:

$$I_{max} = \frac{\gamma_2T^2+\beta_2T+\alpha_2-\alpha_1} {\beta_1T}.$$

The equation is a fitted scheduling policy, not a quality theorem. Fewer iterations shorten the critical path but can worsen centroid assignment and candidate recall. Its coefficients must be regenerated when the model, hardware, CPU allocation, prompt distribution, or concurrent build load changes.

At decode, the paper separates three terms that are often collapsed:

query-to-codebook products:
    [H_q, m, 1, d_m] @ [H_kv, m, d_m, 2^b]

code lookup and reduction:
    codes [H_kv, T, m] -> approximate score [H_kv, T]

selected exact attention:
    full K/V [H_kv, k, d_h] -> output [H_q, 1, d_h]

The paper’s simplified decode complexity is:

$$O\!\left( \frac{2^b d^2}{H_qm} +H_{kv}mT +kd +d^2 \right),$$

plus top-k and actual transfer/runtime overhead. Its source-reported m=2,b=6 and m=4,b=8 settings belong to particular evaluation suites; they are not universal defaults. Increasing either knob improves index resolution only conditionally while raising code bytes, lookup work, build work, or all three.

0x07G RetrievalAttention: Query Distribution Is Part Of The Index Contract

Ordinary ANN construction organizes keys according to key-to-key geometry. RetrievalAttention observes that attention queries and keys can occupy different distributions. An index with good in-distribution key-to-key recall can therefore perform poorly for query-to-key search.

During prefill, the paper computes exact query-to-key neighbors and uses those connections to teach a cross-modal graph which keys appear close from the query distribution’s perspective. The conceptual build trace is:

prefill K: [T, d_h]
prefill Q samples: [T_q, d_h]
exact Q->K neighbor edges: [T_q, k_build]
project/contract query bridge into a key-only graph
publish attention-aware index generation g

This is adaptation from observed prefill queries, not a proof that future decode queries are in distribution. Index receipts need the source query sample identity and an OOD diagnostic:

model/layer/query-head/KV-head
RoPE and normalization convention
prompt/cache generation
prefill query sample range
build-neighbor budget and graph parameters
index generation and row-map generation
calibration recall / OOD statistic

0x07G1 Persistent And Retrieved Rows Form One Logical Set

The paper keeps sink/initial and recent-window rows on GPU and indexes the remaining rows on CPU. In a GQA model it builds a search index per query head while indexes in one attention group can share the underlying K/V vectors.

def retrieval_attention_step(q, persistent, index, pool, budget):
    assert index.generation == pool.index_generation
    assert index.row_map_generation == pool.row_map_generation

    gpu_ids, gpu_kv = persistent.visible_rows(q.identity)
    cpu_ids = index.search(q, budget)
    cpu_ids = validate_and_deduplicate(cpu_ids, exclude=gpu_ids)
    cpu_kv = pool.gather_full_rows(cpu_ids)

    gpu_state = online_softmax_state(q, gpu_kv)
    cpu_state = online_softmax_state(q, cpu_kv)
    return merge_online_softmax_states(gpu_state, cpu_state)

The merge is exact over unique(gpu_ids union cpu_ids). It is not exact over history rows that the index failed to retrieve. If duplicate IDs survive into both partitions, their exponentiated weight is counted twice.

The paper appendix also describes prefill-time pipelining: full attention stays on GPU while K/V movement and per-head index construction proceed on CPU. That build pipeline needs a publication fence. A partially built index must not be visible merely because some K/V rows have already reached host memory.

The source figure’s right panel is the critical distinction from an ordinary key-only proximity graph: exact prefill Q -> K neighbors supply a bridge from the query distribution, and shared-neighbor relations are projected into a key-only graph. During GQA decoding, the paper builds one index per query head because query-head distributions differ, while indexes in the same group share one underlying K/V payload. Index count and payload-copy count are therefore different memory ledgers.

0x07G2 Paper Algorithm And Current Repository Are Not The Same Artifact

The current Microsoft repository at commit 75829e630122d4ea6f568dcd001405698bc2db84 primarily implements the later RetroInfer/WAVE system. It cites RetrievalAttention, but its current steady/retrieval/estimation zones and update kernels must not be retroactively attributed to the 2024 paper. Loop 7 therefore pins the RetrievalAttention paper and appendix algorithm for paper claims, while treating the repository only as lineage evidence.

The paper’s source-reported 1–3% scan ratios and recall/latency results are evidence for its evaluated indexes and query distributions, not an ANN service SLO. A deployment receipt needs at least recall against exact top-k on held-out decode queries, visited-node distribution, graph-build age, and a measured OOD statistic. Search budget should be allowed to escalate before an exact fallback; otherwise a constant candidate count can hide a changing error rate.

0x07H SqueezedAttention: Offline Fixed-Context Geometry

SqueezedAttention targets a different workload: a large fixed prefix reused across incoming user inputs. That fixed context makes an offline clustering cost plausible. It is not a generic append-friendly index contract.

For cluster i, let C_i be the mean key, N_i its number of full rows, and q one incoming query. Define the implementation score coordinate z_i=qC_i^T/sqrt(d_h). The paper writes the unscaled dot product for brevity; the official threshold-calibration path applies the ordinary attention scale. The estimated average attention probability of a row in cluster i is:

$$S_i = \frac{\exp(z_i)} {\sum_j N_j\exp(z_j)}.$$

Why multiply the denominator by N_j? Replacing every key in cluster j by its centroid creates N_j identical approximate logits. Their total unnormalized mass is therefore N_j exp(q C_j^T). Omitting N_j would assign the same total mass to a one-row and a hundred-row cluster.

The method loads all authoritative K/V rows in clusters whose S_i exceeds a calibrated global threshold and computes ordinary attention over that selected support. The threshold is global so heads with diffuse centroid distributions can naturally retain more clusters than highly concentrated heads.

0x07H1 One Level Trades Resolution For Lookup Cost

With c centroids and k selected full rows, one-level lookup costs:

$$O(c+k)$$

per query, rather than O(T). But reducing c makes clusters larger. One important row can be averaged with many irrelevant rows, while selecting a large mixed cluster creates physical overfetch.

0x07H2 A Hierarchy Prunes Coarse Groups Before Fine Centroids

The paper first clusters full keys into fine level-2 centroids, then clusters those centroids into coarse level-1 groups. Query execution becomes:

score all coarse centroids with threshold T1
expand only surviving coarse groups
score their fine centroids with threshold T2
map surviving fine centroids to logical full-row IDs
run sparse exact attention over those rows

If each level retains c' centroids and the tree depth scales as log T, the paper gives the idealized lookup-plus-attention complexity:

$$O(c'\log T+k).$$

The asymptotic statement depends on controlled fanout and pruning. Real bytes also include centroid arrays, labels/row lists, irregular gathers, and duplicate row removal across query heads.

The source figure is especially useful because it exposes non-contiguous gathers. Semantic clustering improves grouping by meaning, but does not promise physical locality in the K/V pool.

The official repository at commit 4a50c64eca2f20232a4e998d754d030ce3a7f6e6 makes the lifecycle concrete: offline_clustering.py creates level-2 key centroids, optionally reclusters them into level 1, calibrates global thresholds, and serializes centroid and label tensors. squeezedattention/kernels.py implements centroid lookup, while the sparse-attention path consumes selected key indices. These files are implementation evidence for that pinned revision, not a claim that every paper measurement used the current checkout unchanged.

The fixed-context manifest must bind:

token bytes/hash and logical positions
model/layer/head plus RoPE convention
centroid tensors and cluster labels
threshold calibration dataset and quantile
hierarchy parent/child mapping
build code/version and index generation

Appending or editing the fixed prefix invalidates those objects. Reusing stale centroids because the sequence length happens to match is a semantic cache hit on the wrong data.

0x07H3 The Runtime Uses Two Centroid Passes And A Ragged Exact Kernel

The pinned Triton implementation does not materialize a dense token-level proxy matrix. Its centroid lookup first streams centroids to build the stable denominator:

$$D(q)=\sum_j N_j e^{qC_j^\top},$$

then streams them again to test exp(q C_i^T) > D(q) T_threshold. During generation it can retain the exponentials from the first pass so the second pass becomes comparison rather than another dot product. The selected cluster labels are compacted into logical key indices.

The sparse attention kernel then faces a different problem: selected counts are ragged across heads. Assigning one program to each head makes the densest head a straggler. The implementation splits long selected-index lists into fixed K/V blocks, computes online-softmax partial states, and merges them in a second phase. The centroid threshold controls semantic support; the block size controls GPU scheduling. They must be ablated separately.

For GQA, multiple query heads share one K/V head but may select different clusters. A runtime can gather the union once and mask per query head, or issue head-specific gathers. The former overfetches rows; the latter duplicates address traversal and transfer. Neither choice follows from the clustering formula alone.

0x07I MagicPIG: Replace Deterministic Omission With A Sampling Estimator

Top-k selected-support attention renormalizes over the chosen rows:

$$o_{\text{top-k}} = \frac{\sum_{i\in S_k} e^{s_i}v_i} {\sum_{i\in S_k} e^{s_i}}.$$

Every omitted row receives exactly zero weight. When attention mass is diffuse, many individually small terms can collectively matter. Increasing top-k can reduce this bias, but it does not turn deterministic omission into a sampling estimator.

MagicPIG starts from the identity:

$$o=\sum_i w_i v_i=\mathbb{E}_{i\sim w}[v_i], \qquad w_i=\frac{e^{s_i}}{\sum_j e^{s_j}}.$$

Sampling directly from w would be unbiased, but computing w already scans all keys. The paper therefore uses self-normalized importance sampling with an LSH-derived proposal/inclusion probability u_i. For a sampled set S, its implemented estimator is:

$$\widehat{o} = \frac{\sum_{i\in S} e^{s_i}v_i/u_i} {\sum_{i\in S} e^{s_i}/u_i}.$$

The 1/u_i factor corrects unequal inclusion likelihood. Removing it biases the result toward rows that collide easily, not necessarily rows carrying the right aggregate value. Self-normalization makes the finite-sample estimator biased in general, though the paper derives consistency as sample budget grows.

0x07I1 Derive The At-Least-Two-Table Inclusion Probability

MagicPIG uses L SimHash tables with K bits per table. For cosine collision probability:

$$p_i =1-\frac{1}{\pi} \arccos\left(\frac{qk_i^\top}{\lVert q\rVert\lVert k_i\rVert}\right),$$

all K bits match in one table with probability p_i^K. Let r_i=p_i^K. The paper admits a key only if it matches at least two tables. Subtract the zero-match and exactly-one-match events from one:

$$u_i =1-(1-r_i)^L-Lr_i(1-r_i)^{L-1}.$$

This derivation matters operationally: changing the collision rule from one table to two changes u_i, so old probability metadata cannot be used by the new estimator.

The paper derives a maximum-inner-product-to-cosine augmentation as theoretical motivation, but explicitly reports that this transform was not used in its experiments. The implemented path instead centers the offloaded keys because softmax is translation invariant. The pinned code subtracts the same host-key mean from sink/local GPU keys and from offloaded CPU keys. Centering only the CPU set would shift its logit frame relative to the GPU state and invalidate a plain online-softmax merge. Sink/local rows form a static exact set with inclusion probability one.

Centering preserves full attention weights because one shared key mean mu_K subtracts the same scalar from every logit:

$$q(k_i-\mu_K)^\top=qk_i^\top-q\mu_K^\top.$$

Softmax cancels the common offset. That invariance does not permit a stale mean: if some rows use one center and appended rows use another, the offset is no longer common across support.

def magicpig_decode(q, static_kv, lsh_index):
    q_code = simhash(q, lsh_index.projectors)       # GPU in paper system
    sampled_ids = lsh_index.matches(q_code, min_tables=2)

    # Static and offloaded keys share the same centered score coordinates.
    static_kv = static_kv.subtract_key_mean(lsh_index.key_mean)
    sampled_kv = lsh_index.cpu_pool.centered_full_rows(sampled_ids)
    u = inclusion_probability(sampled_kv.keys, q,
                              K=lsh_index.K, L=lsh_index.L,
                              min_tables=2)
    assert all(probability > 0 for probability in u)

    ids, kv, probs = union_with_static(sampled_ids, sampled_kv,
                                       static_kv, static_probability=1.0)
    logits = exact_full_key_scores(q, kv.keys)
    corrected_logits = logits - log(probs)
    return softmax(corrected_logits) @ kv.values

The official implementation at commit ac9aa36c866330ca6ad2ce342a7848d7df6f49bb shows this placement boundary: models/attnserver.py centers offloaded keys, creates GPU hash projections, stores sorted hash tables and full K/V on CPU, retrieves IDs on CPU, computes CPU sparse attention, and merges its state with sink/local GPU attention. The pin is code evidence, not a local performance reproduction.

The bandwidth numbers in this source figure describe the paper’s example hardware. The reusable contract is the placement and execution order, not those specific capacities.

K and L control more than sample count. Larger K makes one-table collisions exponentially rarer through p_i^K; larger L creates more chances to reach the two-table admission rule, but expands table storage, random projections, and probes. The pinned C++ runtime allocates 2^K bucket ranges per table and K/V head, stores table contents in CPU memory, accumulates a per-token collision mask, and emits an ID only on its second observed table match. Changing K, L, centering, projector generation, or collision count requires rebuilding both search state and the inclusion-probability contract.

The runtime placement also changes the traffic ledger:

GPU:
    model projections + K*L random projections + static sink/local attention

CPU:
    hash-table probes + sampled full-row attention

cross-device:
    query/hash metadata outward + mergeable attention state inward

This is compute-near-data over a sampled support, not selected full-row H2D gather. It therefore combines Loop 5’s heterogeneous partial-state algebra with Loop 7’s approximate estimator contract.

The source figure is a placement contract: model projections and SimHash query projections execute on GPU, while large hash tables, sampled full K/V, and sparse attention live on CPU before online-state merge. It does not prove that CPU attention is hidden at arbitrary concurrency or that the sampled estimator meets every downstream quality envelope.

0x07J Loop 7 Executable Reconciliation

The deterministic indexed_sampling_retrieval_fixture.json and indexed_sampling_retrieval_oracle.py under docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/ reconciles all four contracts without claiming paper accuracy or hardware performance:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/indexed_sampling_retrieval_oracle.py
python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/index_sampling_oracle.py

The first fixture reconciles the common authoritative pool, byte ledgers, deduplicated partial-state merge, fixed-context identity, and 38 invalid contracts. The second is an independent adversarial pass focused on nearest PQ-code ranking failure, centroid false negatives, and finite-sample estimator bias. Agreement is expected on invariants, not on their deliberately different toy tensors.

PQCache:
    codebook shape -> code IDs -> lookup table -> approximate rank
    -> full-row gather -> selected-support attention

RetrievalAttention:
    persistent IDs + retrieved IDs -> deduplicated union
    -> two online-softmax states -> direct-union equality

SqueezedAttention:
    immutable fixed-context hash -> cluster means and sizes
    -> weighted centroid denominator -> hierarchy -> full-row support

MagicPIG:
    common key centering -> cosine collision
    -> at-least-two-table inclusion probability
    -> fixed sampled set + static rows -> corrected logits -> estimator

It derives RetrievalAttention’s prefill Q -> K neighbors and projected key edges, reconstructs both partial softmax states from the selected full rows and matches the direct union exactly, derives SqueezedAttention’s fine/coarse centroids from the immutable prefix, and proves common-centering softmax invariance for MagicPIG. Thirty-eight one-field invalid fixtures reject stale generations, dangling IDs, malformed PQ geometry, non-covering hierarchies, changed collision rules, zero-probability samples, and proxy-as-payload errors.

The counterexample is intentionally adversarial. PQ lookup misses the exact best row even though every code is a valid nearest centroid; the centroid hierarchy selects the higher-average cluster while omitting a lower-average cluster containing the exact best row; and one valid MagicPIG sample produces a high-error ratio estimate. Enumerating its finite inclusion outcomes confirms that the self-normalized ratio estimator is not finite-sample unbiased. Those outcomes do not refute the papers. They prevent index validity, asymptotic consistency, or selected-support exactness from being reported as dense attention equality.

The fixture also records independent ledgers:

Failure Lab

Loop 8 changes the unit of stored context into memory units, semantic clusters, and events. It does not reopen PQ, ANN, fixed-prefix centroid, or LSH estimator algebra.

0x07K Recallable Memory Changes The Unit, Not The Authority

The previous selectors leave the stored context structurally unchanged. A page index ranks physical pages; a PQ or graph index ranks existing rows; an LSH table samples rows. InfLLM, EM-LLM, and ClusterKV go further: they define a memory unit whose membership, representative, update schedule, and recall granularity become part of the inference state.

The distinction is not cosmetic:

All four still need the Loop 2 authority chain:

representative or centroid
    -> unit/cluster/event ID at index generation g
    -> original committed positions
    -> authoritative K/V rows at cache generation c
    -> attention over the selected support

A centroid is not a replacement key, an event summary is not a full value tensor, and a memory-unit cache hit is not permission to ignore tenant, checkpoint, RoPE, layer, K/V-head, or committed-frontier identity.

Source figure: ClusterKV Figure 1, preserved from A25. White cells in the recallable lanes are not destroyed state: they remain recoverable from the backing tier. Page recall preserves spatial neighbors; semantic recall groups noncontiguous rows. Neither selected support is automatically dense support.

This boundary also separates three meanings of “memory”:

M5.4 permanent retention:
    decides which internal K/V rows remain recoverable at all

M5.5 recallable context memory:
    groups and retrieves already-computed internal K/V rows

document RAG:
    retrieves external text or objects and re-encodes them through the model

Calling all three RAG would erase where information re-enters the computation. The methods below alter the attention support directly; they do not prepend retrieved documents to a new prompt.

0x07L InfLLM: Fixed Memory Units With Query-Dependent Full-Unit Recall

InfLLM partitions context into three regions for each layer: always-visible initial rows I, distant evicted rows E, and a recent local window L. For current chunk X, its retrieved context can be written as:

$$C(X)=\operatorname{Concat}\!\left(I, f(X,E), L\right),$$

and attention consumes current rows as well:

$$O_X = \operatorname{Attn}\!\left( Q_X, \operatorname{Concat}(C_K,K_X), \operatorname{Concat}(C_V,V_X) \right).$$

Source figure: InfLLM Figure 1. The figure makes two representations visible: compact representative tokens drive lookup, while selected memory units restore full K/V rows to the current attention context. It does not establish paper-reported quality or latency on the local fixture.

The important change hides inside f(X,E): distant rows are partitioned into fixed contiguous units B_1,...,B_M, each of size l_bs. Retrieval ranks units, not individual rows.

0x07L1 Representative Keys Are A Compressed Search Surface

InfLLM does not average every block into one centroid. It chooses r_k representative tokens whose keys were influential for later local queries. For candidate row m, the paper scores how strongly its key is used by a following local span:

$$r_m = \frac{1}{l_L} \sum_{j=1}^{l_L} q_{m+j}k_m^\top.$$

Selecting the largest r_k scores in each unit stores a small search surface:

$$R(B_u)=\{k_{u,b_1},\ldots,k_{u,b_{r_k}}\}.$$

For a current chunk with query rows Q_X, the unit score sums similarities to those representatives:

$$s(X,B_u) = \sum_{i=1}^{l_X} \sum_{j=1}^{r_k} q_{i+l_P}k_{u,b_j}^{\top}.$$

The top k_m units are then expanded to all their member rows. The search therefore costs roughly M r_k key comparisons per query aggregate while the selected attention support costs up to k_m l_bs full K/V rows:

$$M_{\text{repr}} =M r_k d_h b_K, \qquad M_{\text{selected K/V}} =k_m l_{bs}d_h(b_K+b_V)$$

per K/V head before GQA sharing and tier caching. Reducing representative bytes does not reduce the selected full-unit payload by the same factor.

The Loop 8 fixture uses three units of three rows and one representative each. For query [1,0], representative scores are 0.95, 0.20, and -0.80. Selecting units 0 and 1 recalls positions [1..6]; initial position 0 and local positions [10,11] are added separately. The support is therefore [0..6,10,11], not the three representative rows.

0x07L2 Position Handling Is Part Of The Method

Retrieved units can be far apart in original sequence position. Passing their compacted rank as a new absolute position would silently change a checkpoint’s positional contract. InfLLM assigns distant memory rows a common relative distance associated with the local-window boundary rather than exposing large, discontinuous distances outside the model’s familiar range.

That is a paper-specific extrapolation design, not a universal K/V-cache rule. The backing store must still retain original position identity so that a different position policy, checkpoint, or exact fallback cannot mistake compacted order for source order.

0x07L3 Tensor And Pinned-Code Trace

The official repository is pinned here at commit 12b70798. Its ContextManager exposes the paper’s control variables directly:

n_init, n_local
block_size, topk
repr_topk
max_cached_block
cache_strategy, score_decay
faiss, perhead

Representative extraction and lookup follow this tensor path:

# Conceptualized from the pinned context manager; not copied runtime code.
# K_block: [kv_heads, unit_rows, d_h]
# influence: [kv_heads, unit_rows]
rep_idx = topk(influence, r_k, dim="unit_rows")
K_rep = gather(K_block, rep_idx)          # [kv_heads, r_k, d_h]

# Q_chunk: [q_heads, chunk_rows, d_h]
# GQA ownership maps q heads to their K/V head before unit lookup.
unit_score = sum_over_query_and_representatives(Q_chunk @ K_rep.T)
selected_units = topk(unit_score, k_m, dim="memory_units")
K_sel, V_sel = load_full_units(selected_units, generation)

The code keeps full memory blocks in host-backed structures, tracks selected blocks in a bounded GPU cache, and supports either exact vector ranking or a Faiss path for representative lookup. Its GQA path expands representative keys according to query-head ownership before scoring. Those are code-level facts at the pin, not proof that every paper experiment used every option.

Two cache policies illustrate a design choice:

The paper reports settings such as chunk size 512, unit size 128, four representatives, and 16 or 32 selected units for named model configurations. They are source-reported operating points, not defaults inferred for arbitrary hardware or checkpoints. Larger units reduce index entries but increase overfetch; more representatives improve unit discrimination but expand GPU metadata and lookup; more selected units increase recall and full-row traffic.

0x07M EM-LLM: Events Need Boundary Construction And Two Retrieval Policies

EM-LLM replaces fixed block boundaries with variable-length events. Its biological analogy motivates the design, but the executable contract is entirely mechanical:

token surprisal
    -> candidate boundaries
    -> key-similarity graph refinement
    -> variable event blocks
    -> semantic retrieval
    -> temporal-neighbor expansion
    -> full event K/V attention

Source figure: EM-LLM Figure 3C, shown with the paper’s adjacent diagnostic panels. Surprise proposes event boundaries; the event store then feeds a similarity buffer and a separate FIFO-style contiguity buffer. The biological analogy and source-reported benchmark results remain paper claims.

0x07M1 Surprise Produces Candidates, Not Final Events

For token x_t, surprisal under the frozen language model is:

$$S_t=-\log P_\theta(x_t\mid x_{<t}).$$

Over a trailing window of length tau, EM-LLM forms a moving threshold:

$$T_t=\mu_{t-\tau:t}+\gamma\sigma_{t-\tau:t},$$

and proposes a boundary when:

$$S_t>T_t.$$

This form follows from treating surprise relative to its recent level rather than using one global threshold. gamma controls sensitivity: reducing it creates more, shorter events and more representatives; increasing it creates fewer, longer events and more retrieval amplification. The paper’s formula uses sigma; this article treats it as the trailing standard deviation rather than calling it variance.

The local fixture’s surprise trace proposes boundaries at positions 3 and 7. It then considers nearby cuts because a probability spike can occur just before or after the most coherent key-space division.

0x07M2 Graph Refinement Changes Where The Cut Lands

For each head, tokens around a candidate boundary form a weighted graph with key similarity:

$$A^{(h)}_{ij}=\operatorname{sim}(K_i^{(h)},K_j^{(h)}).$$

One possible objective is graph modularity. For communities induced by a cut:

$$\mathcal Q = \frac{1}{2m} \sum_{i,j} \left(A_{ij}-\frac{d_i d_j}{2m}\right) \mathbf 1[c_i=c_j].$$

Another is conductance:

$$\phi(S) = \frac{\operatorname{cut}(S,\bar S)} {\min(\operatorname{vol}(S),\operatorname{vol}(\bar S))}.$$

A modularity policy chooses a higher score; a conductance policy chooses a lower cut ratio. They are not interchangeable scalar conventions. The pinned repository supports modularity, conductance, and an intra/inter-similarity ratio in em_llm/attention/similarity_refinement/. It searches positions around the candidate while preserving boundary count and enforcing size constraints.

The local oracle publishes the candidate and refined arrays separately. Its toy score windows keep cuts at 3 and 7, yielding event lengths [3,4,3] before the local window. A mutation that reorders boundaries or removes a search window is rejected before retrieval.

0x07M3 Semantic Retrieval And Temporal Continuity Are Separate Buffers

An event representative compresses a variable number of rows into lookup metadata. Let z_e be an event summary and q the current query summary. Semantic retrieval ranks:

$$s_e=qz_e^\top$$

and selects a similarity set S_sim. EM-LLM then adds neighboring events to a contiguity set S_ctg:

$$S_{event} = S_{sim}\cup \bigcup_{e\in S_{sim}}\{e-r,\ldots,e+r\}.$$

The union matters. A semantically similar event may start midway through a procedure or argument; adjacent events restore local narrative context even when their representative similarity is lower. But temporal expansion also spends token budget, and variable event lengths mean k events are not a stable byte or token contract:

$$T_{retrieved}=\sum_{e\in S_{event}}|B_e|.$$

The runtime therefore needs a token-aware cap, deterministic deduplication, and a rule for which side of the temporal neighborhood is trimmed first.

The pinned official implementation at commit edb2e4a6 stores variable MemoryBlock objects, representative vectors, and a token-budget-aware contiguity queue. The current code can offload full blocks to CPU or disk and can move representative vectors to CPU. Its block summary averages selected representative keys before insertion into the vector store; that implementation detail must not be generalized into a claim that every EM-LLM formulation uses one ordinary centroid.

def event_memory_step(committed_rows, surprise, query, manifest):
    candidates = moving_surprise_boundaries(surprise, tau, gamma)
    boundaries = refine_with_key_graph(candidates, metric=manifest.metric)
    events = seal_variable_blocks(committed_rows, boundaries)

    semantic = top_events(query, events.representatives,
                          token_budget=manifest.semantic_budget)
    temporal = neighboring_events(semantic, manifest.neighbor_radius)
    selected = deduplicate_and_fit_tokens(semantic | temporal,
                                          manifest.total_event_budget)
    return load_authoritative_event_rows(selected, manifest.generation)

Segmentation is online inference work, not model training. It consumes model probabilities, key similarities, boundary metadata, and per-layer retrieval state. The paper does not require fine-tuning the base checkpoint for this memory policy. That makes deployment easier than a retrained long-context architecture, but it does not make event construction or retrieval free.

0x07N ClusterKV: Noncontiguous Semantic Membership With Full-Row Recall

ClusterKV argues that fixed pages can overfetch spatial neighbors while splitting semantically related keys across many pages. It clusters key rows in semantic space so that a query selects groups by content rather than original adjacency.

Source figure: ClusterKV semantic-space motivation, preserved from A25. The right-hand ordering is a cluster-selection explanation, not evidence that all members have equal attention weight.

0x07N1 Assignment Uses Cosine Geometry

ClusterKV defines cosine distance between keys:

$$D(i,j) = 1- \frac{k_i k_j^\top}{\lVert k_i\rVert_2\lVert k_j\rVert_2}.$$

After normalizing keys \bar k_i=k_i/||k_i||_2, minimizing cosine distance is equivalent to maximizing \bar k_i\mu_c^T. K-means alternates:

$$z_i = \arg\max_c \bar k_i\mu_c^\top,$$ $$\mu_c = \operatorname{normalize}\!\left( \frac{1}{|C_c|} \sum_{i:z_i=c}\bar k_i \right).$$

Normalization after the mean keeps the next assignment in cosine geometry. Empty clusters require an explicit retention or reinitialization rule; they cannot emit a valid centroid/member list by accident.

The Loop 8 fixture clusters rows [1..8] from deterministic initial keys at positions 1, 5, and 8. It converges to memberships:

cluster 0: [1,2,3]    centroid approximately [ 0.991, 0.136]
cluster 1: [4,5,6]    centroid approximately [ 0.118, 0.993]
cluster 2: [7,8]      centroid approximately [-0.998, 0.055]

This toy verifies geometry only. It does not reproduce the paper’s clustering quality on a model checkpoint.

0x07N2 Query Ranking And Token Budget Need Two Prefix Sums

At decode, cluster importance uses the current query against each centroid:

$$a_c=q\mu_c^\top.$$

The implementation then sorts cluster IDs by a_c, gathers cluster sizes in that order, and computes a cumulative token count:

$$P_j = \sum_{r=1}^{j}|C_{\pi_r}|, \qquad \pi=\operatorname{argsort}_{c}(-a_c).$$

The selected prefix is the largest set that fits the token budget B. If the next cluster crosses B, ClusterKV can trim that final cluster rather than dropping all of it. With fixture sizes [3,3,2] and B=5, the query ranks clusters [0,1,2]; cluster 0 contributes three rows and cluster 1 contributes two of its three rows. This exposes a second selection contract inside the last cluster. The trim must remain deterministic and tied to member ordering.

scores = query @ centroids.T                 # [kv_heads, n_clusters]
cluster_order = argsort(scores, descending=True)
ordered_sizes = gather(cluster_size, cluster_order)
ordered_prefix = cumsum(ordered_sizes)
selected_clusters, last_trim = fit_prefix(ordered_prefix, token_budget)
row_ids = gather_members(selected_clusters, last_trim)
K_sel, V_sel = host_kv.gather(row_ids, cache_generation)

The clustering step uses normalized-key cosine geometry, while decode ranking uses the query-centroid inner product in the audited paper/code path. Keeping those two score definitions explicit avoids silently normalizing a query or centroid at the wrong stage.

0x07N3 Prefill Clusters And Decode Append Clusters Have Different Lifetimes

The paper’s source-reported configuration keeps initial sink rows and forms a prefill cluster set whose count scales with prompt length. One stated schedule can be written:

$$C_0=80\frac{L}{6400}=\frac{L}{80},$$

which gives 400 clusters at 32K rows. During decode, it does not rerun K-means over the entire growing history every token. After a window of m=320 new rows, it forms four append clusters and extends centroid/member metadata.

Those are paper operating choices, not universal constants. Their structural purpose is more important:

prefill generation g:
    cluster a large immutable prefix, publish centroids/labels/member order

decode window w:
    keep new rows directly visible while the window is open
    cluster the sealed window
    append new centroids and member ranges at generation g+1
    offload the sealed full K/V rows

Queries cannot mix centroids from g+1 with member prefix sums from g. Cluster metadata needs one publication transaction covering centroids, sizes, size prefix sums, sorted member IDs, full-row authority, and GPU-cache mapping.

The official implementation is pinned at commit c7380819. ClusterKVAttention.py builds prefill clusters for configured layers, triggers append clustering on the decode-window boundary, waits on build events before selection, and offloads prefill/window K/V. Its controller stores:

centroids:            [layers][kv_heads, clusters, d_h]
cluster_size:         [layers][kv_heads, clusters]
cluster_size_ps:      [layers][kv_heads, clusters]
cluster_key_indices:  [layers][kv_heads, committed_rows]
GPU/CPU K/V buffers, selected-index maps, stream events, generations

The pinned code also exposes restricted assumptions, including limited batch handling in audited paths and special handling for early layers. It is an implementation trace, not evidence of arbitrary continuous-batching safety.

Source figure: ClusterKV system overview, preserved from A25. Centroids & Metadata are the selection surface; CPU memory remains the authority for offloaded full rows; selected K/V are the attention payload.

0x07N4 ClusterKV Is Not SqueezedAttention With A Different Name

Both methods use centroids, but their complete contracts differ:

Collapsing them into “centroid retrieval” would lose the update schedule that dominates serving integration.

0x07O Paper-By-Contract Comparison

The common approximation is support choice, not necessarily K/V representation. A system may later compress offloaded payloads, but that is a separate M8.6 representation error and must not be hidden inside retrieval quality.

Runtime costs also differ before attention begins:

$$C_{\text{decision}} = C_{\text{boundary/update}} +C_{\text{representative lookup}} +C_{\text{member expansion}} +C_{\text{tier gather}}.$$

For InfLLM, boundary construction is fixed and cheap while full-unit overfetch can dominate. For EM-LLM, event construction and variable-length budget fit are substantial. For ClusterKV, K-means is amortized over prefill/windows, but centroid growth, prefix sums, member indirection, cache replacement, and irregular gathers remain online state.

Failure Lab

0x07P Loop 8 Executable Reconciliation

The article-local fixture and oracle are reproducible with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/recallable_memory_oracle.py
python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/recall_unit_oracle.py

It freezes one 12-row authoritative K/V pool and derives all three policies from that same identity:

InfLLM:
    [3 units, 3 rows/unit, d_h=2]
    -> representative scores [0.95, 0.20, -0.80]
    -> units [0,1]
    -> six full distant rows + initial/local support

EM-LLM:
    surprise candidates [3,7]
    -> refined cuts [3,7]
    -> event lengths [3,4,3]
    -> semantic event 0 + temporal event 1
    -> seven historical rows + local support

ClusterKV:
    three cosine clusters with sizes [3,3,2]
    -> query order [0,1,2]
    -> budget 5
    -> cluster 0 plus two rows of cluster 1
    -> append window published as a new cluster generation

The first oracle rejects twenty-four one-field failures around cache/index identity, tenant, committed frontier, position basis, unit/event/cluster publication, and cross-article ownership. A second independent oracle starts from a row-structured tensor fixture and rejects thirty-six more failures: malformed K/V shapes, overlapping roles, incomplete unit coverage, impossible representative budgets, invalid surprise windows, event representatives outside their event, sink rows entering K-means, stale cached labels, malformed decode append clusters, and incomplete paper-comparison records.

canonical identity/boundary oracle: 24 rejected invalid fixtures
independent tensor/lifecycle oracle: 36 rejected invalid fixtures
combined Loop 8 failure surface:    60 isolated mutations

The oracles deliberately use different row pools and expected selections. Their agreement claim is structural, not that they must emit the same support: both preserve authoritative full rows, separate lookup metadata from payload, bind online updates to a generation, and reject selected-support exactness being reported as dense-support exactness.

Loop 9 receives the complete selector family. It must reconcile candidate recall, attention-mass recall, output error, metadata and fetch bytes, overfetch, and exact/degraded fallback without pretending that one recall percentage is a downstream quality or service guarantee.

0x08 Runtime Integration: Retrieval Must Fit The Scheduler

An isolated attention microbenchmark can hide the hardest part. A serving runtime multiplexes requests with different context lengths, retrieved-set sizes, deadlines, adapters, and cache hit rates.

0x08A Request State

@dataclass
class RetrievalDecodeState:
    request_id: str
    cache_identity: KVChunkKey
    committed_length: int
    pool_generation: int
    index_generation: int
    resident_blocks: set[int]
    inflight_promotions: dict[int, Future]
    staging_bytes: int
    retrieval_budget: int
    exact_fallback_allowed: bool
    deadline_ns: int

committed_length matters because speculative or cancelled work must not become searchable committed history. M7 adds branch-local K/V states; M5.5’s pool and index should receive only rows that cross the commit boundary.

0x08B Batch Formation Uses More Than Sequence Length

Two requests with the same logical length can impose very different transfer work:

request A: 95% GPU hit, 128 dynamic rows
request B: 10% GPU hit, 4096 dynamic rows

A scheduler that batches by token count alone can create a layer barrier dominated by B’s transfer. Useful batch keys include:

predicted fetch bytes
source tier and NUMA node
retrieval candidate count
deadline/slack
exact versus approximate contract
adapter/model cache identity
staging-buffer availability

The scheduler may bucket similar transfer shapes, cap aggregate in-flight bytes, or defer a bandwidth-heavy request. Each choice trades fairness, utilization, and latency.

0x08C Backpressure Must Reach Admission

def schedule_retrieval_step(batch, budgets):
    plans = [plan_fetch(req) for req in batch]

    if sum(p.staging_bytes for p in plans) > budgets.gpu_staging:
        batch = split_by_staging_pressure(batch)
        plans = [plan_fetch(req) for req in batch]

    if sum(p.transfer_bytes for p in plans) > budgets.link_window:
        batch = prioritize_by_deadline_and_hit_rate(batch)
        plans = [plan_fetch(req) for req in batch]

    for plan in plans:
        submit_prefetch(plan)

    ready, late = wait_until_attention_boundary(plans)
    handle_late_fetches(late)
    return run_attention(ready)

If late fetches are common, the correct fix may be lower concurrency, a larger resident set, different candidate budgets, or rejecting long-context requests. Queueing more asynchronous copies does not create bandwidth.

0x08D Exact And Approximate SLOs Should Be Separate

exact tiered mode:
    correctness target = full-cache same-format output
    failure action = wait, promote, recompute, or reject

approximate retrieval mode:
    correctness target = declared quality/error envelope
    failure action = increase budget, exact fallback, or declared degradation

Do not silently switch an exact request to approximate retrieval to protect latency. That changes the product contract, not merely the backend.

0x08E Security And Multi-Tenant Isolation

Persistent and remote tiers can outlive a process, so identity is also a data-isolation boundary:

• namespace cache keys by tenant and authorization domain;
• encrypt or otherwise protect remote/persistent copies according to the deployment threat model;
• zero or cryptographically invalidate reclaimed storage where required;
• prevent timing and index metadata from exposing another tenant’s prompt;
• include adapter/model provenance in audit records;
• define retention and deletion behavior independently of allocator eviction.

A hash collision, broad prefix namespace, or stale directory entry can expose semantic state across requests even when tensor shapes look valid.

0x08F A Runtime Trace Has Four Evidence Classes

“The runtime supports offload” is too weak for an implementation contract. It can mean a documentation promise, one release’s source behavior, a local policy model, or a measured deployment. Those classes must remain separate:

Loop 10 pins vLLM v0.26.0 and LMCache v0.5.2. The pins are not decoration. Both projects are moving runtime surfaces, so an unpinned source description silently changes underneath the article.

0x08G Scheduler, Worker, And Storage Manager Own Different State

The pinned vLLM connector splits responsibility at the scheduler/worker boundary. The scheduler.py path owns request lookup state, load/store job identifiers, per-worker completion fan-in, block-reuse protection, stale-job epochs after reset, and aggregated metrics. Its metadata is a command envelope, not transferred K/V:

scheduler:
    lookup immutable offload keys
    reserve destination GPU blocks
    build load jobs and stores-to-flush
    publish connector metadata for this engine step

worker:
    map physical K/V pages into a canonical byte view
    submit load jobs before the model uses those pages
    defer ordinary stores until the next step
    flush protected stores before source block reuse
    report completed job IDs plus byte/time receipts

scheduler:
    decrement completion fan-in
    publish manager load/store completion
    release protected request/job state

The pinned worker.py also exposes a hard limitation: completed transfer results are asserted successful, and the source says job failures are not currently supported. That observation forbids an article claim that this release already provides the full failure state machine derived below.

The documented vLLM tier path has one additional invariant. Only the pinned CPU primary tier has direct GPU access; filesystem, object, or other secondary tiers stage through it. Therefore “filesystem to GPU” is physically at least:

secondary read -> CPU primary buffer -> GPU destination page

The CPU buffer, secondary read queue, and GPU copy each need ownership and credits. A single transfer_bytes counter does not explain which stage is backpressured.

LMCache exposes a related but different integration. Its pinned lmcache_connector_v1.py forwards scheduler-side token matching, post-allocation state, and per-step metadata separately from worker-side cache registration, load start, layer-level wait, layer save, and save completion. Its vllm_v1_adapter.py tracks token IDs, allocated block IDs, saved frontiers, load/save permissions, preemption, and request configuration. A layerwise hook exposes a place to pipeline transfer; it does not prove that a particular deployment hides the transfer.

This article stops at connector-local tier orchestration. Routing a request between distinct prefill and decode workers, choosing a remote producer, and handling that network failure domain belong to M5.6.

0x08H Derive Credits Before Writing An Async Loop

Let total and already-reserved pinned host memory, GPU staging memory, and link window be (P_total,P_reserved), (S_total,S_reserved), and (C_total,C_inflight). Let (Q_total,Q_pending) describe the bounded read queue. The resources available to a new schedule step are:

$$P_{free}=P_{total}-P_{reserved}, \qquad S_{free}=S_{total}-S_{reserved},$$ $$C_{free}=C_{total}-C_{inflight}, \qquad Q_{free}=Q_{total}-Q_{pending}.$$

For an admitted request set A, let U(A) be immutable backend units after legal coalescing. Let B_u be a backend unit’s bytes and S_r be each request’s destination staging reservation. A minimal admission condition is:

$$\sum_{u\in U(A)}B_u \le \min(P_{free},C_{free}), \qquad \sum_{r\in A}S_r\le S_{free}, \qquad |A|\le Q_{free}.$$

Backend bytes and destination bytes differ deliberately. Two requests may coalesce one immutable backend read while retaining distinct GPU block slots. Charging destination staging once would assume shared physical destination pages and M5.3 refcounts that the manifest has not established.

An individual timing estimate also needs topology:

$$T_r^{estimate} = T_{queue} +T_{setup} +\frac{B_r}{BW_{effective}} +\mathbf{1}[NUMA_{src}\ne NUMA_{worker}]T_{NUMA}.$$

BW_effective is a measured service rate for the named path, not PCIe or SSD marketing bandwidth. Increasing I/O threads can raise concurrency until the device, memory controller, lock path, or queue saturates. It does not increase the physical link limit. Likewise, pinned memory is finite host state; using all of it for K/V can starve network stacks, dataloaders, or other processes.

0x08I Three Legal Policies Buy Different Request Sets

The Loop 10 fixture starts with these free resources:

Four requests then compete for two queue slots. req-alpha and req-beta share one 4,096-byte immutable backend unit but require separate destination staging. req-gamma has a declared approximate contract. req-delta cannot finish its transfer before deadline but can recompute exactly in 0.8 ms.

The third policy saves 4,096 backend bytes through legal fan-in, but it is not universally better. It prioritizes exact work over the approximate request. FIFO preserves arrival order. Earliest-deadline ordering protects slack but may admit a large approximate transfer before an exact one. Policy selection therefore belongs in the scheduler product contract from M5.2; M5.5 supplies the transfer demand and semantic fallback receipts.

0x08J Coalescing, Index Update, And GC Need Publication Receipts

Coalescing is legal only when all fields that affect bytes and meaning match:

tenant/authorization namespace
model and revision
adapter and K/V layout
layer or group and logical position range
representation, dtype, codec, and generation
source tier authority and integrity state

Coalescing by token hash alone can join incompatible model revisions or tenant namespaces. Once a transfer has multiple waiters, cancellation is also reference-counted: cancelling one request removes one waiter, not the shared read needed by the others.

Index update follows the same publication discipline as K/V movement:

def publish_index_entries(committed_chunks, index):
    generation = index.reserve_generation()
    entries = build_entries(committed_chunks)
    index.write_unpublished(generation, entries)
    receipt = index.validate(generation, entries)
    index.publish(receipt)
    return receipt

Only committed, authoritative rows may enter the searchable generation. GC can reclaim an older entry or payload only after:

no request or shared waiter holds a lease
no in-flight lookup references its generation
a newer tombstone/publication is visible
backend deletion or invalidation has a durable receipt

Otherwise an asynchronous lookup can return an index hit whose payload was already deleted, or a cancelled speculative row can become shared history.

0x08K Metrics Must Identify The Failed Stage

The pinned vLLM metrics.py exports load/store bytes, time, size, lookup sync/async delay, and allocation failure. LMCache’s pinned metrics reference further separates request/token hits, retrieve/store/lookup latency, remote bytes, CPU eviction, pinned objects, message-queue size, pending remote puts, storage events, and connector state.

Those source metrics are useful but do not replace the article’s semantic receipts:

Request IDs and logical K/V keys belong in bounded sampled traces or transaction receipts, not unbounded Prometheus labels. Metrics should use bounded dimensions such as runtime release, tier, direction, exactness, and fallback action.

0x08L Overload Is A Typed State Transition, Not More Futures

The runtime admission path can be written without pretending asynchronous work is free:

def admit_transfer(plan, credits, semantic_gate):
    if plan.predicted_finish_ns > plan.deadline_ns:
        return semantic_fallback(plan, reason="DEADLINE")
    if not credits.reserve(plan.backend_bytes,
                           plan.destination_staging_bytes,
                           plan.queue_slots):
        return semantic_fallback(plan, reason="CREDIT_PRESSURE")

    job = submit_with_generation(plan)
    return ADMIT_TRANSFER(job, credits.receipt(job))

semantic_fallback consumes the typed Loop 9 contract. It can recompute, postpone, explicitly degrade an approximation-enabled request, or reject. It cannot convert an exact request to approximate output. A transfer completion must release credits exactly once, and stale completions after reset must be rejected by generation rather than applied to a reused job ID.

LMCache’s pinned documentation includes a fault_inject adapter that can report a lookup hit and then deterministically fail selected reads. It is explicitly diagnostic-only. That is a useful failure test because it proves lookup success and payload arrival are different receipts; it is not a production recovery guarantee.

0x08M Loop 10 Executable Reconciliation And Bounded Handoff

Run all policies or isolate one policy with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/runtime_backpressure_oracle.py

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/runtime_backpressure_oracle.py \
  --policy exact_deadline_coalescing \
  --output /tmp/m5-5-runtime-policy.json

The oracle validates immutable unit/tenant/generation identity, derives free credits, charges coalesced backend bytes separately from request-owned GPU staging, rejects transfers that cannot meet deadline, executes three policies, and routes unadmitted work through typed semantic actions. It rejects 37 isolated mutations around identity, capacity, queue, topology, unit, request, exactness, and policy fields.

Loop 11 receives complete runtime receipts and source limits. It compares whole paper/runtime contracts, sweeps workload and topology controls, closes security and resource-exhaustion failures, and derives release/disable/rollback behavior. M5.6 still owns cross-worker prefill/decode placement and routing.

0x09 Evaluation: Prove Capacity, Correctness, And Latency Separately

An offload benchmark can look successful because the request no longer runs out of GPU memory while token latency becomes unusable. A retrieval benchmark can look successful because average task score is unchanged while a rare retrieval-dependent case fails. Evaluation needs independent ledgers.

0x09A Capacity Ledger

Report at least:

logical committed K/V bytes
GPU persistent K/V bytes
GPU staging and workspace peak
CPU K/V bytes
secondary/remote bytes
index and rehearsal bytes
duplicate bytes during transactions
allocator fragmentation/reserved-but-unused bytes

Useful ratios are:

$$r_{GPU} = \frac{M_{GPU,peak}}{M_{logical\ KV}}, \qquad r_{metadata} = \frac{M_{index}+M_{directory}+M_{descriptors}}{M_{logical\ KV}}.$$

r_GPU below one demonstrates reduced residency, not a complete performance result.

0x09B Transfer And Pipeline Ledger

Per generated token, collect:

requested bytes by source tier
actual bytes transferred by source tier
resident-hit bytes
coalesced-promotion savings
transfer submission latency
gather/pack latency
DMA duration
exposed wait at attention boundary
effective bandwidth under concurrency
p50/p95/p99 in-flight bytes

Derive:

$$r_{overlap} = 1- \frac{T_{exposed\ transfer\ stall}}{T_{transfer\ critical\ path}}.$$

Define the denominator carefully. If transfers and compute contend for the same memory fabric, summing isolated times can overstate the theoretically hideable window.

0x09C Retrieval Ledger

For each layer/head/query or for a justified aggregate, measure:

candidate count before deduplication
candidate count after deduplication
top-k ID recall against full attention
omitted full-attention softmax mass
attention-output relative error
full-attention fallback frequency
index miss and stale-entry frequency
pool miss frequency

An offline oracle run that computes full attention can provide reference scores and omitted mass. This is expensive, but it separates retrieval error from unrelated generation variance.

The executable oracle used by M5.4 is also the minimum retrieval-evaluation artifact for this section:

docs/numinens-evidence/m4-m5-p1/attention_oracle.py
docs/numinens-evidence/m4-m5-p1/attention_fixture.json

It consumes candidate position IDs, dense logits, and value vectors from the same layer/head/query. It then computes top-k ID recall, retained and omitted dense softmax mass, and the error between the dense output and the renormalized selected-set output. A real retrieval run should emit this fixture shape before discarding its dense oracle tensors. Generation quality remains a separate full-model measurement on the same prompt IDs; the checked-in fixture leaves that field null and is not evidence for InfiniGen or RetrievalAttention quality.

0x09D End-To-End Serving Matrix

One request is insufficient. Use a matrix such as:

Report:

TTFT
inter-token latency and its tail
tokens/s and goodput under an SLO
request rejection/timeout rate
GPU, CPU, link, and storage utilization
quality/error metrics for approximate modes

0x09E Baselines And Ablations

full GPU cache:
    establishes exact quality and no-offload latency where it fits

exact CPU offload without overlap:
    isolates raw movement cost

exact CPU offload with pipeline:
    measures overlap benefit

static resident window plus exact offload:
    measures locality benefit without approximate retrieval

retrieval with full-attention oracle:
    measures selection error

retrieval with exact fallback:
    measures quality recovery and tail-latency cost

recompute baseline:
    tests whether transfer is actually preferable for cold state

Selector ablations must follow the method’s actual work decomposition:

SparQ:
    selected query dimensions r
    selected token rows k
    local-window length
    running-mean correction on/off
    paper algorithm versus simplified runtime variant

Quest:
    physical page size
    selected page budget
    mutable-last-page policy
    dense early-layer policy
    metadata traffic and bound looseness

InfiniGen:
    calibration workload and basis generation
    rehearsal dimension fraction
    threshold alpha and candidate cap
    equalized per-head count policy
    pool policy and prefetch distance

index-based retrieval:
    index parameters and training sample
    persistent GPU set
    candidate budget and rerank depth
    rebuild cadence and CPU parallelism

Changing r, page size, or partial-column fraction changes both selector work and approximation. Report traffic, selector latency, omitted mass, output error, and fallback frequency on the same sweep; a quality-only or latency-only ablation cannot locate the trade-off.

0x09F Source-Reported Results Are Evidence, Not Capacity Planning

InfiniGen and RetrievalAttention report substantial memory and latency improvements in their evaluated environments. Keep those numbers attached to:

paper version
model and attention architecture
hardware and interconnect
context and output lengths
batch/concurrency regime
accuracy tasks and retrieval settings

A later GPU with more bandwidth may improve promotion. A later model with GQA reduces K/V bytes. A high-concurrency service can saturate CPU search or PCIe even when a single-request paper result looks favorable. Reproduce the workload before using a source-reported speedup as an SLO assumption.

0x09G Candidate IDs, Selected Units, And Attention Rows Are Different Sets

The selector families in 0x05-0x07 do not expose one interchangeable quality coordinate. They expose a chain of objects with different namespaces:

candidate row/page/cluster IDs
    -> selected physical units
    -> unit-expanded rows plus persistent rows
    -> retained dense-attention probability mass
    -> value-weighted local attention output
    -> later-model and measured task/service outcomes

Let S be the legal dense row support for one immutable (request generation, layer, K/V head, query digest). A selector returns candidate unit IDs C and chooses units U subseteq C. Let E(U) expand those units to authoritative row positions and let P be a persistent row set. The actual attention support is:

$$R=\operatorname{dedup}\left(E(U)\cup P\right)\subseteq S.$$

The deduplication is semantic, not an optimization. If a selected cluster already contains a persistent row, evaluating that row twice changes the softmax denominator. The Loop 9 cluster fixture deliberately overlaps row 0 and proves that the union contains it once.

Candidate recall against a dense top-k row set T_k is meaningful only when candidate IDs are rows in the same position namespace:

$$\operatorname{recall}_{candidate} = \frac{|C\cap T_k|}{|T_k|} \quad\text{only when }C\text{ contains row IDs}.$$

Page and cluster IDs cannot be intersected with row IDs. After expansion, support recall is always well-defined:

$$\operatorname{recall}_{support} = \frac{|R\cap T_k|}{|T_k|}.$$

Even support recall 1 does not prove dense equivalence. Retrieving the top-k rows leaves a tail. Using dense reference probabilities p_i, report:

$$\rho_R=\sum_{i\in R}p_i, \qquad \delta=\sum_{i\in S\setminus R}p_i=1-\rho_R.$$

rho_R and delta are specific to the layer, head, query, and dense support. They are not permanent properties of an index configuration.

The executable comparison makes the namespace distinction concrete:

The page method expands two page IDs to four rows, so it recovers all three top-k rows. That does not make its candidate IDs row IDs, and it does not make the omitted tail disappear.

0x09H Derive Local Output Error From The Dense Denominator

Section 0x05A introduced the mixture identity. Completing the derivation shows exactly what omitted mass controls. The dense output is:

$$o=\sum_{i\in S}p_i v_i.$$

For 0 < delta < 1, define conditional outputs over retained and omitted support:

$$o_R=\sum_{i\in R}\frac{p_i}{1-\delta}v_i, \qquad o_O=\sum_{i\in S\setminus R}\frac{p_i}{\delta}v_i.$$

Splitting the dense sum and factoring each support’s mass gives:

$$o=(1-\delta)o_R+\delta o_O.$$

Therefore:

$$o_R-o=\delta(o_R-o_O), \qquad \lVert o_R-o\rVert=\delta\lVert o_R-o_O\rVert.$$

If every ||v_i|| <= V_max, both conditional outputs are convex combinations of bounded values. The triangle inequality yields:

$$\lVert o_R-o\rVert \le\delta(\lVert o_R\rVert+\lVert o_O\rVert) \le2\delta V_{max}.$$

The bound is local and can be loose; it is not a generation-quality theorem. The oracle checks the mixture identity and bound independently for all four supports:

Relative error divides by ||o||. It becomes ill-conditioned when the dense output norm approaches zero, even if absolute error is small. A release report should publish the denominator or pair relative and absolute error.

0x09I Downstream Amplification Needs A Declared Model

The local fixture does not execute later Transformer blocks. It applies one declared synthetic linear map A only to prove how a propagation bound is accounted. For local error e=o_R-o:

$$\lVert Ae\rVert_2 \le\lVert A\rVert_2\lVert e\rVert_2 \le\lVert A\rVert_F\lVert e\rVert_2.$$

The first inequality is the induced operator-norm definition. The second holds because the spectral norm is no larger than the Frobenius norm. The oracle uses A=[[1.2,-0.3],[0.4,0.9]] and checks actual synthetic error against the Frobenius bound:

This is executable algebra, not evidence about a full model. Real downstream claims require one or more of:

paired dense/selected activation traces at later layers
logit or token-distribution divergence
named task evaluations with immutable prompts and decode settings
service evaluation with TTFT, TPOT, goodput, and fallback rates

No local bound is promoted to task quality or hardware latency in the fixture.

0x09J Derive Unit Expansion, Deduplicated Payload, And Transfer Bytes

The representation ABI fixes one full K/V row’s payload:

$$B_{row}=2d_hb_{KV}.$$

The factor 2 accounts for K and V. The Loop 9 teaching shape uses d_h=2 and FP16, so B_row=8 B. Small dimensions make every row and byte inspectable; they are not a production cache size.

For selected units U, expansion E(U), persistent rows P, and resident rows G, charge the following separately:

$$M_{candidate}=|C|B_{candidate}, \quad M_{descriptor}=|U|B_{descriptor}, \quad M_{support}=|\operatorname{dedup}(E(U)\cup P)|B_{row},$$ $$M_{transfer} = |\operatorname{dedup}(E(U)\cup P)\setminus G|B_{row}.$$

Index-probe bytes, selector MACs, descriptor bytes, K/V payload, and fixed transfer setup are different ledgers. The unit expansion ratio is:

$$A_{unit}=\frac{|E(U)|}{|U|}.$$

It describes rows per selected unit, not waste. A page can expand to extra rows that improve top-k support; a cluster can expand to rows useful for future queries. Query-local false-positive bytes require an explicit row reference.

The table does not rank token retrieval as universally superior. Page and cluster units can improve locality, coalescing, selector quality, useful prefetch, and future reuse. It proves that those benefits must be measured against physical expansion and deduplicated transfer rather than candidate count alone.

0x09K Fallback Is A Typed Prepublication State Machine

Dense omitted mass is available to an offline oracle because it evaluates full attention. An online selector cannot use delta directly unless it also runs a dense shadow path. Its gate must consume a declared estimator, confidence interval, calibration artifact, or exact verifier. The fixture therefore names estimated_omitted_mass and confidence; it never pretends they are the dense reference.

The request’s exactness contract governs every action before publication:

def choose_prepublication_action(s, gate):
    uncertain = (
        s.estimated_omitted_mass > gate.max_estimated_omitted_mass
        or s.confidence < gate.min_confidence
    )

    if not s.force_exact and not uncertain:
        return ACCEPT_APPROXIMATE
    if s.phase == "initial" and uncertain and s.can_widen:
        return WIDEN_SEARCH
    if s.exact_fetch_ms <= s.deadline_slack_ms:
        return FULL_EXACT_FETCH
    if s.exact_recompute_ms <= s.deadline_slack_ms:
        return RECOMPUTE_EXACT
    if s.retryable and s.retry_window_ms <= s.deadline_slack_ms:
        return POSTPONE_FOR_EXACT_RETRY
    if s.approximation_allowed and not s.force_exact:
        return ACCEPT_DECLARED_DEGRADED
    return REJECT_NO_LEGAL_PATH

An exact request cannot silently cross into either approximate state. Widening cannot repeat after the post_widen phase, and a postponed request publishes nothing until a later legal path succeeds.

0x09L Loop 9 Executable Reconciliation And Bounded Handoff

Reproduce the tensor, row-unit, byte, and fallback trace with:

python3 docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/error_cost_fallback_oracle.py

The oracle binds one request/cache/index/query identity, computes a dense reference, expands row/page/cluster units, deduplicates persistent overlap, checks retained mass and local output bounds, applies the bounded synthetic downstream map, charges metadata/work/payload separately, and executes all seven fallback actions. It rejects 47 isolated mutations around identity, tensor shapes, numerical representation, cost units, unit partitions, candidate/selected sets, persistent rows, quality gates, exactness, widening, retry, and expected action labels.

Loop 10 receives typed gates and separate work/byte ledgers. It must locate them in a pinned serving path, prove where credits and backpressure act, and map identity, retrieval, transfer, fallback, and queue failures to observable state. Prefill/decode worker placement remains M5.6 ownership.

0x0A Design Comparison: Which Contract Are You Buying?

The categories can be composed. A system may keep recent blocks on GPU, exact cold blocks on CPU, retrieve a query-dependent subset for normal execution, and retain an exact fallback. Composition does not erase contracts; it creates more states to test.

0x0AA A Practical Decision Sequence

1. Does the full K/V cache fit at the target concurrency?
   yes -> use the simpler exact path unless another constraint dominates

2. Can exact offload meet the link and tail-latency budget?
   yes -> establish it as the quality-preserving capacity baseline

3. Is access sparse enough that retrieval can reduce transferred work?
   unknown -> measure full-attention mass and oracle recall first

4. Can candidate selection finish early enough to overlap transfer?
   no -> retrieval may save bytes while adding latency

5. Is approximation allowed for this product/request class?
   no -> require exact fallback or decline the mode

6. Are index, pool, and cache identities transactionally consistent?
   no -> do not publish the runtime path

0x0AB Compare Complete Pipelines, Not Method Names

The comparison above is useful only after each row has a complete contract. “Retrieval,” “offload,” and “sparse attention” are not implementations. A pipeline is identified by at least ten fields:

support          which logical positions may enter attention
representation   native K/V, proxy dimensions, extrema, codes, centroids, ...
tier             where authoritative payload and search metadata reside
unit             row, page, head group, request, cluster, or event
index            block table, hash directory, ANN, PQ, LSH, hierarchy, ...
update phase     offline, prefill, append, periodic decode, or every query
compute site     GPU, CPU, storage, or a partial-state combination
exactness        exact state movement, approximate support, or estimator
fallback         wait, widen, promote, recompute, degrade, or reject
workload         fixed context, multi-turn, interrupted, or long decode

Two systems that reduce GPU-resident bytes can therefore purchase different products. HeadInfer moves full head groups to GPU and computes exact full-support attention. NEO assigns whole requests to CPU or GPU attention. SparQ first reads selected query dimensions, then fetches full rows chosen by that proxy. MagicPIG samples rows and estimates an attention result. InfLLM recalls fixed contiguous units, while EM-LLM recalls variable semantic events. Comparing their headline speedups without these axes discards the mechanism that produced them.

The registry behind this figure contains twenty pipelines. It is intentionally not a leaderboard. The following compact cut shows why:

The last row is deliberately cautious. A paper may report preserved accuracy, but that statement does not turn its selector into exact dense attention. It means the reported evaluation did not observe a disallowed quality loss under that paper’s models, tasks, budgets, and implementation.

0x0AC Derive Sensitivity Before Selecting A Design

Start from the architecture payload already derived in this article. Across all layers, one token’s dense K/V row occupies

$$B_{\text{row}} = L\,2H_{kv}d_hb.$$

For context length $T$ and concurrency $C$, the full per-request and resident service footprints are

$$M_{\text{full,req}} = T B_{\text{row}}, \qquad M_{\text{resident}} = C M_{\text{full,req}}.$$

Let $h_{reuse}$ be the fraction already available through the tier and path required by this step, $f_{sel}$ the requested fraction of the remaining cold support, and $c_{coal}$ the fraction of those requested backend bytes coalesced across requests. Data reusable only in a slower source tier does not count as a hit when this step still has to transfer it. A first-order byte ledger is

$$M_{\text{cold,expected}} = (1-h_{reuse})M_{\text{resident}},$$ $$M_{\text{selected,requested}} = f_{sel}M_{\text{cold,expected}}, \qquad M_{\text{backend,unique}} = (1-c_{coal})M_{\text{selected,requested}}.$$

With setup $t_{setup}$, queue delay $t_q$, and effective bandwidth $BW_{eff}$,

$$t_{service} = t_{setup}+t_q+ \frac{M_{\text{backend,unique}}}{BW_{eff}}.$$

For $O$ decode steps, a deliberately pessimistic no-cross-step-reuse ledger is

$$M_{\text{decode,total}} = O M_{\text{backend,unique}}.$$

This last equation is not a prediction of a real runtime. It is a sensitivity surface that exposes which missing reuse assumption would otherwise be hidden. Likewise, CPU capacity $M_{CPU}$ can hold at most

$$N_{\text{whole caches}} = \left\lfloor \frac{M_{CPU}}{M_{\text{full,req}}} \right\rfloor$$

whole dense request caches before allocator, index, and staging overhead.

The Loop 11 baseline uses $L=32$, $H_{kv}=8$, $d_h=128$, $b=2$, $T=32768$, $C=4$, $O=128$, and $h_{reuse}=1/2$. It produces a 128 KiB all-layer row, 4 GiB per request, 16 GiB resident state, 8 GiB of expected cold state, and 0.75 GiB of unique backend bytes per synthetic step at $f_{sel}=1/8$ and $c_{coal}=1/4$. At 24 GiB/s with 0.08 ms setup and 0.15 ms queueing, the transfer-only estimate is 31.48 ms. Across 128 decode steps, the deliberately pessimistic no-cross-step-reuse ledger reaches 96 GiB. That severe result is a warning: a selector is not useful merely because its fraction is less than one. Placement, reuse, overlap, and repeated decode traffic decide whether the support reduction becomes service improvement.

The evidence fixture changes one control at a time:

The central prohibition is explicit: quality is never inferred from $f_{sel}$. Loop 9 owns support, retained-mass, local-error, and task-quality evidence. Loop 11 only changes cost controls.

0x0AD Recent Methods Extend Different Axes

Three 2026 preprints help test whether this article’s taxonomy remains useful. They do not form a new common benchmark:

ScoutAttention primarily changes compute placement and overlap. IceCache changes selection units, layout, and hierarchy updates. KVDrive changes the multi-tier placement and pipeline as a whole. Their numbers are inseparable from their paper-specific models, hardware, workloads, baselines, and evaluation rules. The local registry therefore records the claims but refuses to rank them.

0x0AE Security And Crash Recovery Are Part Of Correctness

Tiered K/V state persists beyond one kernel and sometimes beyond one process. The recovery contract must cover both confidentiality and publication order:

Encryption does not replace identity. A correctly encrypted object returned to the wrong tenant, model, adapter, representation, or logical positions is still incorrect. Conversely, allocator eviction is not a deletion policy: a durable or remote copy may remain after its GPU page is reclaimed.

0x0AF Release Is A Reversible State Machine

Runtime correctness cannot be established by enabling a new path at 100% and watching aggregate latency. Use an ordered state machine:

OFF -> SHADOW -> CANARY -> RAMP -> FULL
          |          |        |
          +----------+--------+-> EXACT_BASELINE on any failed gate

Each transition binds an immutable model/adapter/representation manifest and checks separate gates:

exact mode:      exact-logit error under controlled comparison
approximate mode: declared task-quality envelope on held-out traffic
both modes:      p99 TPOT ratio and transfer-exposed stall
runtime:         allocation-failure and illegal-fallback counts
security:        tenant-isolation failures
recovery:        crash/restart and idempotent-replay tests

Shadow validates receipts without serving candidate output. Canary limits the blast radius by request class and tenant. Ramp changes one traffic fraction at a time. Full remains reversible because a later topology, model, adapter, or workload shift can invalidate earlier evidence.

The synthetic Loop 11 release fixture illustrates the policy rather than a production threshold. An exact tiered candidate passes its shadow gates and moves to canary. An approximate retrieval candidate exceeds its declared quality-drop allowance and records an illegal fallback, so it returns directly to EXACT_BASELINE. It does not remain enabled merely because its latency is attractive.

0x0AG Loop 11 Executable Reconciliation And Handoff

The executable package under docs/numinens-evidence/m5-5-kv-retrieval-offload-hierarchical-memory/ contains the twenty-entry paper-contract registry, source-reported evaluation registry, one-control sensitivity fixture, release/failure fixture, and loop11_oracle.py. The oracle derives all byte and service ledgers, advances or rolls back both synthetic candidates, reconciles ten typed failure actions, and rejects 35 isolated manifest mutations.

Loop 12 closes formula, source, rendering, visual, and cross-article boundaries. It must not add another mechanism family unless that family changes an existing contract axis and has primary evidence.

0x0B Failure Lab

0x0B0 Failure Matrix

0x0B1 Failure Injection

Do not wait for production to discover these races. A test harness should inject:

faults = [
    delay_copy_completion(layer=7, by_ms=20),
    complete_after_request_cancel(),
    reuse_gpu_block_before_old_callback(),
    corrupt_chunk_generation(),
    omit_one_rope_position(),
    publish_index_before_pool_row(),
    drop_secondary_tier_read(),
    force_candidate_cap_every_layer(),
]

Each fault needs an expected outcome. “The request completed” is not sufficient; verify cache identity, output comparison, memory reclamation, and telemetry.

0x0B2 Tensor And State Assertions

def assert_attention_inputs(plan):
    assert plan.k.shape[-2] == plan.v.shape[-2]
    assert plan.k.shape[-1] == plan.head_dim
    assert plan.logical_positions.is_strictly_compatible_with(plan.mask)
    assert_unique(plan.logical_positions)
    assert plan.owner_key == plan.request.cache_identity
    assert all(h.state == GPU_VISIBLE for h in plan.gpu_handles)
    assert all(h.generation == plan.generation for h in plan.gpu_handles)

For approximate retrieval, also log the policy version and candidate budget so an output can be traced to the selection contract that produced it.

0x0B3 Exactness Test

For exact tiering, compare against a full-GPU execution with controlled kernels and dtype:

same tokens
same model and adapter
same positions and mask
same sampling state
same kernel/dtype where possible

Compare per-layer attention outputs or logits before relying only on final generated text. A stale row can perturb logits without changing a greedy token on one test prompt.

For approximate retrieval, report the difference rather than applying the exact test’s pass/fail threshold blindly.

0x0C What This Article Restores From A25

A25’s long-context K/V article provides the local source path for retrieval/offload and its InfiniGen explanation. M5.5 reconstructs that material around a narrower systems contract:

A25 contribution retained:
    why permanent eviction loses future-important tokens
    host K/V pool
    layer-ahead partial rehearsal
    orthogonal skewing/SVD intuition
    thresholded candidate selection
    prefetch and pool-management choices

depth added here:
    exact offload as baseline control
    versioned tier transactions
    transfer and working-set derivations
    omitted-softmax-mass error derivation
    RetrievalAttention alternative
    scheduler, admission, security, and failure injection

The goal is not to shorten A25. It is to isolate one mechanism family and extend it from paper idea to an auditable runtime design.

0x0D Handoffs

To M5.4

M5.4 owns policies that permanently remove, merge, or compress logical history. M5.5 owns moving preserved history and selecting rows from a larger pool. A hybrid must state which omission came from retention and which came from retrieval.

To M5.6

M5.5 assumes a local decode worker and its memory hierarchy. M5.6 adds network ownership, prefill/decode worker placement, KV transfer protocols, and distributed cache pools. The versioned identity and publish-before-visible contracts remain applicable, but the transport and failure domain change.

To M5.7 And M5.8

MQA/GQA and MLA reduce the bytes represented by 2 H_kv d_h b or change the cached representation itself. Apply their architecture-specific payload before using M5.5’s transfer equations. Do not combine a dense-K/V formula with a latent-cache implementation.

To Output-Distribution And Document Retrieval

M5.5 owns retrieval whose result enters attention as historical state. kNN-LM-style neighbor distributions belong with the generation/output interface, while document and prompt retrieval belong with evidence acquisition and context construction. A combined system must name each index separately and must not report document recall or neighbor-label recall as K/V-row recall.

To M7

Speculative decoding creates temporary branch K/V and a commit boundary. Retrieval indexes and persistent tiers should expose committed prefix state only. Accepted branch rows can be published after their positions and generations become authoritative; rejected rows must not leak into the reusable pool.

To M8

Quantized K/V changes bytes, dequantization work, scale metadata, and numerical equivalence. “Exact offload” then means exact movement of the declared quantized representation, not equality with an unquantized baseline.

0x0E Publish Gate

• Exact offload and approximate retrieval are separate semantic contracts.
• Exact payload, physical read amplification, H2D, D2H, descriptor, and iteration traffic are separate ledgers.
• The two-slot event recurrence distinguishes data readiness, copy-engine availability, slot reuse, and layer dependency.
• The worked timeline reconciles no-overlap service, hidden transfer, uncovered stall, scheduler queueing, and request-visible latency.
• Variable staging and fixed-slot reservation are reconciled against persistent hits, new rows, descriptors, and workspace.
• The cached-to-prefill critical ratio is derived dimensionally and tested with one-control sensitivity cases.
• The exact-offload fixture rejects invalid identity, block, traffic, schedule, bandwidth, and working-set contracts.
• Per-token, per-step, transfer-time, overlap, and peak-working-set equations are derived.
• Persistent, resident, committed, and visible state are distinguished.
• Promotion/demotion use reserve-copy-validate-publish ordering.
• Generation fencing covers cancellation and physical-block reuse.
• Inactive completed turns and interrupted in-flight requests use separate manifested state machines.
• CachedAttention layerwise preload, asynchronous save, scheduler hints, and positional/truncation design are derived and scope-qualified.
• InferCept preserve, swap, discard, chunking, shared-budget ordering, and duration estimation are reconstructed from the primary paper.
• Resume rejects stale lifecycle epochs, wrong tenant/call/result, uncommitted rows, partial exact support, unverified checksum, and shifted post-position state.
• Result application is failure-atomic across reservation, cache validation, private result prefill, append commit, acknowledgment, retry, duplicate, and competing-result paths.
• The Loop 4 oracle reconciles host restore versus disk recompute and rejects 33 one-field lifecycle failures.
• Full K/V movement, head-wise staging, whole-request CPU attention, exact support partitioning, and approximate in-storage attention remain separate placement contracts.
• HeadInfer peak staging and total dense-history traffic are derived separately.
• NEO’s asymmetric overlap equation and whole-request CPU/GPU placement are reconstructed without importing M5.2 scheduling ownership.
• Remote exact attention receipts bind support, query identity, dtype, deadline, and fallback to M4.3’s (m,l,u) merge contract.
• InstInfer is classified as compute-near-data placement with approximate SparF support rather than mislabeled as exact dense attention.
• The Loop 5 oracle reconciles traffic, overlap, and dense-versus-merged output while rejecting 44 one-field failures.
• Omitted-softmax-mass error is derived rather than replaced by recall alone.
• SparQ derives salient query dimensions, L1-coverage temperature, local support, full-row recomputation, running-mean correction, and its paper/runtime variant boundary.
• Quest derives its coordinate-box upper bound, page metadata lifecycle, physical read fraction, bound looseness, and finite-budget miss contract.
• InfiniGen orthogonal skewing and SVD basis selection are derived.
• InfiniGen includes weight folding, prefill column selection, adjacent-layer tensor/code traces, threshold/cap receipts, and pool-policy separation.
• The Loop 6 oracle reconciles all three selector geometries and rejects 31 one-field failures without claiming paper-kernel or benchmark reproduction.
• Dimension-, page-, and adjacent-layer selectors remain distinct from Loop 7 index and sampling estimators.
• RetrievalAttention is treated as an alternative design with an exact union-combine derivation.
• PQCache derives product-code construction, query lookup tables, approximate MIPS, full-row fetch, separate index/payload bytes, and overlap boundaries.
• RetrievalAttention binds query-distribution adaptation, persistent/retrieved deduplication, index generations, and exact partial-state merge over an approximate union.
• SqueezedAttention derives cluster-size-weighted centroid scores, fixed-context manifests, coarse/fine hierarchy traversal, and sparse full-row attention.
• MagicPIG derives oracle sampling motivation, self-normalized importance weighting, at-least-two-table LSH inclusion probability, static-row union, and CPU/GPU placement.
• The Loop 7 oracle reconciles four distinct approximation stages and rejects 38 one-field identity, index, hierarchy, payload, and sampling failures.
• Index representation error, support omission, full-row payload error, and reduction/estimator error remain separate ledgers.
• InfLLM fixed units, EM-LLM variable events, and ClusterKV noncontiguous clusters are separated by construction, representative, lookup, payload, position, and update contracts.
• Recallable internal K/V memory is distinguished from M5.4 permanent retention, Loop 7 ordinary indexing, and document RAG.
• The Loop 8 canonical and independent oracles reconcile full-row authority, unit/event/cluster lifecycle, token budgets, update generations, and 60 isolated failure mutations without claiming paper quality or runtime performance.
• InfLLM, EM-LLM, and ClusterKV paper figures are source-attributed and their benchmark implications remain source-reported.
• Candidate row, page, and cluster IDs are separated from unit-expanded, persistent-unioned, deduplicated row support.
• Support recall, retained mass, local output error, relative-error conditioning, bounded synthetic propagation, task quality, and service behavior remain separate evidence surfaces.
• Metadata, selector work, unit descriptors, expanded rows, resident hits, K/V payload, setup bytes, and top-k false-positive/false-negative bytes are separately reconciled.
• The Loop 9 oracle executes seven exactness-labeled prepublication outcomes and rejects 47 isolated identity, tensor, unit, byte, gate, and fallback mutations.
• vLLM v0.26.0 and LMCache v0.5.2 scheduler, worker, tier, metric, and source-visible failure responsibilities are pinned to immutable commits.
• Pinned host, GPU staging, link-credit, and queue capacity are derived before admission; I/O threads are not treated as bandwidth.
• Backend-read coalescing is separated from request-owned GPU destination staging and fenced by tenant, model, representation, position, and generation identity.
• Three runtime teaching policies reconcile admitted sets, byte/staging reservations, deadline failure, exact recompute, postponement, and declared degradation.
• Index publication, cancellation fan-in, GC, and stale completion require versioned receipts.
• Metrics localize lookup, backend, host, GPU-copy, selector, and product-action stages without using request IDs as unbounded labels.
• The Loop 10 oracle rejects 37 isolated identity, capacity, queue, topology, request, exactness, and policy mutations without claiming runtime or hardware reproduction.
• Twenty paper pipelines are compared by support, representation, tier, unit, index, update phase, compute site, exactness, fallback, and workload rather than by method name alone.
• Exact movement, exact placement, compute-near-data, query-dependent fetch, compact indexing, sampling, and recallable-memory contracts remain distinct.
• Context, concurrency, output length, selector fraction, reuse, coalescing, effective bandwidth, queue delay, and CPU capacity are swept one control at a time.
• Selector fraction is explicitly prohibited from standing in for retained mass, local error, or task quality.
• ScoutAttention, IceCache, and KVDrive mechanism and evaluation claims remain source-reported and are not turned into a local or cross-paper ranking.
• Tenant isolation, authenticated payloads, deletion with leases, stale completion, index/payload divergence, crash recovery, and idempotent receipt replay are release gates.
• Release proceeds through OFF -> SHADOW -> CANARY -> RAMP -> FULL with typed rollback to an exact baseline.
• The Loop 11 oracle reconciles nine one-control sweeps, two release decisions, ten failure actions, and 35 isolated invalid mutations without claiming model-quality or hardware reproduction.
• Pool/index consistency, scheduler integration, admission, and security are included.
• Evaluation separates capacity, traffic, latency, retrieval error, and task quality.
• Failure injection and exactness tests are specified.
• Attention-state, output-distribution, and document retrieval are separated by query, returned object, and injection point.
• M5.4, M5.6, M5.7, M5.8, M7, and M8 boundaries are explicit.
• Build, formula rendering, image loading, and mobile layout are verified in the repository.

0xFF References And Provenance

Source-Local Articles

1. A25 - Exploring the Transformer Series: KV Cache Optimization for Long Text Sequences. Local source for InfiniGen, host-pool retrieval, partial rehearsal, skewing, prefetch, and pool policies.
2. M5.3 - KV Layout and Exact Reuse. Block identity, prefix reuse, and exact cache-layout boundary.
3. M5.4 - Long-Context K/V Policies. Logical retention, merging, and compression boundary.
4. M5.6 - Prefill/Decode Disaggregation and K/V Placement. Inter-worker transfer and distributed placement boundary.

Primary Papers And Official Runtime Documentation

1. J. Lee et al., InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic K/V Cache Management, USENIX OSDI 2024. Primary source for orthogonal skewing, partial rehearsal, dynamic K/V selection, host-pool management, and source-reported evaluation.
2. Z. Liu et al., RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval, 2024. Primary source for attention-aware CPU indexing, persistent GPU rows, dynamic retrieval, and partial attention combination.
3. G. Gao et al., Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention, 2024. Primary source for inactive-session K/V reuse, hierarchical placement, preloading, asynchronous saving, and position-aware truncation.
4. Y. Abhyankar et al., InferCept: Efficient Intercept Support for Augmented Large Language Model Inference, 2024. Primary source for preserve/swap/discard waste equations, swap and recompute chunking, joint interruption scheduling, original-arrival preservation, and duration estimation.
5. Y. Sheng et al., FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, 2023. Primary source for exact GPU/CPU/disk tensor placement, zig-zag block scheduling, overlap, and latency-insensitive throughput optimization.
6. S. Qin et al., Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading, arXiv:2601.19910v1, 2026. Primary preprint used for the cached-to-prefill transfer/compute boundary; measurements remain source-reported.
7. vLLM, KV Offloading Connector documentation, v0.26.0. Pinned official documentation for completed-block CPU offload, secondary-tier staging, block-size constraints, and experimental token limits.
8. vLLM, GPUWorker CPU-offload implementation at commit 568afb3a. Pinned implementation evidence for directional transfer streams, events, pinned descriptors, ordering, and byte/time receipts; no runtime benchmark is inferred from source inspection.
9. LMCache, Architecture documentation. Official, version-sensitive documentation for GPU, CPU, local, and remote K/V cache tiers.
10. Y. Wu et al., Memorizing Transformers, 2022. Primary source for a non-differentiable memory of recent internal key/value pairs retrieved with approximate nearest-neighbor search to augment attention.
11. U. Khandelwal et al., Generalization through Memorization: Nearest Neighbor Language Models, 2020. Primary source for hidden-representation lookup, neighbor next-token values, and interpolation with the language-model output distribution.
12. P. Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. Primary source for dense passage retrieval coupled to a sequence-to-sequence generator and the RAG-Sequence/RAG-Token distinction.
13. C. Luo et al., HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading, 2025. Primary source for head-group K/V staging, ping-pong transfer, adaptive grouping, and GPU head-wise attention execution.
14. X. Jiang et al., NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference, 2024. Primary source for whole-request CPU/GPU cache placement, asymmetric sub-batches, PACPU, and load-aware overlap conditions.
15. X. Pan et al., InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference, 2024. Primary source for computational-storage decode attention, SparF, dual K indexing, flash-page filtering, and GPU-CSD peer-to-peer movement.
16. H. Ribar et al., SparQ Attention: Bandwidth-Efficient LLM Inference, ICML 2024. Primary source for salient query dimensions, L1-coverage temperature, local support, selected full-row attention, running-value-mean correction, and traffic accounting.
17. Graphcore Research, sparq-llama.cpp at commit 2b5a8e1e. Pinned implementation evidence for the simplified decode-only two-stage SparQ path and alternate K layout; it does not implement every paper Algorithm 1 feature.
18. X. Tang et al., Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference, ICML 2024. Primary source for per-page key extrema, query-dependent upper-bound scoring, page selection, and source-reported layer policy.
19. MIT HAN Lab, Quest reference implementation at commit 01c1623. Pinned implementation evidence for separate metadata pages, sign-normalized bound tests, and mutable-last-page handling; the revision postdates the paper and is not treated as its exact experimental artifact.
20. H. Zhang et al., PQCache: Product Quantization-based KVCache for Long Context LLM Inference, SIGMOD 2025. Primary source for per-layer/head PQ codebooks, approximate MIPS, full-row fetch, block caching, and overlapped prefill/decode scheduling.
21. C. Hooper et al., Squeezed Attention: Accelerating Long Context Length LLM Inference, 2024. Primary source for offline fixed-context clustering, cluster-size-weighted centroid scores, hierarchical lookup, and sparse full-row attention.
22. Squeeze AI Lab, SqueezedAttention implementation at commit 4a50c64e. Pinned implementation evidence for offline centroid/label artifacts, threshold calibration, and centroid lookup kernels; no paper benchmark is reproduced locally.
23. Z. Chen et al., MagicPIG: LSH Sampling for Efficient LLM Generation, 2024. Primary source for top-k failure analysis, self-normalized importance sampling, SimHash inclusion probability, key centering, and heterogeneous execution.
24. Infini AI Lab, MagicPIG implementation at commit ac9aa36c. Pinned implementation evidence for GPU hash projection, CPU hash/K/V state, sparse CPU attention, and partial-state merge; no accuracy or performance result is reproduced locally.
25. Microsoft, RetrievalAttention repository at commit 75829e63. Lineage evidence only: the current tree primarily implements later RetroInfer/WAVE mechanisms and is not treated as the 2024 paper’s implementation artifact.
26. X. Xiao et al., InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory, NeurIPS 2024. Primary source for fixed contiguous memory units, influential representative tokens, token-relevant full-unit recall, distant-position treatment, and CPU/GPU memory placement.
27. THUNLP, InfLLM implementation at commit 12b70798. Pinned implementation evidence for context-manager tensor shapes, representative extraction, exact/FAISS unit lookup, host blocks, and bounded GPU caching; no paper benchmark is reproduced locally.
28. Z. He et al., Human-like Episodic Memory for Infinite Context LLMs, ICLR 2025. Primary source for surprise-proposed event boundaries, local graph refinement, variable-length episodic blocks, and separate semantic and temporal-contiguity retrieval.
29. EM-LLM authors, EM-LLM implementation at commit edb2e4a6. Pinned implementation evidence for modularity/conductance refinement variants, token-budgeted similarity and contiguity buffers, representative summaries, and CPU/disk offload.
30. Y. Yang et al., ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression, 2024. Primary source for cosine K-means over keys, query-centroid ranking, token-budgeted member expansion, append clusters, CPU full K/V, and selected-cluster caching.
31. SJTU Zhao Lab, ClusterKV implementation at commit c7380819. Pinned implementation evidence for prefill and periodic decode clustering, cluster-size prefix sums, full-row CPU offload, and selected-cluster GPU caching; its restricted execution assumptions are not production generality claims.
32. vLLM, Offloading connector scheduler, worker, metrics, and tier source at release v0.26.0, commit 568afb3a. Pinned source evidence for scheduler/worker ownership, primary/secondary staging, transfer receipts, and the current unsupported-job-failure limitation; no production result is inferred.
33. LMCache, Connector, storage, observability, and diagnostic fault-injection source at release v0.5.2, commit cd2c0d6a. Pinned source evidence for connector hooks, request trackers, tier metrics, and diagnostic missing-read injection; P/D routing remains M5.6 and no production result is inferred.
34. Q. Zhang et al., ScoutAttention: Efficient KV Cache Offloading via Layer-Ahead CPU Pre-computation for LLM Inference, 2026. Primary source for collaborative block-sparse CPU/GPU attention, layer-ahead CPU precomputation, periodic recall, and source-reported evaluation.
35. Y. Mao et al., IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs, 2026. Primary source for semantic clustering integrated with PagedAttention, dynamic hierarchy updates, and source-reported evaluation.
36. J. Lin et al., KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference, 2026. Primary source for joint GPU/DRAM/SSD placement, heterogeneous pipeline scheduling, cross-tier coordination, and source-reported evaluation.

Figure Provenance

• infinigen-fig6-system.png is adapted from the InfiniGen paper’s system overview and is used to explain the layer-ahead rehearsal/prefetch pipeline.
• infinigen-fig18-latency-breakdown.png is a source figure from InfiniGen and is explicitly discussed as source-reported evidence.
• numinens-m5-5-exact-offload-pipeline.svg and its mobile variant are native event-schedule reconstructions backed by exact_offload_pipeline_fixture.json; their timing is synthetic and does not reproduce FlexGen or vLLM performance.
• numinens-m5-5-critical-ratio-regimes.svg and its mobile variant reconstruct the dimensional transfer/compute boundary and the article’s one-control sensitivity cases.
• transformer25-52.jpg is a CachedAttention conceptual figure preserved through A25 and reused to introduce repeated multi-turn prefill versus persistent history.
• numinens-m5-5-session-lifecycle.svg and its mobile variant reconstruct the Loop 4 completed-turn and interrupted-request state machines with publication fences.
• numinens-m5-5-interception-decision.svg and its mobile variant visualize the synthetic two-request preserve/swap/recompute reconciliation; they do not reproduce InferCept measurements.
• headinfer-fig4-headwise-ping-pong.png reproduces HeadInfer Figure 4 under the source paper’s Creative Commons license and is used to distinguish head-wise move-to-compute from CPU attention.
• neo-fig5-asymmetric-pipeline.png reproduces NEO Figure 5 under the source paper’s Creative Commons license and is used to explain whole-request asymmetric CPU/GPU overlap.
• instinfer-fig7-architecture.png reproduces InstInfer Figure 7 and is used to distinguish in-storage placement from SparF’s approximate support semantics.
• numinens-m5-5-heterogeneous-attention-placement.svg and its mobile variant are native contract comparisons backed by heterogeneous_attention_fixture.json; byte labels are synthetic.
• numinens-m5-5-partial-state-merge.svg and its mobile variant consume M4.3’s online-state algebra and visualize the remote receipt validation added by M5.5.
• numinens-m5-5-selector-geometry.svg and its mobile variant are native comparisons of SparQ dimension-first selection, Quest page bounds, and InfiniGen adjacent-layer rehearsal; they are backed by selector_geometry_fixture.json and do not report paper performance.
• sparq-fig5-selector.png is cropped from SparQ Figure 5 and is used to preserve the paper’s two-gather plus running-mean algorithm contract.
• quest-fig5-page-criticality.png is cropped from Quest Figure 5 and is used to explain page extrema and query-dependent page ranking; its illustrated selection is not a proof of no finite-budget false negatives.
• numinens-m5-5-index-lifecycle.svg is a native Loop 7 build/publish/query/gather/reduce reconstruction backed by the synthetic fixture.
• pqcache-fig5-pipeline.png is cropped from PQCache Figure 5 and is used to distinguish compact search state from the authoritative uncompressed K/V payload.
• retrievalattention-fig4-design-index.png is cropped from RetrievalAttention Figure 4 and is used to explain CPU/GPU partial attention and query-to-key index construction.
• numinens-m5-5-centroid-hierarchy.svg is a native fixed-context hierarchy and invalidation reconstruction; squeezedattention-fig3-hierarchy.png preserves the corresponding paper mechanism.
• numinens-m5-5-topk-vs-sampling.svg is a native error-contract comparison; magicpig-fig7-placement.png preserves the paper’s GPU/CPU placement while keeping its hardware numbers source-reported.
• infinigen-fig8-10-prefetch-selector.png is cropped from InfiniGen Figures 8-10 and is used to separate offline weight preparation, prefill partial-column state, decode rehearsal, selection, and prefetch.
• numinens-m5-5-recall-unit-contracts.svg and its mobile variant are native contract reconstructions that compare a physical page, an InfLLM unit, an EM-LLM event, and a ClusterKV cluster without equating their membership or update rules.
• infllm-fig1-context-memory.png is a clean crop of InfLLM Figure 1 and is used to distinguish representative-token lookup metadata from the full selected memory-unit payload.
• emllm-fig3-event-memory.png is a clean crop of EM-LLM Figure 3 and is used to separate surprise-driven event construction, semantic retrieval, and the temporal contiguity buffer.
• transformer25-33.jpg through transformer25-36.jpg preserve ClusterKV Figures 1, 2, 4, and 5 through A25 and are used for recall taxonomy, semantic geometry, K-means/member selection, and CPU/GPU system placement. Their illustrated results are not local benchmark evidence.
• numinens-m5-5-error-chain.svg and its mobile variant are native Loop 9 reconstructions that separate candidate-unit namespaces, expanded row support, dense mass, local output error, and measured outcomes.
• numinens-m5-5-quality-cost-fallback.svg and its mobile variant reconcile the synthetic unit-expansion/transfer ledger and seven exactness-labeled prepublication actions; they are not hardware or model-quality measurements.
• numinens-m5-5-runtime-source-trace.svg and its mobile variant reconstruct the pinned scheduler/worker/tier ownership boundary and explicitly defer cross-worker routing to M5.6.
• numinens-m5-5-tier-credit-queues.svg and its mobile variant visualize the deterministic Loop 10 four-credit ledger and separate backend coalescing from request-owned staging.
• numinens-m5-5-metric-failure-map.svg and its mobile variant map bounded observability surfaces to runtime stages; metrics not present in the pinned sources remain design requirements.
• numinens-m5-5-paper-contract-matrix.svg and its mobile variant are native Loop 11 contract comparisons backed by paper_contract_registry.json; they intentionally omit benchmark rankings.
• numinens-m5-5-release-rollback.svg and its mobile variant reconstruct the deterministic Loop 11 release fixture; its thresholds and decisions are synthetic rather than production recommendations.
• Native Transformer System SVGs in this article are explanatory reconstructions. They are not benchmark figures and do not claim paper-measured scale.