M5.6 - Prefill/Decode Disaggregation and KV Placement

M5.6 - Prefill/Decode Disaggregation and KV Placement

0x00 Historical Introduction And Owner Contract

Owner question: when should prefill and decode run on different workers, where should their K/V state live, and what makes the handoff correct and worthwhile?

The question did not begin with a network. It began with a property of autoregressive generation: one request first consumes many prompt rows in parallel, then repeatedly advances by one new token. The same model therefore exposes two execution shapes inside one request. A scheduler can change when those shapes run; chunking can change their service quanta; disaggregation can change where they run. Only the last choice creates a cross-worker state handoff.

The thesis is consequently narrower than “separate phases for speed”:

P/D disaggregation trades temporal interference for a distributed transaction.

The transaction is useful only when avoided queueing and interference exceed
transfer, synchronization, placement, replication, and recovery costs while
both TTFT and TPOT objectives remain satisfied.

0x00A One Request Is An Unequal Two-Phase State Machine

For prompt tokens $x_{0:S-1}$, prefill evaluates many query positions at once and materializes K/V rows for every layer. Decode then consumes that state and evaluates one current query position per active sequence, appending one new K/V row per layer and step. The handoff object is therefore not “the prompt” or “a hidden state.” It is the complete, position-bound cache state required to continue the same causal execution.

request r
  NEW
    -> PREFILLING(prompt tokens 0 .. S-1)
    -> PREFILL_COMPUTE_COMPLETE
    -> KV_READY(model/layout/position/generation identity)
    -> DECODING(token S, S+1, ...)
    -> FINISHED | CANCELLED | FAILED

The original A26 introduction illustrates this asymmetry directly: the first iteration consumes the prompt in parallel, while later iterations generate one token at a time. The annotations are source-local and the figure is explanatory, not a local runtime measurement.

The two phases also expose different latency clocks. TTFT includes arrival, queueing, prefill, any handoff, and first-token production. TPOT or inter-token latency measures the cadence after generation starts. Optimizing only the first can starve active decodes; optimizing only the second can leave new prompts waiting indefinitely.

0x00B Request-Level Batching Hid The First Scheduling Failure

Early static batching treated a request batch as the unit of ownership. Once a batch entered the engine, newly arrived requests waited, and a short request could remain coupled to a longer request until the batch retired. That policy was poorly matched to generation because request completion lengths are unequal and not generally known in advance.

For batch $B$, a request-level release time has the rough form

$$t_{\mathrm{release}}(r_i) \approx t_{\mathrm{start}}(B) + \max_{r_j\in B} T_{\mathrm{service}}(r_j),$$

even when $r_i$ itself finished earlier. This is a scheduling artifact, not a Transformer dependency.

ORCA’s primary contribution at this boundary was iteration-level scheduling: construct the active batch again at each model iteration, and use selective batching where request-specific tensor shapes prevent an operation from sharing the same batch dimension. This lets completed requests leave and new requests enter without waiting for an entire original batch. It does not, however, make a long prompt small and it does not move prefill to another worker.

0x00C Iteration Scheduling Exposed The Second Interference Problem

Once the scheduler can mix work at iteration granularity, a new conflict becomes explicit. A decode row is latency-sensitive but usually too small to saturate the device alone. A long prefill is compute-rich and efficient, but a single large prefill iteration can delay every active decode token behind it.

decode-priority policy:
    protects token cadence
    -> prompts may queue and TTFT rises

prefill-priority policy:
    admits large prompt work promptly
    -> decode gaps may stretch and TPOT rises

This is phase interference: the two shapes compete for the same execution slots and inherit one shared parallelism and placement plan. Continuous batching solves batch membership; it does not by itself bound the duration of the work admitted into one iteration.

0x00D Chunked Prefill Changes The Quantum, Not The Owner

Chunked prefill shortens the prefill contribution to a shared iteration. M5.2’s chunked-prefill section owns chunk construction, causal-work accounting, and the distinct SARATHI, Sarathi-Serve, and pinned-runtime scheduling policies. This changes the scheduling quantum, not which worker owns each phase.

M5.6 consumes the resulting colocated service profile. Smaller chunks can protect decode cadence at an overhead cost; they do not by themselves establish that a separate prefill pool is worthwhile. That placement comparison belongs here.

0x00E Disaggregation Moves The Boundary Across Workers

DistServe and Splitwise take a different step. They assign prompt computation and token generation to different worker pools so each phase can receive its own queue, parallelism plan, hardware placement, and capacity. Temporal interference on one worker becomes a distributed pipeline:

admit request
  -> route to prefill placement
  -> produce complete prompt K/V
  -> transfer or publish that exact state
  -> admit to decode placement
  -> generate tokens under the decode SLO

The separation is not itself approximate. If both workers use compatible model state, numerical representation, cache layout, token identity, positions, and attention visibility, continuing decode from transferred K/V represents the same causal state as colocated execution. Approximation enters only if a different policy deliberately changes those tensors or the visible support.

0x00F Four Execution Lanes Before System Names

These are alternatives and building blocks, not a simple replacement chain. A disaggregated prefill pool may still use chunking; a decode pool still needs iteration scheduling; a colocated deployment may be the better answer when transfer dominates avoided interference.

0x00G The Owner Contract

This article owns:

• phase-specific worker pools and parallelism;
• prefill-to-decode K/V handoff identity, transfer, and publication;
• cross-worker placement, routing, admission, credits, and backpressure;
• distributed K/V discovery needed by the phase handoff;
• two-clock goodput under TTFT and TPOT constraints;
• P/D observability, release, disable, and rollback.

It consumes rather than re-teaches:

The depth firewall is explicit:

local offload != cross-worker phase transfer
prefix lookup != phase routing
transport completion != cache publication
raw throughput != two-clock goodput
KV-Runahead != ordinary P/D disaggregation
paper-reported performance != local evidence

The remainder of the article derives each object in that contract. System names appear only after the scheduling and handoff problem they address is defined. The next contract is deliberately mechanical: one immutable request, phase, placement, tensor, transfer, publication, and generation manifest. It will not yet choose a placement policy or claim that a paper benchmark was reproduced.

0x01 Claim Status And Scope

The article’s main distinction:

chunked colocated scheduling:
    reduce prefill/decode interference on the same workers

P/D disaggregation:
    place prefill and decode on different workers and transfer K/V

distributed KV placement:
    route around where K/V already lives or can move cheaply

0x01A Immutable Request, Placement, And Handoff Manifest

A placement estimate is meaningful only after the state being placed is frozen. Queue lengths can change while the scheduler deliberates; request, model, tensor, and generation identity cannot. The minimum handoff manifest therefore has four namespaces:

The corresponding source-to-tensor trace is deliberately mechanical:

tenant/request/generation/attempt + prompt digest + token interval
    -> checkpoint/tokenizer/adapter/position/mask identity
    -> layer/block/K-or-V/head/token/head-dimension tensor coordinates
    -> dtype/axis-order/layout/partition ABI
    -> source worker epoch + destination worker epoch + topology path
    -> transfer ID + destination reservation + expected chunk manifest
    -> received bytes/checksums (Loop 6)
    -> atomic destination publication (Loop 6)

Every arrow narrows interpretation. A checksum proves only that received bytes match the bytes named by the checksum manifest; it does not prove that the checkpoint, adapter, positional convention, tensor layout, request generation, or destination reservation is the intended one. The full identity fence is therefore (tenant, request, generation, attempt, source epoch, destination epoch, reservation), while the tensor ABI separately fixes how each byte is decoded. A stale completion can have a valid checksum and still fail the generation or worker-epoch fence.

This yields three different sets for prompt universe $U_r$:

$$\begin{aligned} C_P(r) &= U_r \setminus R_P(r),\\ S_P(r) &= R_P(r) \cup C_P(r)=U_r,\\ X_{P\rightarrow D}(r) &= U_r \setminus V_D(r), \end{aligned}$$

where $R_P$ is exact state already reusable by prefill worker $P$, $C_P$ is the prompt work that worker must compute, $S_P$ is the complete exact source state after prefill, and $V_D$ is exact state already visible to decode destination $D$. The critical invariant is:

compute_set != transfer_set in general

compute_set:
    prompt state absent from the chosen prefill worker

transfer_set:
    prompt state absent from destination-visible exact state

For example, let an eight-block prompt be [b0,b1,b2,b3,b4,b5,b6,b7]. If $P$ reuses b0..b2 but $D$ already sees only b0, then:

compute_set  = [b3,b4,b5,b6,b7]       # five blocks
transfer_set = [b1,b2,b3,b4,b5,b6,b7] # seven blocks

Using compute_set for transfer accounting would silently omit b1,b2. Those blocks need not be recomputed, but their exact state must still become visible at $D$, whether by direct transfer, a shared cache service, or another validated route.

The transfer byte contract follows the transfer set, not the compute set:

$$M_{P\rightarrow D} = |X_{P\rightarrow D}| \,B_{block} \,L \,2H_{kv}d_h \,b.$$

Here $|X_{P\rightarrow D}|$ counts transferred blocks, $B_{block}$ is tokens per block, and $b$ is bytes per scalar. Layout padding, descriptors, checksums, retries, and transport framing belong in a separate physical-traffic ledger; they must not be hidden inside logical K/V bytes.

Finally, a manifest in UNPUBLISHED state is only a reservation. Network and device-copy completion can populate destination storage, but:

transport completion != publication

Publication requires the expected identity, generation, byte/chunk coverage, checksums, and destination visibility to pass the Loop 6 transaction gate. The repository’s synthetic Loop 2 oracle deliberately stops before that gate:

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/handoff_manifest_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/handoff_manifest_fixture.json

Its toy prompt derives five compute blocks, seven transfer blocks, and 7,168 logical transfer bytes. These values test set and unit reconciliation only; they are not measured bandwidth, latency, or production correctness.

0x02 Phase Profiles

Prefill and decode are not just two names for the same forward pass.

The source A26 describes the same split: prefill can saturate GPU compute even with small batches or long prompts, while decode repeatedly reads the KV cache and often needs larger batches to expose enough work. SARATHI externally verifies the same shape: prefill saturates compute at small batch sizes, while decode generates one token at a time per request and has low utilization.

The consequence:

one GPU configuration rarely optimizes both phases equally

Prefill may want more compute and tighter prompt latency. Decode may want stable large batches, high memory bandwidth, and predictable token gaps. If both phases share the same worker, the scheduler is always trading TTFT against TPOT.

0x02A Derive Both Phases From Rows And Causal Pairs

The qualitative labels become useful only after one accounting convention produces both phase ledgers. M4.1 owns the full model-operation convention; M5.6 consumes a deliberately bounded transformer-block model to expose the phase difference. Let one layer have model width $d$, feed-forward width $d_{ff}$, $H_q$ query heads, $H_{kv}$ K/V heads, and head width $d_h$. For a three-matrix SwiGLU MLP, the modeled parameters per layer are:

$$P_{layer} = d(H_qd_h)+2d(H_{kv}d_h)+(H_qd_h)d+3dd_{ff}.$$

The terms are Q, K/V, output projection, and the gate/up/down MLP matrices. With one multiply-add counted as two FLOPs, applying those matrices to $N$ new rows across $L$ layers costs:

$$F_{linear}(N)=2LNP_{layer}.$$

Attention is controlled by causal pairs, not just new rows. If prefill adds $N_P$ rows after $T_0$ already-visible rows, its query-key pairs per head are:

$$A_P=N_PT_0+\frac{N_P(N_P+1)}{2}.$$

The first term attends from every new query to the old prefix. The triangular term is the causal attention among the new rows themselves. For one decode iteration with ragged visible lengths $s_1,\ldots,s_B$ after each current row is appended:

$$A_D=\sum_{i=1}^{B}s_i.$$

QK transpose and probability-times-V each contribute one dot product, so the modeled attention matmul work is:

$$F_{attn}(A)=4LH_qd_hA.$$

This leaves normalization, elementwise operations, embeddings, sampling, and the vocabulary head outside the toy block ledger. It is not a substitute for M4.1’s complete estimator; it is the smallest shared derivation that makes the P/D phase asymmetry auditable.

Persistent K/V production follows a different coefficient:

$$B_{KV/token}=LH_{kv}d_h(b_K+b_V), \qquad B_{KV,write}=N\,B_{KV/token}.$$

For the same synthetic example, unique visible K/V state is counted once:

$$B_{KV,read}^{P}=(T_0+N_P)B_{KV/token}, \qquad B_{KV,read}^{D}=\left(\sum_i s_i\right)B_{KV/token}.$$

That is a state-byte lower bound, not measured kernel traffic. A tiled kernel may reload some state, retain some tiles on chip, fuse writes, or avoid materializing scores entirely. Likewise, a naive logical score tensor would contain $H_qA$ scalars, while FlashAttention changes its materialization and I/O schedule. M4.3-M4.6 own that kernel derivation; M5.6 records only the phase work handed to the runtime.

The executable fixture uses $L=4$, $d=64$, $d_{ff}=160$, $H_q=8$, $H_{kv}=2$, $d_h=8$, and two-byte weights/K/V:

The example is small on purpose. It proves that the same weights can be paired with many new rows and dense causal work in prefill, but only a few new rows and a large retained-state scan in decode. It does not prove where a real kernel sits on a hardware roofline.

0x02B Interference Is A Shared Service-Slot Contract

For policy reasoning, package each phase as a work vector rather than a single label:

$$W_{phase}= (F,\ B_{weights},\ B_{KV,read},\ B_{KV,write},\ A,\ N_{new}).$$

Given synthetic compute rate $C$ and state-byte rate $R$, a lower-bound service model is:

$$\underline{t}(W)= \max\left(\frac{F}{C}, \frac{B_{weights}+B_{KV,read}+B_{KV,write}}{R}\right).$$

At $C=10^{12}$ FLOP/s and $R=10^{11}$ B/s, the illustrative ridge is 10 FLOPs per state byte. The prefill vector above lies on the compute side and has a 5.513 microsecond lower bound. The decode vector lies on the state-byte side and has a 3.661 microsecond lower bound. If one mixed iteration reads the weights once and releases decode tokens only after all mixed work completes, its lower bound is 6.647 microseconds: 1.816 times the decode-only slot.

That ratio is a synthetic cadence-dilation bound, not a latency prediction. Real kernels, launch overheads, batching efficiency, quantization, cache hierarchy, parallel collectives, and overlap can move every number. The useful contract is narrower:

decode-only slot:
    completion clock follows decode work

mixed prefill/decode slot:
    decode rows cannot retire before the shared slot completes

therefore:
    phase mixing couples TPOT to admitted prefill work

The complete derivation and its one-field failures are executable:

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/phase_cost_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/phase_cost_fixture.json

0x03 Colocated Interference

In a colocated runtime, one worker runs both phases:

worker batch =
    some prefill rows
    plus some decode rows

That can be a good engineering choice. It avoids inter-worker KV transfer. It keeps deployment simpler. It lets decode rows piggyback on a compute-heavy prefill iteration when the batch is shaped carefully.

But interference is the hard case:

t0  active decode requests A/B/C need their next token
t1  new request D arrives with a long prompt
t2  D prefill is admitted to the same worker
t3  D prefill dominates the iteration
t4  A/B/C wait for the shared iteration to finish
t5  their next tokens arrive late

Source: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, Figure 1. The curves are the paper’s 13B-model, A100, synthetic-workload measurement; they provide evidence for phase interference, not a universal throughput ratio.

The tradeoff is symmetric:

DistServe frames the same issue as resource coupling: colocated systems batch prefill and decode together, causing strong interference and forcing one phase’s latency target to compete with the other.

0x03A Consumed Iteration And Service Profile

M5.2 owns iteration scheduling and selective batching. M5.6 consumes its selected work, not a second scheduler implementation:

IterationPlan
    iteration_id
    selected request IDs and immutable request generations
    phase and input-token positions for this call
    request-local K/V handles and visibility ranges
    physical reservations supplied by M5.3

ServiceProfile
    workload, model, hardware, kernels, and policy version
    selected prefill/decode rows and their phase work
    measured or explicitly modeled iteration service and queue delay
    TTFT/TBT outcomes under a named offered load and resource budget

Selection is not committed progress. M5.1 owns validated request-local state transitions, and M5.3 owns physical reservations. Here, one selected iteration becomes one colocated phase service slot. Dynamic membership alone does not bound that slot’s duration or prove that decode meets its SLO.

0x04 Chunked Prefill Is The Bridge

Before paying for a second worker pool, compare against a strong colocated scheduler. M5.2 derives the chunked-prefill alternatives: original SARATHI anchors a batch with one prefill chunk; Sarathi-Serve protects ongoing decode progress under a profiled token budget; the pinned vLLM path has its own frontier and queue contract. They are not interchangeable algorithms.

This article consumes the chosen policy’s service profile: prompt and output mix, offered load, resource budget, local K/V residency, phase service times, queueing, and both latency gates. Keep those coordinates matched when comparing colocation with disaggregation, and label modeled times separately from measurements. A chunk cap or a token budget is not itself a latency guarantee.

Continuous decode progress can still be slower than decode-only service. M5.2’s worked cadence and pipeline-imbalance comparison makes that distinction executable without treating a synthetic schedule as a production performance result. M5.6 uses the resulting cadence and resource costs in the placement decision; it does not reconstruct the scheduler.

0x04G Colocation Versus Disaggregation

Chunking keeps one worker pool and avoids cross-worker K/V transfer. That can be preferable when a service profile finds a chunk cap satisfying both clocks and when shared model-weight reads improve mixed-batch efficiency. It does not independently tune phase parallelism, eliminate all cadence dilation, or free decode workers from prompt K/V reservations.

P/D disaggregation instead pays duplicated model residency, routing, transfer, publication, and failure-recovery costs in exchange for independent phase placement and interference isolation. Neither dominates by definition:

prefer bounded colocation when:
    profiled chunks satisfy TTFT and TBT
    transfer and duplicated-residency costs would dominate
    one placement serves both phase envelopes well

consider disaggregation when:
    phase-specific parallelism or hardware materially differs
    long-prompt bursts break the colocated TBT gate
    two-SLO goodput gains exceed handoff and capacity costs

The A26 boundary figure is source-local synthesis, not a decision oracle:

0x05 Disaggregated Workers

A disaggregated serving path looks like this:

request arrives
router selects prefill worker P
router selects decode worker D
P runs prompt prefill
P produces prompt K/V
K/V moves from P to D
D verifies K/V visibility
D enters decode loop
D appends generated-token K/V

Source: DistServe, Figure 6. The controller, phase-specific instance pools, and KV transfer arrow expose the placement object that the following ledger quantifies.

This can help for three reasons:

1. Prefill and decode can use different resource allocations.
2. Each phase can use different parallelism or batching strategy.
3. Decode workers can protect token cadence while prefill workers absorb prompt bursts.

It can also hurt for three reasons:

1. K/V transfer adds latency and bandwidth pressure.
2. Routing becomes a cluster-topology problem.
3. Decode cannot start until the required K/V state is visible and correct.

SplitWise emphasizes hardware adaptation: prompt computation is compute-intensive, token generation is memory-intensive, and phases may benefit from different machine choices. DistServe emphasizes goodput under TTFT/TPOT constraints and bandwidth-aware placement. MemServe and Mooncake push the same idea further: K/V becomes distributed memory with scheduler-visible placement.

0x06 KV Transfer Ledger

Different workers do not imply that every prompt K/V row must cross the network. Transfer the required state missing at the destination, after validating exact identity and visibility. The earlier handoff contract defines X_{P->D} = U_r \ V_D(r); source-side prefix reuse instead reduces C_P = U_r \ R_P(r), the compute set. A source cache hit does not automatically make the destination warm.

For a uniform, full-layer K/V representation and token-aligned transfer:

S_transfer = number of required token rows missing at the destination
M_KV_transfer = S_transfer * L * 2 * H_kv * d_h * bytes

L           transformer layers
2           K and V
H_kv        key/value heads
d_h         head dimension
bytes       bytes per scalar

Count each missing row once. If transport sends complete physical blocks, include transmitted padding; for partial-layer state, compression, or another cache representation, sum actual tensor/block payload bytes instead. The earlier block-set ledger supplies that more explicit contract.

For multiple requests with the same geometry:

M_transfer_total = sum_r S_transfer_r * L * 2 * H_kv * d_h * bytes

Different geometries require a per-request coefficient. The payload-only service estimate is:

t_payload ~= M_KV_transfer / bandwidth_effective

This is an illustrative ledger, not a profiler. Effective payload bandwidth depends on topology, memory copies, protocol efficiency, congestion, and scatter/gather layout. Setup, validation, synchronization, queueing, and publication add costs; legal overlap can hide part of the service time. A zero-byte handoff therefore does not imply zero handoff latency.

Running configuration:

L = 32, H_kv = 8, d_h = 128, bytes = 2
Delta_M_token = 32 * 2 * 8 * 128 * 2
              = 131072 bytes = 128 KiB

The following figure and first table are specifically a cold-destination, full-prompt baseline, where every required prompt row is missing:

Source and destination reuse affect different ledgers. For 4096 required rows, with full-row transfers and exact, compatible reusable state:

The third row holds the chosen prefill placement fixed: destination reuse alone does not populate the source. Rerouting to exploit that state is a separate placement decision. Only destination entries passing the model, position, tensor-ABI, generation, and visibility fences count as reusable.

Illustrative payload-only transfer times:

Those bandwidths are examples, not claims about a specific cluster. The point is scale: missing K/V can exceed a handoff budget unless movement is fast, legally overlapped, compressed under a declared exactness contract, or avoided through placement and valid destination reuse.

Mooncake measures that topology and software stack directly. Its transfer engine remains below the TCP and Gloo latency curves as cache payload grows in the two tested network configurations, but latency still increases with bytes. The figure therefore supports both halves of the ledger: a better transfer path changes effective bandwidth, while the payload term remains real.

Source: Qin et al., Mooncake, Figure 12. The experiment uses concurrency 64, minimum transfer granularity 128 KB, and the displayed 4 x 200 Gbps and 8 x 400 Gbps NIC configurations. The reported transfer-engine advantage is specific to that implementation and topology; it is not a generic network-bandwidth multiplier.

After transfer, decode still grows:

Delta_M_decode_token =
    L * 2 * H_kv * d_h * bytes
  = 128 KiB

M5.7 changes H_kv; M5.8 changes the cached representation itself. M5.4 may reduce logically visible T, while M5.5 may reduce locally resident T or stage only a retrieved subset. Those change transfer bytes, but they are not themselves P/D placement mechanisms.

0x06A Transport Completion Is Not Publication

The byte ledger says how much state must move. It does not say when that state is safe to consume. A transfer protocol has at least four different completion events:

network send complete
    source transport may reuse its send descriptor

destination receive complete
    bytes reached destination-side staging memory

DEVICE_VISIBLE
    the destination accelerator may read the copied bytes on the declared stream

PUBLISHED
    every expected chunk passed identity, coverage, checksum, and visibility checks,
    and one atomic directory update made the complete cache visible to decode

Collapsing any adjacent pair creates a failure mode. A receive completion does not prove that a device copy is visible. One device-visible chunk does not prove that all layers and token intervals exist. Publication does not by itself grant the source permission to release its only recoverable copy.

The transaction therefore has a request-level state machine and a finer chunk-level state machine:

request transaction:
    PLANNED -> RESERVED -> TRANSFERRING -> VALIDATED -> PUBLISHED
                    \             \            \
                     +-------------+-------------> ABORTED

chunk:
    DECLARED -> CREDIT_ACQUIRED -> SENT -> RECEIVED -> DEVICE_VISIBLE

PUBLISHED is the only request state accepted by the decode scheduler. The source release is a separate acknowledgement after publication, not another way to make an incomplete destination valid.

The immutable key spans more than a request ID:

The transfer ID may make a repeated delivery idempotent, but only when the identity and payload checksum also match. A duplicate ID with different bytes is corruption, not a successful retry.

0x06B A Byte-Credit Window Bounds Unpublished Transfer State

A destination reservation and a transport credit solve different problems:

destination cache reservation:
    protects capacity for the complete prompt K/V and later decode growth

byte-credit window:
    bounds bytes owned by network and destination staging/copy work
    before DEVICE_VISIBLE returns the corresponding credit

For credit capacity $C_{credit}$, let $a_j$ and $r_j$ be the acquisition and return events for chunk $j$, and let $m_j$ be its bytes. The invariant is

$$0 \le B_{inflight}(t) = \sum_j m_j\mathbf{1}[a_j \le t < r_j] \le C_{credit}.$$

The sender must acquire byte credit before issuing a chunk. This article’s protocol returns that credit at DEVICE_VISIBLE, not at network receive, because destination staging and device-copy work still own the buffer between those events. A different implementation may define a different safe return event, but it must name the resource that becomes reusable and prove that no later operation still depends on it.

The local teaching rule is therefore explicit: release credit at DEVICE_VISIBLE.

def transfer_chunk(txn, chunk, credits, destination):
    credits.acquire(chunk.nbytes)
    try:
        send(chunk.payload, transfer_id=txn.transfer_id)
        received = destination.receive(chunk.chunk_id)
        device_copy = destination.copy_to_reserved_kv(received)
        wait_until_device_visible(device_copy, destination.stream)
        txn.mark_device_visible(chunk.chunk_id, chunk.checksum)
        # This credit protects staging/copy ownership, not total KV residency.
        credits.release(chunk.nbytes, event="DEVICE_VISIBLE")
    except Exception:
        # No release here: transport may still own the staging buffer.
        # abort_unpublished returns this credit after it fences callbacks.
        txn.mark_failed(chunk.chunk_id)
        raise

Returning credits does not free the request’s reserved cache blocks. Those blocks remain unpublished until commit, then become decode-owned until the ordinary M5.1/M5.3 lifecycle releases them.

0x06C Streaming Overlap Has A Producer Gate And A Consumer Gate

Layer- or block-streamed transfer can hide work only between two legal gates:

producer gate:
    send_start(chunk) >= K/V-ready event for that exact layer/token interval

consumer gate:
    publish_time >= max(device_visible_time for every expected chunk)

Sending earlier reads state that prefill has not produced. Publishing earlier exposes a partial cache. Waiting for the whole request before the first send is correct, but gives up overlap.

The checked-in synthetic fixture uses four layers, two 1 KiB chunks per layer, and a 2 KiB credit window. The final layer becomes producer-ready at 86 ms; all chunks become device-visible by 116 ms; validation and publication occur at 118 ms and 120 ms.

These values test ordering and arithmetic. They do not measure RDMA, PCIe, HBM, a production network, or a paper implementation.

0x06D Validate, Publish, Admit, Then Acknowledge

Publication is one compare-and-swap-like ownership transition, not the last line of every chunk callback:

def validate_and_publish(txn, expected, decode_directory):
    assert txn.identity == expected.identity
    assert txn.tensor_abi_hash == expected.tensor_abi_hash
    assert txn.chunk_ids == expected.chunk_ids
    assert txn.coverage == expected.layer_token_coverage
    assert txn.checksums == expected.checksums
    assert all(c.device_visible for c in txn.chunks)

    handle = decode_directory.publish_if_generation_current(
        request_id=txn.request_id,
        generation=txn.generation,
        transfer_id=txn.transfer_id,
        unpublished_handles=txn.destination_handles,
    )
    txn.state = "PUBLISHED"
    decode_scheduler.admit(handle)
    acknowledge_source_release(txn.transfer_id, txn.generation)

The order matters:

1. Validate exact identity, tensor ABI, complete layer/token coverage, checksums, and device visibility.
2. Publish the complete destination handle atomically under the current generation.
3. Admit decode only from that published handle.
4. Acknowledge source release under the same generation after the chosen recovery policy permits it.

A decode reservation may exist before transfer starts, but it is not permission to read unpublished blocks. Likewise, publication may be retried idempotently; it must not create two owners or replace a newer generation.

0x06E Failure Atomicity And An Executable Oracle

Every failure before publication follows one cleanup path:

def abort_unpublished(txn, reason):
    if txn.state == "PUBLISHED":
        raise CannotAbortPublishedTransfer(txn.transfer_id)
    cancel_outstanding_chunks(txn.transfer_id)
    wait_for_callbacks_or_fence_worker_epoch(txn.destination_epoch)
    release_staging_credits(txn.transfer_id)
    release_unpublished_destination_blocks(txn.destination_handles)
    txn.state = "ABORTED"
    record_terminal_failure(txn.transfer_id, txn.generation, reason)

The local oracle checks eight chunks, complete layer/token coverage, checksums, producer-ready times, a 2 KiB byte-credit window, device visibility, atomic publication, decode admission, and source-release order. Twenty-four one-field mutations prove that each fence is active:

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/transfer_protocol_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/transfer_protocol_fixture.json

The oracle is a synthetic protocol test. Mooncake, DistServe, Splitwise, and MemServe motivate efficient movement and distributed ownership, while M5.3’s cache ABI and PagedAttention’s block model constrain the destination handles. The exact transaction and credit event names here are a Transformer System integration contract, not a claim that every cited system implements the same state machine.

0x07 Transfer Schedule: Request, Layer, Or Stream

The simplest handoff is:

run full prefill
transfer full prompt K/V
start decode

That is easy to reason about, but it can leave decode waiting behind a large transfer.

The more aggressive option is streamed or layer-wise transfer:

for each layer or chunk as prefill produces K/V:
    send K/V to target decode worker
    overlap later prefill compute with earlier K/V transfer

The tradeoff:

By-layer transfer is not automatically better. It can reduce the exposed handoff time, but it makes correctness and recovery more complex. The decode side must know exactly which layers, blocks, and token ranges are complete.

0x08 Placement And Routing

Once K/V transfer is real, placement is no longer “pick two workers.” It is a two-timescale decision:

offline or periodic placement:
    choose phase parallelism, GPUs per instance, replica counts, and topology

online routing:
    choose one prefill replica and one feasible decode replica for a request

The first decision determines the capacity and link envelope that the second is allowed to use. A short-queue router cannot repair an infeasible model shard, insufficient decode K/V memory, or a placement whose transfer demand exceeds the available path.

0x08A Freeze The Search Coordinates

A placement result is meaningful only for one immutable coordinate set:

DistServe fits a workload distribution from historical traces, builds a phase latency model from profiling and interpolation, enumerates feasible parallel configurations, and uses simulation plus binary search to estimate the maximum rate satisfying the attainment goal. The Loop 7 fixture is deliberately smaller: it consumes checked latency samples on an immutable rate grid. It does not reproduce the paper’s simulator.

0x08B Why Parallelism Preference Changes With Rate

The prefill queueing argument starts with an M/D/1 approximation. Let requests arrive as a Poisson process at rate $R$, and let deterministic prefill service take $D$ seconds. Utilization is $\rho=RD<1$. The Pollaczek-Khinchine waiting-time term is:

$$W_q = \frac{R\,\mathbb{E}[S^2]}{2(1-\rho)} = \frac{RD^2}{2(1-RD)}.$$

Adding service gives the paper’s single-device approximation:

$$\operatorname{AvgTTFT} = D+\frac{RD^2}{2(1-RD)}.$$

At low rate, the first term dominates, so an intra-operator strategy that reduces one request’s execution time can win. As rate approaches capacity, the denominator shrinks and queueing dominates. A pipeline can then win by increasing stage throughput even if one request’s end-to-end service time is not lower.

For the paper’s bounded two-way comparison, inter-operator parallelism assumes request latency $D_s\approx D$ but a slowest-stage service interval $D_m\approx D/2$:

$$\operatorname{AvgTTFT}_{inter} \approx D+\frac{RD^2}{4(2-RD)}.$$

For two-way intra-operator parallelism with measured speedup $1<K<2$, service is $D/K$:

$$\operatorname{AvgTTFT}_{intra} \approx \frac{D}{K} + \frac{RD^2}{2K(K-RD)}.$$

These equations explain the tradeoff; they do not choose a universal winner. $K$ changes with tensor shape, model, kernels, and interconnect, while variable prompt lengths violate the deterministic-service assumption. DistServe uses profiles and simulation for the actual search.

Decode has a related but different boundary. Tensor/intra-operator parallelism can reduce one decode step when TPOT is stringent, but communication and smaller local matrices create diminishing returns. Once the TPOT gate is met, pipeline/inter-operator parallelism or replication can scale rate. Decode K/V capacity also bounds the active batch. M4 owns the detailed TP/PP collective and pipeline ledgers; M5.6 consumes candidate service and memory profiles.

The A26 source-local diagram introduces inter- versus intra-operator coordinates. Its taxonomy is background, not evidence that a particular candidate wins:

0x08C From Attainment Samples To Phase Goodput

For candidate prefill configuration $c_p$ and request rate $\lambda$, define the empirical TTFT attainment over $N$ held-out samples:

$$a_f(\lambda,c_p) = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{TTFT_i(\lambda,c_p)\le\tau_f\}.$$

Decode attainment is analogous:

$$a_o(\lambda,c_d) = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\{TPOT_i(\lambda,c_d)\le\tau_o\}.$$

The phase goodputs at attainment target $p$ are:

$$g_p(c_p)=\max\{\lambda:a_f(\lambda,c_p)\ge p\}, \qquad g_d(c_d)=\max\{\lambda:a_o(\lambda,c_d)\ge p\}.$$

DistServe’s high-node-affinity search selects each phase by goodput per GPU, not by the lowest single-request latency:

$$c_p^*=\arg\max_{c_p}\frac{g_p(c_p)}{G(c_p)}, \qquad c_d^*=\arg\max_{c_d}\frac{g_d(c_d)}{G(c_d)}.$$

For target traffic $R_{target}$, replica counts are:

$$n_p=\left\lceil\frac{R_{target}}{g_p(c_p^*)}\right\rceil, \qquad n_d=\left\lceil\frac{R_{target}}{g_d(c_d^*)}\right\rceil.$$

The resulting capacity is bottlenecked by the smaller phase pool:

$$g_{system} = \min\left(n_pg_p(c_p^*),\ n_dg_d(c_d^*)\right).$$

The synthetic oracle uses ten samples per rate, 90% attainment, TTFT 250 ms, and TPOT 100 ms:

At target traffic 10 rps, the fixture needs two selected prefill replicas and one selected decode replica. Six GPUs provide prefill capacity 16 rps and decode capacity 12 rps, so the system bottleneck is 12 rps, or 2 rps/GPU. The selected configuration also holds three complete model copies: 396 GiB of synthetic model residency. Disaggregation’s additional residency is part of the resource ledger, not a footnote.

0x08D Topology Can Invalidate The Phase Optimum

For logical K/V bytes per request $M_{KV}$ and system rate $g_{system}$, the minimum payload demand is:

$$B_{demand}=g_{system}M_{KV}.$$

The fixture’s 256-token MHA cache has 64 layers, eight K/V heads, head width 128, and two-byte K/V scalars:

$$M_{KV}=256\cdot64\cdot2\cdot8\cdot128\cdot2 =64\ \text{MiB}.$$

At 12 rps, logical payload demand is 768 MiB/s. A synthetic high-affinity path with 4 GiB/s payload capacity is feasible. A 256 MiB/s cross-node path is not. This is still only logical demand: headers, retries, contention, registration, copy engines, and overlap belong in the Loop 6 physical trace.

DistServe’s low-node-affinity algorithm uses a structural fact: K/V transfer occurs between corresponding model layers. With pipeline parallelism, prefill stage $j$ and decode stage $j$ can be placed in the same node so their handoff uses the high-bandwidth intra-node path. The fixture puts both prefill replicas’ stage 0 and decode stage 0 on node 0, and all stage 1 segments on node 1. Each node uses three of four GPUs; the 8 GiB/s intra-node path passes.

This distinction is conditional:

high node affinity:
    search phase configurations independently
    replicate to target rate
    verify aggregate payload and physical path capacity

low node affinity:
    co-search compatible PP stage geometry and intra-node packing
    pair corresponding prefill/decode stages
    reject configurations that fit numerically but cannot be placed

The paper reports that its stage-local placement made transfer less than 0.1% of total latency for its OPT-175B/ShareGPT breakdown and kept more than 95% of transfers below 30 ms on its testbed. Those are source-reported consequences of that specific placement, not a general promise that K/V transfer is small.

0x08E Online Routing Consumes The Placement

DistServe’s runtime sends a request to the shortest prefill queue, then to the least-loaded decode instance. It uses a pull-style handoff under burstiness so the decode side fetches K/V when it has capacity, leaving completed prompt K/V as a buffer on the prefill side. The paper also periodically replans when the observed workload distribution shifts. The primary-paper runtime figure in 0x05 already shows this controller and phase-instance path; it is not repeated here.

The generic M5.6 router adds exact reuse and transfer-state facts supplied by the other M5 owners:

Colocated scheduling can ask:

which worker has capacity to run this request?

Disaggregated routing asks:

which prefill worker should run the prompt?
which decode worker should receive the K/V?
is transfer time acceptable?
will decode capacity still satisfy TPOT?
is there an exact prefix hit or local K/V already available?

A placement estimate:

predicted_TTFT =
    queue_prefill
  + prefill_time
  + transfer_time
  + decode_admission_delay

predicted_TPOT =
    decode_step_time(decode_worker, active_batch, KV_residency)

A simple route admission:

accept route (P, D) only if:
    predicted_TTFT(P, D) <= tau_f
    predicted_TPOT(D) <= tau_o
    transfer_buffer_available(P, D)
    decode_KV_capacity_available(D)

DistServe places phases according to resource allocation and cluster bandwidth. Mooncake’s scheduler makes the same issue more explicit: the best worker is not always the one with the longest prefix hit if that worker is loaded or the KV transfer path is expensive.

Conceptual routing pseudocode:

def route_request(req):
    candidates = []

    for P in prefill_workers:
        for D in decode_workers:
            prefill_reuse = exact_state_visible(req, P)
            decode_reuse = exact_state_visible(req, D)

            compute_set = req.prompt_blocks - prefill_reuse
            transfer_set = req.prompt_blocks - decode_reuse

            prefill_time = estimate_prefill(P, compute_set)
            transfer_bytes = logical_kv_bytes(transfer_set, req.cache_schema)
            transfer_time = estimate_transfer(P, D, transfer_bytes)
            decode_tpot = estimate_decode_tpot(D, req)

            ttft = queue(P) + prefill_time + transfer_time + admission_delay(D)

            if ttft <= tau_f and decode_tpot <= tau_o:
                candidates.append((P, D, ttft, decode_tpot, transfer_bytes))

    return choose_best_goodput_route(candidates)

This pseudocode is not DistServe Algorithm 1 or 2. It is a cross-module routing interface. M5.2 owns fairness and preemption; M5.3/M5.5 provide exact cache visibility; Loop 6 owns handoff publication. A route is not admitted until its decode reservation, transfer credits, model identity, and K/V capacity are all valid.

The executable placement reconciliation is:

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/distserve_placement_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/distserve_placement_fixture.json

Its rate samples and links are synthetic. It checks arithmetic and ownership, not DistServe’s simulator accuracy or evaluation performance.

0x09 Distributed KV Pools

DistServe and SplitWise split phases. MemServe and Mooncake move toward a stronger idea:

K/V is a named distributed object,
not just temporary worker-local state.

A distributed KV pool needs:

• allocation and ownership metadata;
• active and historical KV distinction;
• request-to-cache index;
• transfer APIs;
• placement and routing policy;
• failure detection and invalidation;
• admission control that accounts for future decode growth.

MemServe frames LLM serving as stateful and introduces an elastic memory pool that manages distributed memory and KV caches across serving instances. Mooncake makes the architecture KVCache-centric: prefill and decode clusters coordinate through distributed cache placement, asynchronous transfer, and global scheduling.

This is not the same as M5.3’s block table. M5.3 asked how to address a request’s exact K/V inside a memory manager. M5.6 asks where that memory object should live across workers.

0x0A Goodput Under Two SLOs

Raw throughput is not enough. A system can emit many tokens per second while delivering a bad product:

• prompts wait too long before the first token;
• active streams stutter between tokens;
• tail latency violates the service contract.

For measured requests $i=1,\ldots,N$ at offered rate $\lambda$, keep the two attainment functions separate:

$$\hat a_f(\lambda) = \frac{1}{N}\sum_i\mathbf{1}\{TTFT_i\le\tau_f\}, \qquad \hat a_o(\lambda) = \frac{1}{N}\sum_i\mathbf{1}\{TPOT_i\le\tau_o\}.$$

DistServe’s two-phase goodput contract can then be written as:

$$\lambda_{good} = \max\left\{\lambda: \hat a_f(\lambda)\ge p \quad\land\quad \hat a_o(\lambda)\ge p \right\}.$$

Here $\tau_f$ and $\tau_o$ are the TTFT and TPOT thresholds, and $p$ is the required attainment fraction. The aggregation window, warmup, cancellation policy, failed-request denominator, TPOT definition, and confidence interval must be fixed with the SLO. Otherwise two systems can report the same “90% attainment” from different denominators.

Separate phase attainment is not automatically the same as joint per-request attainment. If the release contract requires each counted request to pass both clocks, measure:

$$\hat a_{joint}(\lambda) = \frac{1}{N}\sum_i \mathbf{1}\{TTFT_i\le\tau_f\ \land\ TPOT_i\le\tau_o\}.$$

In general, $\hat a_{joint}\le\min(\hat a_f,\hat a_o)$. The Loop 7 oracle reconstructs DistServe-style phase-level configuration search; it does not infer a joint probability from marginal samples.

For a deployment with phase capacities $G_p$ and $G_d$, request goodput cannot exceed the slow phase:

$$G_{system}=\min(G_p,G_d).$$

Per-GPU goodput divides by every GPU provisioned across both pools, including replicas whose extra model residency is part of the cost:

$$G_{per\ GPU} = \frac{G_{system}} {n_p\operatorname{gpus}(c_p^*)+n_d\operatorname{gpus}(c_d^*)},$$

where $\operatorname{gpus}(c)$ is GPUs per instance. This denominator prevents the common mistake of reporting the decode bottleneck rate while ignoring prefill replicas and duplicated model copies.

P/D disaggregation helps only if the new placement increases this SLO-constrained rate relative to the strongest comparable colocated baseline. A higher raw request rate with worse token gaps is not better goodput. A lower TTFT with broken TPOT is not better goodput either. A simulator estimate is a planning result; only an immutable workload replay on the target release is a release measurement.

0x0B When Disaggregation Helps

Disaggregation tends to help when:

workload:
    many long prompts
    active users need smooth streaming
    colocated prefill bursts delay decode

cluster:
    enough bandwidth or locality for K/V transfer
    prefill and decode pools can be sized independently
    routing can keep decode workers fed without queue explosions

Then the system can:

• use prefill workers for compute-heavy prompt bursts;
• keep decode workers focused on stable token cadence;
• choose separate parallelism/resource plans per phase;
• route around queue and transfer bottlenecks;
• improve goodput under both TTFT and TPOT SLOs.

This is the DistServe case.

0x0C When Disaggregation Hurts

Disaggregation tends to hurt when:

workload:
    short prompts
    short outputs
    low concurrency

cluster:
    poor cross-worker bandwidth
    high routing overhead
    decode workers wait on K/V visibility
    cache placement metadata is stale or congested

Then colocated serving can be simpler and faster:

no remote K/V transfer
no global placement decision
no decode-side wait for visibility
fewer failure modes

The red flag:

separate workers remove one shared bottleneck,
but they add a network and placement bottleneck.

0x0D System Variants Change Different Boundaries

“P/D disaggregation” describes a family, not one architecture. A26 preserves an important lineage: colocated systems first improve scheduling inside one worker, then disaggregated systems choose different boundaries for phase pools, cache ownership, transfer, elasticity, and prediction.

Consumed Colocated Baseline: ORCA And SARATHI

M5.2 owns the ORCA request-pool algorithm, selective-batching tensor trace, chunk causal algebra, and the distinct SARATHI, Sarathi-Serve, and pinned-vLLM scheduling contracts. M5.6 consumes only their resulting colocated service profile:

ORCA baseline:
    membership can change at an iteration boundary

SARATHI baseline:
    long prefills can be chunked and mixed with decode rows

service profile exported to M5.6:
    measured or modeled TTFT, TPOT, phase service time,
    queue interference, local K/V residency, and no transfer path

This article neither re-derives those scheduler policies nor treats their source-reported gains as a universal colocated optimum. It uses the strongest declared colocated profile as the baseline for evaluating separation:

compare two-SLO goodput at matched offered load and resource budget:
    best feasible colocated scheduling profile
    versus disaggregation with transfer, queueing, and capacity costs included

The comparison is only valid when workload, model, hardware, SLO, and measurement coordinates match. M5.6 now owns what changes after the boundary: phase-specific workers, transfer, topology, independent scaling, and placement.

DistServe: Optimize Parallelism And Placement Per Phase

Sections 0x08 and 0x0A reconstruct the full DistServe contract. The short variant boundary is:

The paper’s selected parallelism is workload-specific. Its Appendix B, for example, reports different prefill and decode TP/PP coordinates across OPT-13B, OPT-66B, OPT-175B, ShareGPT, HumanEval, and LongBench. Those rows illustrate why phase plans may differ; they are not defaults for another model or cluster.

The reported 7.4x maximum request-rate gain, 12.6x tighter SLO, transfer-time CDF, simulator agreement, and search runtime all remain source-reported. The local oracle establishes only that the final form of the placement accounting is internally consistent and rejects topology, fit, attainment, or claim-scope mutations.

SplitWise: Prompt, Token, And Mixed Pools

DistServe asks which independent prefill and decode configurations maximize two-SLO goodput. Splitwise starts from a different operational concern: a configuration chosen from yesterday’s prompt/output distribution can fragment today’s capacity. A prompt burst can leave token machines underused; a long generation tail can do the reverse. Its third pool is therefore not another permanent phase. It is a reversible escape from a dedicated role:

prompt pool:
    prompt computation only

token pool:
    token generation only

mixed pool:
    machines temporarily admit work from both phases
    while retaining an origin role to which they return

This contract is stronger than saying “add a fallback GPU.” All machines have the model loaded. A request is assigned a prompt machine and token machine together, the token destination must reserve memory for the future cache, and the prompt machine can begin transferring layer state toward that known destination while later layers still compute.

0x08F The Two Schedulers Own Different Decisions

Splitwise uses a hierarchical scheduler rather than one omniscient queue. The cluster-level scheduler (CLS) owns pool membership and request routing. Each machine-level scheduler (MLS) owns local memory accounting, its pending queue, and the batch admitted at an iteration boundary.

The CLS uses join-the-shortest-queue, but the queue coordinate in the paper is pending tokens, not request count. For phase role $r$, a teaching form is:

$$m_r^*=\arg\min_{m\in\mathcal M_r} Q_m^{tokens},$$

subject to model identity, role epoch, and K/V headroom. The paper says workers report queue and memory changes periodically, not at every iteration. A stale report can therefore make the nominal shortest queue wrong; the destination must still reject an admission whose local memory fence no longer passes.

def route_splitwise(req, cluster_snapshot):
    prompt = jsq_by_pending_tokens(cluster_snapshot.prompt_pool)
    token = jsq_by_pending_tokens(
        workers_with_kv_headroom(cluster_snapshot.token_and_mixed, req)
    )

    if prompt.pending_tokens > spill_threshold:
        prompt = jsq_by_pending_tokens(cluster_snapshot.mixed_pool)
        if prompt is None:
            prompt = borrow_opposite_pool_machine(req)
            transition_role(prompt, to="mixed")

    reserve_decode_kv(token, req.cache_manifest)
    return bind_pair(req, prompt, token, snapshot_epoch=cluster_snapshot.epoch)

The last three lines are a Transformer System hardening of the paper-level description. Loop 2 owns the immutable model/cache manifest and Loop 6 owns publication. Routing does not make a stale role, an unreserved token destination, or an unpublished layer consumable.

0x08G Mixed Is A Reversible State, Not A Fourth Hardware Type

If the selected dedicated queue exceeds a threshold, the CLS first looks in the mixed pool. If that pool is full, it may borrow a machine from the opposite dedicated pool. The machine retains its origin role and returns when opposite- phase work drains. Larger distribution shifts trigger infrequent coarse repurposing between prompt and token pools.

A production state trace should be explicit even though the paper reports no noticeable pool-switch latency because models are already loaded:

TOKEN(role_epoch=12)
    -- spill threshold crossed -->
MIXED(role_epoch=13, origin=TOKEN)
    -- opposite-phase queue drains -->
TOKEN(role_epoch=14)

Every queued item is admitted under one role epoch. A worker must not silently interpret a token-only reservation under a later mixed-role policy. The local oracle increments the epoch on both transitions and rejects a stale role assignment. That epoch is rewrite interpretation, not a claim that the Splitwise prototype exposes this exact field.

The capacity ledger also must not add phase rates as though mixed execution created a second machine. Let $N_P,N_T$ be dedicated counts, $b$ borrowed prompt-origin machines, $g_P,g_T$ dedicated request capacities, and $g_M$ the profiled end-to-end capacity of one mixed machine under the target prompt/output distribution. Then the fixture uses:

$$G_{static}=\min(N_Pg_P,N_Tg_T),$$ $$G_{mixed}=\min\left((N_P-b)g_P+bg_M,\;N_Tg_T+bg_M\right).$$

With two prompt workers at 6 rps and two token workers at 4 rps, $G_{static}=\min(12,8)=8$ rps. Borrowing one prompt-origin worker whose mixed profile sustains 4 rps gives prompt-side capacity 10 rps and token-side capacity 12 rps, hence 10 rps system capacity. The same mixed 4 rps appears in both phase constraints because those requests execute both phases there; it is not 8 rps of independent work. These rates are synthetic reconciliation coordinates, not Splitwise measurements.

0x08H Local Scheduling Is Phase Specific

The paper’s MLS policies follow the phase asymmetry:

The prototype uses a 2048-prompt-token batching threshold. That number came from its model/hardware characterization; it is a profile coordinate, not a portable constant. For an FCFS queue [1000, 700, 600], a 2048-token cap admits the first two entries (1700 tokens) and leaves the third for a later batch. Reordering the queue to fill the residual 348 tokens would violate FCFS.

Mixed-machine prompt priority needs a second contract. Prompt arrivals can preempt token work when no batch capacity remains, but repeated preemption can destroy TBT. Splitwise raises a token request’s priority with age and caps its preemption count:

def choose_mixed_phase(prompt_pending, token):
    if not prompt_pending:
        return "token"
    if token.age_ms >= age_override_ms:
        return "token"
    if token.preemptions >= max_preemptions:
        return "token"
    return "prompt"

M5.2 owns the general fairness and preemption proofs. M5.6 records why a mixed P/D worker requires those proofs and checks three boundary cases: a fresh token is preempted, an aged token runs, and a token at the cap runs.

0x08I Serialized And Per-Layer Transfer Are Two Different Schedules

Request-level transfer waits for prompt computation to finish, then moves the complete prompt cache. Per-layer transfer starts an asynchronous put after each layer produces its K/V while the next layer computes:

For layer $l$, let $r_l$ be K/V-ready time, $x_l$ transfer duration, and $v_l$ destination-visible time on one serialized transfer engine. The exact teaching recurrence is:

$$v_l=\max(r_l,v_{l-1})+x_l,\qquad v_{-1}=0.$$

The request-level schedule instead completes at:

$$C_{request}=r_{L-1}+\sum_{l=0}^{L-1}x_l.$$

For ready times [25, 50, 75, 100] ms and four 10 ms transfers, per-layer visibility is [35, 60, 85, 110] ms. The serialized request completes at 140 ms. Per-layer streaming exposes 10 ms after final prompt compute and hides 30 ms of transfer, before synchronization/interference overhead. The common max(C_prefill,C_transfer) expression is only a lower-bound intuition; the recurrence reveals pipeline fill, serialization, and a slow-link tail.

Splitwise selects serialized transfer for small prompts and per-layer transfer for larger prompts because fine-grained synchronization can itself increase TTFT. Its reported H100 crossover below 512 prompt tokens is an evaluation coordinate, not a universal threshold. The prototype uses MSCCL++ one-sided puts, a per-request semaphore, and block-contiguity coalescing. The semaphore signals only after all layer puts have been issued. Loop 6 adds the stronger consumer rule: issued is not device-visible, validated, or published.

0x08J What The Oracle Proves

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/splitwise_pool_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/splitwise_pool_fixture.json

The oracle reconciles pending-token JSQ, simultaneous prompt/token binding, K/V headroom, mixed-role entry and return, FCFS prompt batching, starvation guards, phase bottleneck capacity, transfer-method selection, and the per-layer visibility recurrence. It does not reproduce the Splitwise simulator, provisioning search, scheduling quality, transfer measurements, or cost/power results.

The paper reports less than 7% visible transfer overhead in its experiment, roughly 8 ms/5 ms non-overlapped per-layer transfer on its A100/H100 setups, and up to 1.4x throughput with 20% lower cost or 2.35x throughput under equal power and cost. These are source-reported results under the paper’s traces, models, interconnects, SLOs, and simulator. The durable lesson is the contract: phase-specific pools need a mixed escape state, hierarchy-aware routing, starvation control, and a transfer schedule selected from prompt geometry.

0x08K AA, HH, HA, And HHcap Are Provisioning Coordinates

Pool control decides how requests consume a chosen fleet. Provisioning decides which fleet exists. SplitWise evaluates four assignments:

These suffixes are not performance rankings. The provisioning search consumes a pinned model performance profile, prompt/output trace, TTFT/TBT/E2E SLOs, hardware and network coordinates, a throughput constraint, and an objective such as cost or provisioned power. It then searches prompt- and token-machine counts with the paper’s event-driven simulator.

The design alternatives expose different risks:

The paper’s 27-prompt/3-token design-space example, hardware comparisons, and cost/power/throughput results remain source-reported. The local oracle does not reconstruct its simulator or infer that A100, H100, or a specific power cap is universally correct for either phase.

KV-Runahead: Pipeline Causal Prefill Across Processes

KV-Runahead is neither ordinary tensor/sequence parallel prefill nor P/D disaggregation. It parallelizes prompt-cache generation itself by partitioning the causal context across ordered processes and passing the growing K/V prefix forward. Tensor/sequence parallelism distributes one forward graph and uses collectives. Ordinary P/D separation runs prompt generation on one phase owner and transfers the completed cache to a decode owner. KV-Runahead instead creates the prompt cache through an ordered chain inside the prompt phase.

The paper’s comparison makes the ownership change visible. The TSP baseline uses equal context partitions and layer-synchronous all-gather. KV-Runahead uses unequal contiguous partitions, point-to-point sends, and layer-misaligned execution because a later process consumes the prefix produced by its predecessor.

These are source-reported paper diagrams, not local measurements. They establish the dependency topology. M4 still owns tensor/sequence-parallel collective derivations; this section only needs that baseline to explain why KV-Runahead is a different prompt-cache construction contract.

0x0CA Partition Dense Blocks, Not Only Exact Causal Pairs

Let process p own n_p consecutive prompt rows and let:

$$C_p=\sum_{i < p}n_i$$

be the number of earlier rows. Every local query sees C_p earlier keys and its own causal local prefix. Two work conventions must be kept separate.

An ideal triangular local kernel evaluates only valid score pairs:

$$W^{\triangle}_p =n_pC_p+\frac{n_p(n_p+1)}{2}.$$

The paper’s block argument assumes the usual dense local matrix multiplication followed by a causal mask. Its local diagonal block is therefore an n_p x n_p square:

$$W^{\square}_p=n_pC_p+n_p^2.$$

Summing the dense-block convention over all processes gives:

$$\begin{aligned} \sum_p W^{\square}_p &=\sum_p n_pC_p+\sum_p n_p^2 \\ &=\frac{C^2-\sum_p n_p^2}{2}+\sum_p n_p^2 \\ &=\frac{C^2+\sum_p n_p^2}{2}, \end{aligned}$$

where C = sum_p n_p. The first equality partitions the lower off-diagonal rectangles; the second uses C^2 = sum_p n_p^2 + 2 sum_p n_p C_p. This is why the paper’s TTFT lower-bound model contains both C^2 / 2 and sum n_p^2 / 2. A custom triangular kernel can remove part of the local diagonal waste, but it does not remove pipeline dependencies or the need to balance processes.

For a synthetic 96-token prompt on three processes:

The latter two coordinates are synthetic. They show why earlier processes can own more rows and later ones fewer: equal token count makes the last process the dense-block bottleneck. The paper does not derive a universally optimal closed form because compute, communication, kernels, and topology all perturb the minimum.

0x0CB Traffic Is A Prefix Sum, Not An All-Gather

For equal partitions across P processes, the paper counts TSP traffic in K/V-token entries as:

$$\operatorname{Net}_{\mathrm{TSP}}(C,P)=(P-1)C.$$

Each of the P processes receives the other P-1 shares. In KV-Runahead, link i -> i+1 carries the prefix accumulated through process i. With equal partitions, the network sees:

$$\begin{aligned} \operatorname{Net}_{\mathrm{KVR}}(C,P) &=\sum_{i=1}^{P-1}\frac{iC}{P} \\ &=\frac{P-1}{2}C. \end{aligned}$$

The paper’s factor-of-two statement applies to this total equal-partition entry model. It is not a promise that every link, rank, arbitrary uneven partition, or measured transfer time is halved.

For arbitrary partitions, define S_i = sum_{j<=i} n_j. Logical K/V bytes are:

$$B_{\mathrm{KVR}} =L\left(2H_{kv}d_hb\right)\sum_{i=0}^{P-2}S_i.$$

The factor in parentheses is one token’s K and V payload for one layer. Headers, alignment, packing copies, retransmission, and protocol state are excluded. In the synthetic searched coordinate above, S=[40,72]; with four layers and 4096 bytes per token per layer, the logical traffic is:

(40 + 72) * 4 * 4096 = 1,835,008 bytes

That is more than the equal KVR partition’s 96 token-rows but still below the equal TSP coordinate’s 192 token-rows. Partition search therefore balances a critical path, not traffic alone.

0x0CC The Layer Trace Has Two Overlap Windows

The paper inserts receive before local-cache concatenation and send immediately after concatenation. Receive can overlap Q/K/V projection. Send can overlap the local attention/softmax work, provided the transmitted buffer remains immutable until completion.

for layer_id, layer in enumerate(model.layers):
    recv_future = (
        irecv_prefix(layer_id, predecessor, recv_buffer[layer_id % 2])
        if rank > 0 else completed(empty_kv())
    )

    q_local, k_local, v_local = layer.qkv_projection(local_hidden)
    prefix_kv = recv_future.wait()       # required before concatenation
    visible_kv = concat(prefix_kv, k_local, v_local)

    send_future = (
        isend_prefix(layer_id, successor, visible_kv)
        if rank + 1 < world_size else completed()
    )
    local_hidden = layer.causal_attention(q_local, visible_kv)

    retain_until_complete(visible_kv, send_future)

The minimal event order is:

IRECV_PREFIX
QKV_PROJECTION       # may overlap receive
WAIT_PREFIX
CONCAT_LOCAL_KV
ISEND_PREFIX
CAUSAL_ATTENTION     # may overlap send
LAYER_DONE

This is a tensor-level dependency, not merely a scheduler preference. A later rank cannot attend a layer before its exact earlier-token K/V is visible. Physical fragmentation creates another boundary: the paper notes that efficient communication expects contiguous buffers. A paged runtime may need contiguous prompt allocations, an iovec-capable transport, or an explicit packing copy. M5.3 remains the cache-ABI and physical-layout owner; M5.6 records how that ABI constrains the pipeline.

0x0CD KVR-E, KVR-S, And KVR-P Are Different Contracts

The paper’s hierarchical search coarsens a partition grid, selects a promising region, then refines the stride. Its lookup key must therefore include more than prompt length in a production system: model, process count, kernel, dtype, topology, and cache layout can change the optimum. Interpolation is valid only inside a compatible manifest; it is not evidence that one table transfers across hardware.

The final process owns the only complete prompt cache. Decode must either start there, or the cache must pass through the Loop 6 transfer transaction:

complete every layer on the final KVR process
validate model/cache ABI and exact token coverage
publish the complete cache
admit decode locally or transfer to a selected decode owner

Publishing an intermediate process’s prefix as a complete prompt cache is a correctness bug. So is reusing a searched partition after a topology or kernel change without invalidating its profile epoch.

0x0CE Executable Causal-Pipeline Closure

alternative_pipeline_oracle.py recomputes dense-block and ideal triangular work, arbitrary-partition prefix traffic, final-cache ownership, event order, and publication order for KVR-E, KVR-S, and KVR-P. Its numbers are synthetic contract evidence, not a reproduction of the paper’s Llama/Falcon, A100, bandwidth, or TTFT results.

Failure and tuning surfaces now become concrete:

equal context partition despite causal work imbalance
lookup table reused across an incompatible model/kernel/topology epoch
receive consumed before the complete predecessor prefix is visible
send buffer overwritten while an asynchronous transfer still reads it
paged K/V packed on the critical path without accounting for the copy
slow process or link creates a layer pipeline bubble
partial cache on an intermediate rank is published as complete
decode placement ignores the final complete-cache owner

The paper’s speedups, custom-kernel discussion, and bandwidth experiments remain source-reported. The durable contribution is a causal work partition, an ordered K/V pipeline, and a hardware-indexed partition-selection contract.

MemServe: Memory Pool Is A First-Class Distributed Service

MemServe shifts the owner question again:

not only:
    which P worker sends to which D worker?

also:
    which distributed memory tier already owns reusable context state?

Its MemPool abstraction supports inter-request reuse, intra-request movement, and elastic memory capacity. Compute workers publish and consume K/V through a distributed memory plane rather than a single direct P-to-D edge.

The first conceptual change is lifetime. Active K/V belongs to an ongoing request and can still grow. Historical K/V is retained after a request (or stable prefix) finishes and is indexed for another request. Moving a block from active to historical state is not a byte copy alone; it is a publication event that creates a reusable token-prefix mapping.

0x09A MemPool Exposes Memory, Index, And Transfer APIs

The paper’s API families separate concerns that ordinary P/D handoff code often fuses:

An address encodes its owning instance. An address list therefore says where bytes are allocated, not whether the associated cache is valid for a model, request, layer, token range, or RoPE coordinate. The Loop 2 manifest remains the semantic identity; the MemPool address is physical location.

The index granularity also consumes the inference engine’s allocator. In the paper’s adapted vLLM example, radix-tree nodes point to the engine’s 16-token cache blocks. A different block size changes match granularity, address-list length, transfer-call count, internal fragmentation, and eviction choices.

0x09B Transfer-With-Insert Is A Distributed Transaction

The primary paper decomposes transfer into allocation, transmission, and optional insertion. The hardened M5.6 ordering is:

sender -> receiver: ALLOCATE(requested tier, bytes, cache manifest)
receiver -> sender: ADDRESSES + destination parallelism/layout
sender: reshape/partition only under an explicit compatible transform
sender -> receiver: TRANSMIT blocks over selected path
receiver: wait for device visibility and validate coverage/digest/identity
receiver: INSERT token-prefix -> address-list mapping when requested
receiver: PUBLISH mapping and cache generation
receiver -> sender: ACK

The destination parallelism response matters. A TP=2 prompt owner and a PP=2 decode owner do not necessarily expose the same block ownership or address order. A transfer implementation must reconcile the exact source/destination layout; a byte count alone cannot prove compatibility.

def transfer_with_insert(tx, destination):
    addresses, destination_layout = destination.allocate(tx.manifest)
    try:
        chunks = transform_layout(tx.source_blocks, destination_layout)
        transmit(chunks, addresses)
        wait_device_visible(addresses)
        validate_identity_coverage_digest(tx.manifest, addresses)
        destination.insert_unpublished(tx.prefix_keys, addresses)
        destination.publish(tx.generation)
        return ACK
    except Exception:
        destination.abort_index(tx.generation)
        destination.free(addresses)
        raise

Transport completion is still not publication. If the sender times out after transmission but before insertion, the destination must not expose an orphaned mapping. If insertion succeeds but the acknowledgement is lost, transaction ID and generation make retry idempotent instead of allocating and publishing a second copy.

0x09C Parallelism, Network, And Memory Are Three Heterogeneities

MemPool’s simple API hides a non-simple path:

parallelism heterogeneity:
    source and destination partition blocks differently

network heterogeneity:
    HBM-HBM, HBM-DRAM, or DRAM-DRAM choose different primitives/copies

memory heterogeneity:
    active or historical blocks may currently reside in HBM or CPU DRAM

The paper prototype used NCCL send/receive for GPU paths and sockets when a side was in DRAM. That implementation is a pinned paper coordinate, not a general MemPool requirement. Modern RDMA, GPUDirect, or runtime connectors may choose another route, but they inherit the same identity, allocation, visibility, and failure obligations.

0x09D Caching Is Added To Disaggregation In Four Milestones

MemServe’s progression makes ownership changes explicit:

PD-Caching-1 preserves prompt-produced history but repeatedly transfers the same state in multi-turn workloads. PD-Caching-2 publishes at decode so a locality- aware scheduler can reuse decode-side history and send only incremental K/V. PD-Caching-3 transfers completed decode history back to prefill, allowing the prefill-side historical prefix to grow across turns.

The return path is not automatically worthwhile. It consumes network and prefill-tier capacity today to avoid possible recomputation tomorrow. MemServe therefore needs the context-caching cost model to compare prompt length, cached ratio, cache locations, batch size, and block size.

0x09E Paging Creates A Network-Call Geometry

Suppose each of $L$ layers stores separate K and V blocks. The discrete layout has $2L$ block addresses. Under the paper’s primitive model:

For the fixture’s four layers, the counts are 8/8, 1/8, and 1/1. Aggregation reduces launch and network-API pressure but waits for the aggregate region and can increase copy, packing, and temporary-memory cost. Per-layer transfer exposes more overlap but cannot reduce below layer-granular synchronization.

The paper reports that by-layer won under low load while request aggregation won at higher load in its setup. That crossover depends on block count, network primitive, queueing, and prompt compute; it is not a universal transfer rule.

0x09F Index Locality Must Compete With Load

The global scheduler maintains prompt trees associated with prefill-only, decode-only, and colocated instances. A match returns both a reusable prefix and the address/instance ownership needed to reach it. Longest-prefix routing alone can create a hot instance, so policy consumes the distributed load table too:

$$C(i) = T_{queue}(i) +T_{move}(prefix\rightarrow i) +T_{prefill}(S-P_i),$$

where $P_i$ is the usable prefix length at candidate $i$. The candidate with the largest $P_i$ need not minimize $C(i)$.

On instance failure, in-flight remote operations time out and the cluster manager broadcasts updated membership. Allocations owned by the failed instance must be reclaimed and its index locations invalidated. A stale global tree entry must fail closed, not route a request to bytes that no longer exist.

The benefit is decoupling compute placement from one cache copy. The cost is that cache lookup, network transfer, index consistency, membership, and transaction recovery become part of every routing decision.

TetriInfer: Predict Decode Length To Avoid Hotspots

TetriInfer begins from mixed downstream workloads rather than one fixed prompt/output coordinate. Summarization can have long prompts and short outputs; content generation can reverse that ratio. Light and heavy prefill interfere once a mixed batch exceeds the accelerator’s useful prompt-token region. Light and heavy decode interfere because the scheduler does not know future working set and lifetime. Separating the phases removes one interference edge, but it does not reveal output length.

The paper’s workflow combines four objects: a centralized control plane, prefill instances, decode instances, and a small length-prediction model. Its timeline shows why a predicted heavy decode can be directed toward on-demand decode capacity instead of becoming a long-lived hotspot in one mixed worker.

This source-reported figure establishes the paper design; it does not prove the local runtime has reproduced its latency or resource results.

0x0DA Predict A Bucket, Then Carry Its Bounds

TetriInfer does not claim reliable exact output-length regression. Temperature, top-p, and the target model make the same prompt stochastic. The paper trains a small classifier offline to predict one of G output-length buckets.

Given prompt x, target-model output length y, and bucket boundaries [a_g, a_{g+1}), the training label is:

$$g^*(y)=g\quad\text{such that}\quad a_g\le y<a_{g+1},$$

and the predictor minimizes classification cross entropy:

$$L_{\mathrm{pred}}=-\log p_\phi(g^*(y)\mid x).$$

The paper’s data-generation trace is:

prompt-only dataset
  -> run the pinned target model with pinned generation settings
  -> measure generated-token length
  -> map length to a bucket label
  -> split train/evaluation data
  -> fine-tune and evaluate the small classifier

The dataset manifest must include target model, tokenizer, context limit, generation parameters, stop policy, prompt distribution, bucket boundaries, and predictor version. Otherwise a bucket means different resource usage after any of those coordinates change.

For predicted bucket [a_g,a_{g+1}), scheduling should carry a bound rather than silently replacing it with a point estimate:

$$R_{kv}(r)\in \left[a_gB_{kv/token},\;a_{g+1}B_{kv/token}\right].$$

The upper bound is a conservative reservation coordinate. The lower bound is useful for progress and reclamation. Smaller buckets improve resolution but are harder to classify; one giant bucket is accurate but carries almost no scheduling information. The paper evaluates a 200-token granularity and reports its accuracy; that coordinate is source-reported, not a universal default.

The paper also compares predictor execution choices:

The audited paper chooses parallel execution and a predictor at each prefill instance. It trains offline and explicitly leaves online fine-tuning outside its implementation.

0x0DB Two Scheduling Levels Consume Different Knowledge

The global scheduler observes prefill-instance load and chooses only a prefill owner. Prompt length is known, so the local prefill scheduler can order FCFS, SJF, or LJF and then slice/merge prompts into fixed-size prefill-only chunks. Unlike SARATHI’s colocated mixed chunks, TetriInfer does not insert decode rows into those prompt chunks.

The prefill dispatcher owns the second decision. It receives a versioned decode load snapshot from the cluster monitor and the request’s predicted output bucket. The paper’s inter-instance route is:

reservation = bucket_upper * kv_bytes_per_generated_token

alpha = [
    worker for worker in decode_workers
    if worker.free_memory >= reservation
]
beta = [worker for worker in decode_workers if worker not in alpha]

left, right = power_of_two_sample(alpha)
decode_owner = min(
    (left, right),
    key=lambda w: heavy_to_light_ratio_after_admission(w, request),
)

alpha and beta are capacity sets, not quality labels. The paper then uses a power-of-two sample inside alpha and chooses the candidate with the lower heavy-to-light interference coordinate. A production route also needs snapshot version, age, model residency, K/V transfer reachability, tenant policy, and fallback behavior. A stale snapshot can make a previously feasible alpha member oversubscribed before the K/V arrives.

The dispatcher finally sends request metadata and the completed prompt cache. The paper discusses chunk-level transfer as a compute/network overlap alternative but implements request-level transfer for simplicity. It is incorrect to describe chunk-level transfer as a measured TetriInfer path.

0x0DC Reserve-Static And Reserve-Dynamic Are Working-Set Policies

The decode receiver waits until the request metadata and prompt K/V are valid before enqueuing the request. Its local scheduler then compares three policies:

Let M_cap be decode K/V capacity, M_active current reservations, R_new the new request’s upper-bound reservation, and R_short the reservation released by the shortest predicted remaining job. The teaching gates are:

$$\begin{aligned} \mathrm{static}&:\quad M_{active}+R_{new}\le M_{cap},\\ \mathrm{dynamic}&:\quad M_{active}-R_{short}+R_{new}\le M_{cap}. \end{aligned}$$

The second inequality is not free capacity now. It is a promise about the next release event. If the supposed shortest job generates beyond its bucket, the promise can fail and the runtime needs a correction path.

def admit_with_prediction(request, state, policy):
    reserve = request.bucket.upper * state.kv_bytes_per_token

    if policy == "reserve_static":
        return state.used_bytes + reserve <= state.capacity_bytes

    shortest = min(
        state.active,
        key=lambda r: (r.predicted_remaining_tokens, r.request_id),
    )
    projected_used = state.used_bytes - shortest.reserved_bytes
    return projected_used + reserve <= state.capacity_bytes

M5.2 remains the generic scheduler owner for fairness, admission, preemption, and overload. M5.6 owns the narrower fact that a cross-phase output-length prediction becomes a decode-placement and reservation input.

0x0DD Prediction Error Must Reconcile During Decode

Prediction changes uncertainty into explicit error; it does not remove it. Track at least:

bucket coverage by workload and generation policy
underprediction tokens and bytes, especially p90/p99
overprediction tokens and stranded reservation bytes
route changes caused by prediction
swap/preemption events after underprediction
unused capacity caused by overprediction
predictor latency and contention with main prefill
snapshot age at dispatch

Underprediction is the dangerous tail: the request outgrows its reservation, can turn one decode instance into a memory/TPOT hotspot, and invalidates reserve-dynamic completion assumptions. Overprediction lowers utilization and may route a request farther than needed. On each generated token, a runtime can reconcile observed length against the bucket, expand a reservation if headroom exists, stop new admissions to the affected worker, or invoke an explicit preemption/migration policy. Those are local hardening choices, not claims about the TetriInfer prototype.

Instance role changes have a separate safety sequence. The paper’s control plane stops new routes, drains queued/in-flight work, and only then flips the virtual prefill/decode role. A role epoch should invalidate old snapshots and late K/V sends; changing a role flag without draining creates the same stale owner problem described in the Splitwise section.

0x0DE Executable Prediction/Placement Closure

The synthetic alternative_pipeline_oracle.py validates contiguous length buckets, a versioned load snapshot, upper-bound K/V reservation, the eligible decode set, a deterministic power-of-two teaching route, reserve-static versus reserve-dynamic gates, prediction coverage/error, and the paper’s implemented request-level transfer boundary. It deliberately includes covered, underpredicted, and overpredicted requests.

It does not reproduce the paper’s predictor accuracy, vLLM prototype, emulated network, OPT models, TTFT/JCT, or performance-per-dollar results. Those remain source-reported. The paper itself notes that its evaluated transfer path used sockets and an emulated bandwidth mechanism for configurations unavailable on the testbed; that qualification belongs beside any evaluation claim.

Mooncake: KVCache-Centric Placement

Mooncake separates three pools:

prefill compute pool
decode compute pool
distributed KVCache pool over CPU DRAM / SSD / network resources

The paper’s architecture makes that third object concrete. Prefill and decode instances have local paged caches, but both participate in a distributed KVCache pool connected by a transfer engine and controlled by cache-aware, balance, and decode-load schedulers:

Source: Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, Figure 2. The figure’s TTFT, TBT, MFU, DRAM, and VRAM constraints are Mooncake’s system objectives. It demonstrates that disaggregation is a cache-placement problem as well as a compute-pool split.

Read the three horizontal regions as different ownership domains:

prefill pool:
    compute prompt state and maximize reusable-cache hits under TTFT

Mooncake Store:
    retain and route cache blocks across CPU DRAM, SSD, and network paths

decode pool:
    keep active generation state in VRAM and protect token cadence

A26’s source-local copy exposes the same three schedulers and resource pools:

0x09G Prefix Identity Is A Hash Chain, Not A Flat Block Hash

For token block $x_j$, model/cache identity $I$, and previous prefix hash $h_{j-1}$, a teaching key is:

$$h_j=H(I\Vert h_{j-1}\Vert x_j), \qquad h_{-1}=H(\text{root}).$$

Including $h_{j-1}$ means an identical token block under a different preceding prefix does not collide semantically. Including $I$ prevents a match across an incompatible model, RoPE, dtype, head layout, or cache ABI. M5.3 remains the full exact-identity owner; Mooncake consumes block IDs to make cluster placement decisions.

0x09H The Four-Step Request Workflow Has Two Publication Boundaries

Conductor preselects a prefill group and decode instance, then runs:

1. reuse:
    load matched prefix blocks from remote CPU memory into prefill GPU

2. incremental prefill:
    compute uncached blocks; stream newly produced K/V back to CPU memory

3. transfer:
    Messenger streams full request K/V toward decode-node CPU memory

4. decode:
    local scheduler rechecks TBT feasibility; load K/V asynchronously to GPU

The prefix used by prefill must be published before step 1. The complete request cache must be visible and validated at the decode node before step 4 admits the request. Layer-wise overlap changes when bytes move; it does not weaken either publication boundary.

Its global scheduler reasons about both queue time and cache location. A worker with the longest prefix hit is not always best if it has a long queue or the cache transfer is slower than local recomputation.

A conceptual route comparison is:

local recompute cost:
    queue_local + prefill(uncached_tokens_local)

remote reuse cost:
    queue_remote + transfer(prefix_KV) + prefill(uncached_tokens_remote)

choose a feasible path that satisfies TTFT
and leaves a feasible decode placement for TPOT

Mooncake also motivates early rejection: if no likely decode capacity exists after prefill, rejecting before spending prompt compute can improve goodput under overload.

0x09I Cache-Aware Routing Prices Queue, Recompute, And Transfer

For request length $S$, candidate-local prefix $P_i$, best remote prefix $P^*$, queue estimate $Q_i$, per-block prefill time $c_p$, and transfer time $c_x$, the two candidate paths are:

$$C_{local}(i)=Q_i+(S-P_i)c_p,$$ $$C_{remote}(i)=Q_i+(P^*-P_i)c_x+(S-P^*)c_p.$$

Mooncake uses a cache-balancing threshold to decide when the extra remote prefix is large enough to price the second path. It then selects the candidate with the lowest predicted TTFT, separately selects a decode instance by predicted TBT, and rejects when either SLO gate fails.

In the Loop 9 fixture, an eight-block request has an eight-block best remote prefix:

The longest-prefix owner P2 loses to P1 because queue time dominates. Decode D0 predicts 80 ms TBT under a 100 ms gate and is admitted. An overload snapshot whose best decode prediction is 120 ms is rejected before prompt work. This is synthetic routing arithmetic, not Mooncake’s predictor or performance.

Hot-block replication is a related but slower control loop. Replication can reduce sender congestion and create more feasible prefill choices, but it consumes DRAM/SSD capacity and requires generation-aware invalidation. M5.5 owns general tiering and eviction; M5.6 owns how location and transfer enter the cross-worker route.

0x09J Executable Distributed-Memory Closure

python3 docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/distributed_kv_memory_oracle.py \
  docs/numinens-evidence/m5-6-prefill-decode-disaggregation-kv-placement/distributed_kv_memory_fixture.json

The oracle derives the prefix hash chain, validates allocate-to-publish ordering and failure atomicity, checks MemServe’s four cache milestones, reconciles the network-call geometry, derives Mooncake-style prefill costs, and checks early decode rejection. It does not implement either system, reproduce their cache policies, or measure their JCT, TTFT, TBT, bandwidth, or goodput.

Its end-to-end breakdown shows why cache locality must be evaluated together with compute. With no prefix-cache hits, prefill dominates increasingly long prompts in the reported experiment. At a 95% prefix-cache ratio, prefill is smaller and decode becomes a much larger share, while schedule, transfer, and cache-load work remain visible rather than disappearing.

Source: Qin et al., Mooncake, Figure 14. The bars include scheduling, layer-wise prefill, transfer, cache loading, and decoding 128 tokens for the paper’s 0% and 95% prefix-cache-ratio scenarios. The comparison demonstrates one deployed system’s phase shift; it does not predict latency for a different hit distribution, model, output length, or network.

0x0E Variant Comparison Matrix

No row dominates every workload. The design choice depends on:

prompt/output length distribution
prefix reuse rate
arrival burstiness
interconnect topology
GPU compute-to-memory ratios
cache-tier capacity
SLO percentile targets
failure and retry policy

0x0F Runtime Integration Trace

A complete disaggregated request needs more states than prefill -> transfer -> decode:

def serve_disaggregated(req):
    route = global_scheduler.plan(
        prompt_tokens=req.prompt_len,
        expected_output=estimate_output(req),
        prefix_locations=lookup_prefix(req.cache_key),
        prefill_queues=prefill_load(),
        decode_capacity=decode_load(),
        topology=network_state(),
    )

    reservation = reserve_decode_trajectory(route.decode, req)
    if not reservation:
        return reject_before_prefill(req)

    kv_id = run_prefill_with_optional_prefix(req, route.prefill)
    transfer = publish_or_stream_kv(kv_id, route)

    wait_until_kv_complete_and_visible(transfer)
    enqueue_decode(req, route.decode, kv_id, reservation)

Telemetry must connect the phases with one request ID:

prefix hit bytes and source tier
prefill queue/compute time
K/V bytes and transfer overlap
decode reservation wait
first decode enqueue time
TTFT and TPOT percentiles
reroutes, retries, cache misses, and early rejections

Without cross-phase tracing, a TPOT regression can be blamed on decode compute when the real cause is delayed K/V visibility or a poor route.

Executable Transfer-Trace Contract

The repository includes a versioned trace analyzer and a deliberately synthetic fixture:

docs/numinens-evidence/m4-m5-p1/transfer_trace.py
docs/numinens-evidence/m4-m5-p1/transfer_trace_fixture.json

The analyzer requires one transaction identity and generation across:

registration
per-layer gather
network attempts
destination device copy
per-layer visibility
atomic publication

It rejects a transaction unless every expected layer has a successful visibility event before the single successful publication event. It reports phase work, wall time, overlap, retry count, and the stall exposed after the decode reservation becomes ready:

$$T_{exposed} = \max(0,t_{publish\ end}-t_{decode\ ready}).$$

The checked-in 8 MiB fixture contains one failed network attempt, overlapping gather/copy work, and two layer-visibility events. The analyzer reports 220 microseconds of synthetic wall time, 74 microseconds of overlapped work, and 60 microseconds of exposed stall. These numbers test state and unit handling; the fixture labels its link not measured and makes no topology-performance claim.

python3 docs/numinens-evidence/m4-m5-p1/transfer_trace.py \
  docs/numinens-evidence/m4-m5-p1/transfer_trace_fixture.json

A target-topology trace must replace the fixture with timestamps from the actual registration, gather, transport, visibility, retry, and publication path. The official LMCache repository HEAD observed for this pass is recorded in source_pins.json, but an observational source pin is not a runtime trace.

0x0FA A Runtime Claim Needs Four Labels

Runtime interfaces change faster than paper contracts. A connector name without a release, commit, source path, direction, and compatibility matrix is not a reproducible implementation statement. This pass uses four evidence labels:

The runtime surface audited on 2026-08-03 is frozen below. The versions are not recommendations; they are coordinates that keep the prose auditable.

A release tag and commit do not prove that a cited path exists. The Loop 11 manifest therefore records the Git blob identity of all seventeen audited paths. This caught a real audit defect: at LMCache v0.5.2, the 1P1D proxy is examples/disagg_prefill/disagg_proxy_server.py, not the nonexistent examples/disagg_prefill/1p1d/disagg_proxy_server.py. The prose may cite a source path only when repository, commit, path, and blob identity reconcile.

At that vLLM pin, the official page labels disaggregated prefill experimental, runs prefill and decode in separate instances, and says exactly: “Disaggregated prefill DOES NOT improve throughput.” The durable rationale is independent TTFT/ITL tuning and tail-ITL control. Whether total goodput rises is a workload/topology result governed by the earlier ledgers, not a property of enabling a connector.

0x0FB Scheduler Connectors Plan; Worker Connectors Move

The pinned vLLM design has one connector in the scheduler process and one in each worker process. Scheduler output carries connector metadata; worker connectors interact with paged K/V buffers and the data-transfer layer.

This is an official figure from the pinned vLLM source, not a local architecture diagram. It matters because one logical handoff spans two process roles:

scheduler side:
    identify externally available tokens
    reserve destination blocks
    build per-step connector metadata
    reconcile finished receive/send requests and load errors

worker side:
    register K/V cache memory
    execute transfer operations against physical blocks
    establish stream/device completion
    return completion or load-error state

The NIXL facade at this pin exposes separate pull/read and push/write connectors; the backward-compatible NixlConnector name aliases pull. Direction is therefore part of the deployment manifest:

connector class
push or pull mode
producer and consumer roles
engine IDs and PP/TP mapping
K/V layout and buffer device
NIXL backend list
side-channel endpoint and authentication
model architecture and attention backend
K/V dtype, prefix-caching mode, and speculative-decoding mode
topology identity and route epoch

Copying a configuration across releases without this manifest can silently change role defaults, deprecated aliases, layout preferences, or error recovery. The compatibility test must target the exact model architecture, attention backend, cache layout, prefix-caching mode, parallelism, and speculative-decoding features used by the release.

0x0FC LMCache, NIXL, And Mooncake Do Not Collapse Into One Layer

The names often appear together but own different objects:

At the LMCache pin, PDBackend has distinct allocation, cache-query, and proxy notification messages. The sender writes into receiver allocations; receiver objects live in the destination store. A dedicated thread and lock serialize NIXL-agent GPU operations in that source revision. Those are pinned-source observations, not promises that every later LMCache release keeps the same threading or side-channel design.

LMCache documentation also separates two configurations that are easy to conflate:

P/D mode:
    explicit sender/receiver role
    P/D buffer size and device
    transfer channel such as NIXL
    proxy/notification sequencing

NIXL storage-backend mode:
    NIXL is a storage provider selected through extra configuration
    backend can target storage-oriented plugins such as POSIX or GDS

Enabling one is not evidence that the other’s completion and failure contracts apply. Similarly, a vLLM Mooncake connector, Mooncake Transfer Engine, and Mooncake Store must be versioned independently when they cross package release boundaries.

0x0FD Sensitivity Is A Net-Margin Surface

Disaggregation is useful only when phase-interference savings exceed the new costs it exposes. A bounded decision coordinate is:

$$\Delta T =T_{interference\ saved} -T_{transfer\ exposed} -T_{route\ delta} -T_{failure\ penalty}.$$

T_transfer exposed is the non-overlapped portion, not total logical transfer time. T_failure penalty includes retry, recompute, reroute, and cache loss weighted by their observed frequency. Enable P/D for a workload/topology cell only when Delta T exceeds a declared safety margin and TTFT and TPOT attainment gates both pass.

Synthetic examples from runtime_release_fixture.json illustrate the shape:

These numbers are arithmetic tests, not measurements. A real sensitivity grid must vary prompt/output distributions, arrival burstiness, cache hit source, K/V payload architecture, connector direction, buffer device, transport backend, link contention, TP/PP mapping, transfer size, failure rate, and both SLOs.

0x0FE Remote-Memory Metadata Is A Capability

NIXL-style registered-memory metadata contains connection coordinates and keys needed to access specific memory regions. Official documentation warns that it is sensitive. Treating it as harmless scheduler metadata turns a performance interface into an authorization bypass.

A bounded transfer capability should bind:

tenant and workload identity
source and destination worker identities
transfer_id and attempt generation
route epoch
allowed memory regions, direction, and byte bounds
model/cache ABI digest
expiry and nonce
authenticated control-plane issuer

Region Scope And Cache Identity Are Independent Gates

The request names the exact destination regions and byte count it needs. The capability names a superset it permits. Authorization requires set inclusion, not merely a valid peer signature:

$$\mathcal{R}_{request}\subseteq\mathcal{R}_{capability}, \qquad 0 < B_{request}\le B_{capability}.$$

The cache ABI and route generation are separate equalities. A capability that permits the right bytes for the wrong layout is still invalid. A capability from an old route epoch is invalid even when its memory region has not yet been reused. The nonce must be consumed atomically by the control plane; the local oracle checks nonce binding but does not claim to implement a distributed replay store.

def authorize_transfer(request, capability, now):
    assert verify_signature(capability)
    assert capability.issuer in TRUSTED_CONTROL_PLANES
    assert 0 < capability.expires_at - now <= MAX_CAPABILITY_TTL
    assert capability.tenant == request.tenant
    assert capability.source == request.prefill_owner
    assert capability.destination == request.decode_owner
    assert capability.transfer_id == request.transfer_id
    assert capability.generation == request.generation
    assert capability.route_epoch == current_route_epoch()
    assert capability.cache_abi == request.cache_abi
    assert capability.direction == request.direction
    assert set(request.regions) <= set(capability.allowed_regions)
    assert 0 < request.bytes <= capability.max_bytes
    assert capability.nonce == request.nonce
    assert consume_nonce_once(capability.nonce)
    return ALLOW_EXACT_REGIONS_ONLY

Transport authentication and exact K/V identity are separate gates. A valid network peer can still present the wrong model, token, generation, layout, or tenant cache. Conversely, correct cache metadata does not authorize direct memory access. Deny before registration or remote operation creation whenever either gate fails; do not rely on a later checksum to repair an access-control error.

Cross-tenant prefix indexes and remote capabilities must share the same isolation domain. Logs should hash or redact access keys and raw prompt-derived identities. Revocation advances route/capability epochs and removes directory reachability; it cannot assume already issued remote-memory keys disappear by themselves.

0x0FF Release Is A State Machine

The deployment manifest freezes source pins plus model, tokenizer, adapter, cache ABI, connector class/direction, producer/consumer roles, transfer backends, buffer device, topology, route epoch, and fallback mode.

When P/D is a feature of an M8.10 quantized release tuple, this manifest is scoped child evidence, not whole-tuple promotion authority. Freeze its static identity and the allowed P/D/colocated modes and transitions in the parent configuration $C$ before evaluation. The later operational receipt binds the parent tuple and this child manifest; do not create circular hashes by putting that later receipt back into the already-frozen parent. A preregistered, evaluated same-target colocated fallback can remain within $C$. An unregistered mode, changed target/artifact/representation or changed switching rule requires a new tuple and its gates. M5.3 still decides K/V compatibility and lifetime; feature disable does not authorize cross-target cache reuse.

Promotion then follows:

PINNED
  -> verify dependency and source-path hashes
  -> deterministic ABI/exactness smoke tests
  -> failure injection: mid-transfer, stale completion, decode crash, bad capability
SHADOW
  -> score routes and record predicted transfers, but execute colocated baseline
CANARY
  -> bound tenant, workload, topology, and traffic share
  -> compare TTFT, TPOT, goodput, exactness, transfer failure, and fallback
ACTIVE
  -> keep the same gates online; do not treat promotion as permanent

Minimum promotion gates include a verified deployment manifest, a passed failure-injection suite, healthy colocated fallback, a completed rollback drill, zero exactness mismatches, zero stale publications, zero unauthorized accepted transfers, enough canary samples, separate TTFT and TPOT attainment, bounded transfer failure, and a positive net-margin envelope. The canary names the workload/topology cell whose sensitivity margin is being promoted; a good high-bandwidth cell cannot authorize the low-bandwidth cell from the table above. Mean latency alone is insufficient.

Observability needs connector and lifecycle state, not only request latency:

pd_route_total by policy/topology/result
kv_transfer_bytes and exposed transfer time
transfer attempts, completion, timeout, retry, and failure
stale or duplicate publication rejects
capability denials by reason without leaking keys
TTFT and TPOT SLO attainment by workload cell
fallback/recompute/reroute counts
drain depth and rollback duration

0x0FG Stop Routes Before Disabling The Connector

Rollback ordering protects both correctness and availability:

STOP_NEW_ROUTES
ADVANCE_ROUTE_EPOCH
DRAIN_OR_ABORT_TRANSFERS
ENABLE_COLOCATED_FALLBACK
DISABLE_CONNECTOR
REVOKE_CAPABILITIES
RECLAIM_UNPUBLISHED
ROLLBACK_COMPLETE

Stopping the connector first can strand requests whose prompt owner has already committed to a remote decode route. Enabling fallback after disabling transfer creates an avoidable availability gap. Freeing destination buffers before an in-flight DMA/RDMA operation is canceled or completed creates memory corruption.

During drain, published complete caches may finish decode under their original generation. Unpublished transfers either complete under the old epoch and are explicitly rejected, or abort and release staging only after transport quiescence. New work uses colocated prefill/decode or recomputes prompt state. Rollback must preserve request status and output-stream semantics even when it sacrifices efficiency.

The rollback gate is complete only when:

no route can select the disabled connector
no in-flight transfer can publish under the retired epoch
all registered temporary regions are revoked or quarantined
all unpublished allocations are reclaimed
fallback health and both SLO clocks are visible
the incident trace preserves source pins and deployment manifest

0x0FH Executable Runtime-Release Closure

runtime_release_oracle.py validates four immutable source pins, seventeen source paths and their seventeen Git blob identities, eleven claim/evidence records, connector direction and roles, the runtime compatibility matrix, signed region- and byte-bounded capability identity, the promoted sensitivity cell, release gates, failure outcomes, observability names, and exact rollback ordering. Its fixture reaches ROLLED_BACK deliberately so drain, fallback, disable, revocation, and cleanup are tested rather than asserted.

The oracle does not contact or benchmark vLLM, LMCache, NIXL, Mooncake, a GPU, or a network. A production release must replace synthetic margins and rates with target-topology traces while retaining the same immutable manifests and fail- closed state transitions.

0x10 Exactness And Failure Contracts

P/D disaggregation should not change model output if the decode side receives the same K/V state and interprets it under the same execution contract. First separate strict state identity from weaker numerical compatibility.

An exact or lossless handoff requires:

same model/checkpoint
same token-ID sequence and prompt serialization
same adapter or LoRA state
same position IDs and RoPE phase
same attention-mask semantics
same numerical representation, or a lossless transport encoding
same cache layout interpretation
same tensor/pipeline partition mapping or correct remapping
same layer ordering
same token ordering
complete K/V visibility before decode

These conditions preserve the logical K/V values. Bitwise identity additionally requires that transport and remapping preserve the exact stored bits.

The following can be operationally acceptable, but should not be called exact without qualification:

fp32 -> bf16/fp16 casts
KV quantization and dequantization
different arithmetic order or backend reconstruction
lossy compression during transfer

They change numerical values even when shapes and metadata match. A system may adopt them after measuring logit drift, quality, and SLO benefit, but the contract is then “validated approximate transfer,” not exact K/V handoff. Sampling configuration is a separate continuation contract: it can change the generated sequence without changing transferred K/V.

If an exact-handoff field is wrong, it is not a small performance issue. It is a correctness bug. If an approximate transport is intentional, it needs an explicit quality contract and cannot inherit exact-reuse claims.

Versioned Transfer Transaction

The identity list becomes operational through a transfer transaction. The destination must not publish K/V to decode merely because metadata arrived or the final network packet was submitted.

@dataclass(frozen=True)
class KVTransferKey:
    request_id: str
    prompt_digest: str
    model_hash: str
    adapter_hash: str | None
    position_policy_hash: str
    cache_schema_hash: str
    source_partition: str
    destination_partition: str
    token_start: int
    token_count: int
    generation: int

@dataclass
class KVTransferTxn:
    transfer_id: str
    key: KVTransferKey
    state: Literal[
        "RESERVED", "TRANSFERRING", "VALIDATED", "PUBLISHED", "ABORTED"
    ]
    received_layers: set[int]
    destination_handles: dict[int, PendingLayerKV]

The safe order is:

reserve destination under transfer_id and generation
transfer layer/block payloads into unpublished storage
validate identity, lengths, layer coverage, layout, and optional checksums
establish device/stream visibility for destination buffers
atomically publish the complete cache handle to the decode directory
acknowledge source release only after publication policy permits it

def publish_transfer(txn, decode_directory, expected_layers):
    assert txn.state == "VALIDATED"
    assert txn.received_layers == set(expected_layers)
    assert all(x.k_complete and x.v_complete
               for x in txn.destination_handles.values())
    assert all(x.device_visible for x in txn.destination_handles.values())

    with decode_directory.transaction(txn.key.request_id):
        current = decode_directory.lookup(txn.key.request_id)

        if current is not None and current.transfer_id == txn.transfer_id:
            release_pending(txn.destination_handles)
            return READY_FOR_DECODE(current.handle)

        if current is not None and current.generation >= txn.key.generation:
            txn.state = "ABORTED"   # duplicate or stale completion
            release_pending(txn.destination_handles)
            return STALE_OR_DUPLICATE

        decode_directory.publish_complete(
            key=txn.key,
            layer_handles=make_visible(txn.destination_handles),
        )
        txn.state = "PUBLISHED"
        return READY_FOR_DECODE

Retries are idempotent by transfer_id; superseding attempts use a newer generation. A decode worker accepts only PUBLISHED state. This prevents a late completion from an abandoned route from replacing a newer reroute, and it prevents partially received layers from becoming a shape-correct mixed cache.

Failure cases:

Mooncake’s early rejection logic highlights a related point: rejecting after expensive prefill wastes work. A disaggregated system should admit only if both prefill and later decode feasibility are plausible.

Conceptual decode-start fallback:

def start_decode(req, decode_worker):
    if decode_worker.kv_visible(req.kv_id):
        decode_worker.enqueue(req)
        return

    if can_wait_within_ttft(req):
        wait_for_kv(req.kv_id)
    elif can_reroute_to_visible_kv(req):
        reroute(req)
    elif can_recompute_prefill_locally(req):
        recompute_on_decode_worker(req)
    else:
        reject_or_fail(req, reason="kv_not_visible")

0x11 Boundary To M5.7 And M5.8

M5.6 moves K/V between places. M5.7 and M5.8 change what the model stores per token.

M5.6:
    where does K/V move?
    when is transfer worth it?
    which worker owns decode?

M5.7:
    how many full K/V heads exist per token?

M5.8:
    does the architecture use latent cache state and absorbed kernels?

The interaction is simple:

smaller K/V state makes transfer easier,
but it does not remove placement, routing, or exactness contracts

A GQA model still needs a placement decision. An MLA model still needs a decode worker to interpret the cache correctly. Architecture-level cache reduction helps M5.6’s byte ledger, but it does not replace M5.6.

0x12 Publish Gate

References

Local source articles:

• M5.1 - Autoregressive Serving State: KV Cache Lifecycle and Scheduler Pressure
• M5.2 - Continuous Batching and Scheduler Pressure
• M5.3 - KV Memory Layout and Exact Reuse
• M5.4 - Long-Context KV Policies
• M5.5 - KV Retrieval, Offload, and Hierarchical Memory
• M5.7 - MQA and GQA
• M5.8 - MLA Latent Cache and Absorbed Kernels
• Exploring the Transformer Series (20) - KV Cache
• Exploring the Transformer Series (24) - KV Cache Optimization
• Exploring the Transformer Series (25) - KV Cache Optimization for Handling Long Text Sequences
• Exploring the Transformer Series (26) - KV Cache Optimization: P/D Separation or Merging

Primary and high-quality sources:

• Orca: A Distributed Serving System for Transformer-Based Generative Models
• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
• KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• Splitwise: Efficient generative LLM inference using phase splitting
• MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
• Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• vLLM v0.26 Serve CLI: chunked-prefill and token-budget controls
• vLLM v0.26.0: Disaggregated Prefilling
• vLLM v0.26.0: pinned NIXL connector facade
• LMCache v0.5.2: pinned P/D backend
• LMCache configuration: P/D and NIXL storage boundaries
• NIXL v1.3.2 source
• NIXL Connect API: registered memory, metadata, and paired operations
• NIXL remote-memory metadata security warning
• Mooncake v0.3.12.post1 source
• TetriInfer: Inference without Interference