0x00 Overview And Background

M3.9 followed the training path from batches to loss. Training has a rectangular target tensor, so the code can build:

batch.tgt      # decoder input IDs
batch.tgt_y    # next-token labels
batch.tgt_mask # pad + causal visibility
batch.ntokens  # non-pad supervised positions

Inference removes the most important object in that list: there is no tgt_y. The model does not receive the answer sequence and then compare many predicted rows against known labels. It receives the current context, predicts one token, appends that token, and then uses the enlarged prefix as the next input.

The local contrast is:

training:
    source/context + shifted gold target
        -> all visible target positions
            -> many next-token losses

inference:
    source/context + generated prefix
        -> last valid prefix position
            -> one next-token choice
                -> append and repeat

The target-side object is now a runtime prefix. In the Harvard encoder-decoder code, that prefix is called ys:

ys = torch.zeros(1, 1).type_as(src)

For a translation example:

source:
    我吃了一个苹果

target prefix at step 0:
    <bos>

after one selected token:
    <bos> I

after two selected tokens:
    <bos> I ate

The prefix grows because each selected token becomes part of the next decoder input. That is the tensor-level reason inference is sequential.

Background knowledge: generation turns a model call into a stateful process

A causal language model defines a next-token distribution, but generation is the repeated procedure that consumes that distribution. Earlier recurrent models carried hidden state between steps. Transformer generation carries an expanding logical prefix and, in optimized runtimes, cached K/V rows for every layer.

model contract:
    current context -> next-token scores

policy contract:
    scores -> selected token ID

loop contract:
    accept one authenticated decision, append logical output, update request-local
    state, and expose a later model-input suffix or a terminal result

This separation matters because a correct forward pass can still produce incorrect generation when the loop gathers the wrong row, advances finished sequences, misaligns positions, or commits cache entries that do not belong to the accepted prefix.

From recurrent decoding to a Transformer request

Autoregressive inference predates the Transformer. In a recurrent decoder, one step consumes the previous token and a recurrent state:

$$h_t=f_\theta(h_{t-1},e(y_{t-1}),c_t), \qquad p(y_t\mid y_{<t},x)=\operatorname{softmax}(W_o h_t+b_o).$$

The fixed-width vector $h_{t-1}$ is the explicit state carried across calls. An attention-based encoder-decoder adds source context $c_t$ but keeps the same output feedback: the selected token becomes the next decoder input. A Transformer removes recurrent hidden-state updates inside the decoder, not the autoregressive dependency between output decisions. Its logical state is the entire committed prefix. A naive implementation re-executes that prefix; a cached implementation materializes the per-layer K/V representation needed to avoid recomputing historical rows.

recurrent decoder:
    previous token + recurrent state -> next-token scores -> token

uncached Transformer decoder:
    complete committed prefix -> all prefix rows -> last-row scores -> token

cached Transformer decoder:
    newest committed input + retained per-layer K/V
        -> newest hidden row + next-token scores -> token

The execution history matters because several similar phrases name different contracts:

Phrase	Actual object	M3.10 consequence
autoregressive factorization	probability identity over ordered outputs	later tokens condition on earlier selected tokens
teacher-forced execution	all gold prefixes materialized in one masked tensor	owned by M3.9, not the inference schedule
free-running execution	generated tokens become future model inputs	one decision changes every later prefix
K/V caching	retained per-layer projections of processed input rows	removes historical re-execution but introduces request-local state; M5.1 owns its persistent schema and publication lifecycle
generation policy	transform/select operation over one score row	owned mathematically by M3.8 and consumed here
serving scheduler	policy for interleaving many live requests	owned by M5, outside this one-request transaction

The Transformer paper establishes masked autoregressive decoding and the parallel-training contrast. A4 and A5 preserve the handwritten encoder-decoder feedback trace. A15 contributes the generator and search interfaces. This article reconstructs those pieces as one explicit request state machine and checks them against current runtime source rather than treating any one pedagogical loop as a complete production API.

Source continuity and corrections

The local articles are evidence for what the original series taught, but some wording needs a narrower execution contract:

Source-local statement	M3.10 correction
inference no longer needs a decoder mask	a one-token cached call may not materialize a triangular mask, but visibility still has to exclude future/current-chunk rows; multi-token prefill and chunked calls require an explicit or kernel-implied causal contract
the decoder feeds its output back as the next input	it feeds back a selected token ID; a hidden vector, probability row, and token ID are different objects
K/V cache makes inference fast	it saves historical row execution and projections; each decode query still reads retained K/V, and the state consumes memory
beam search keeps the most likely sequence	finite-width beam search keeps the best frontier under a declared score; it does not guarantee the globally most probable completed sequence
model.eval() is inference	eval() changes module behavior such as dropout; autograd mode, precision, cache policy, generation policy, and stop/output contracts remain separate

These are not stylistic qualifications. Each correction maps to a testable failure later in the article.

Claim status and evidence classes

Status	Meaning here
Source-local	preserved from A4, A5, A15, or another named local article, with its limits stated
Externally verified	checked against a primary paper or an official, pinned implementation/documentation source
Rewrite interpretation	a synthesis, naming convention, or owner boundary introduced by Transformer System
Synthetic contract test	a deterministic local oracle over shapes, counters, or state transitions; not a hardware benchmark
Unmeasured	an analytical performance consequence for which no local latency or throughput result is claimed

Owner question

Given one prompt and one declared decoding policy, what state exists before and after every model call, when does each token acquire K/V state, and how do positions, stopping, streaming, beams, and cache mutation remain aligned?

M3.10 owns the logical requirements for one single-request state machine:

prompt preparation -> prefill -> first decision
selected token -> logical append -> tagged pending input -> later model processing -> next decision
position/mask/cache alignment
EOS, limits, strings, structured completion, and streaming
beam parent reorder for one request
cache/no-cache logical equivalence and recovery requirements

M3.8 owns the authenticated local score-to-decision policy. M3.10 consumes one such decision exactly once into one request’s logical transaction. M5.1 owns persistent request/cache schema and publication; M5.2 owns multi-request admission and scheduling; M5.3 owns pages, allocation, reference counts, and copy-on-write. M7 changes how candidates are proposed and verified, but must still return a decision that can enter this state model.

The boundary is precise enough to audit:

Surface	M3.10 relationship	Authoritative owner
tokenization, normalization, chat-template serialization, vocabulary IDs, special-token identity	consumes immutable ordered input artifacts	M2.1
admitted initial model-width state and embedding/output-tie eligibility	consumes the checkpoint-declared start-state interface	M2.2
hidden-to-vocabulary projection, processor/warper mathematics, local sampling distributions	consumes one decision-row policy	M3.8
one request’s logical prefill/decode phases, token/output/processed frontiers, stop state, stream horizon, beam-state application, and recovery requirements	owns requirements	M3.10
per-call FLOPs, traffic, kernel materialization, and measured roofline behavior	declares call shapes but defers physical accounting	M4
persistent cache objects/publication plus multi-request admission, scheduling, paging, prefix sharing, offload, placement, and service clocks	exposes logical requirements but defers ownership	M5.1–M5.8
expert balance, placement, dispatch, topology, precision, communication, and expert runtime	exposes only one-call completion requirements	M6.2/M6.6
speculative candidates, target verification, acceptance, and tree state	supplies commit invariants but defers the algorithm	M7
live cache quantization	records a representation socket but defers design and quality gates	M8.6
artifact conversion, loader, kernel, and runtime admission	exports compatibility requirements only	M8.9
deployment, evaluation, canary, monitoring, rollback, and release	exports requirements only	M8.10
non-text unitization, encoder/projector/fusion, and mixed-origin runtime state	consumes a declared interface; does not construct it	X1.2/X1.3

This means M3.10 may derive how one beam reorder couples token rows and cache rows, but not how a serving scheduler batches beams from different requests. It may compare a dynamic and static cache API, but not re-derive paging or offload policy. It may define proposal/commit separation, but not reproduce speculative acceptance mathematics.

Immutable request envelope and canonical identity

An input string and a random seed do not identify a generation run. Before the first model call, freeze the objects that determine token identity, visibility, score processing, mutable-state interpretation, and returned output:

Plane	Immutable fields before execution	Why it changes the result or its meaning
candidate identity	checkpoint/source hash, model config, adapter set, weight dtype/layout, M8.9 admission requirement	identifies the intended function/profile; it does not prove a runnable physical artifact exists
text interface	tokenizer and vocabulary hash, normalization, chat template, BOS/EOS/PAD/decoder-start IDs	determines the exact input IDs and stop-token identities
prompt	serialized text/bytes, token IDs, valid-token mask, segment/modal metadata, declared prompt length	defines the initial visible sequence
position/visibility	position-ID rule, cache-position rule, causal/prefix/cross-attention mode, maximum context	defines which coordinates and edges exist
local policy	ordered M3.8 processors/warpers, sampling/search family, beam width, length score, constraints	turns one or more score rows into branch decisions
randomness	RNG algorithm/device, seed/key, initial counter/state, beam-stream rule	makes stochastic draws replayable under the same runtime
cache/runtime	cache family and ABI, compute/logit/cache dtypes, framework and code revision, device/model-shard topology, compile/kernel mode	defines how logical state is represented and which numerical tolerance is plausible
termination	minimum/maximum new tokens, total-length cap, EOS set, stop byte/text patterns, grammar completion, timeout/cancellation policy	defines when selection stops
output	prompt echo, special-token inclusion, stop-string inclusion, token/logprob fields, streaming holdback, final-cache promise	defines what complete means to the caller

The request identity can be represented by a canonical serialization and hash, but the word canonical is earned only by a closed byte-level contract. The envelope must fix at least:

schema name + version and allowed migration path
UTF-8 validity and Unicode normalization policy
closed field set, required/optional fields, and unknown-field rejection
duplicate-key rejection before host-language object construction
exact scalar grammar, integer ranges, finite-float encoding, and null/absent rules
map-key ordering plus whether every list is ordered, a set, or a multiset
maximum bytes, tokens, list lengths, nesting depth, patterns, beams, and output
canonical byte encoding, digest algorithm/domain tag, and digest envelope

If a parser has already discarded duplicate keys or coerced a large integer to a floating value, later key sorting cannot reconstruct a canonical request. Validate the raw bytes against the closed schema first, serialize the admitted value deterministically second, and hash those bytes third:

$$I_{request} = H(\texttt{"numinens.request.v1"}\Vert \operatorname{CanonicalBytes}(M_{artifact},M_{text},M_{prompt},M_{position}, M_{policy},M_{rng},M_{runtime},M_{stop},M_{output})).$$

This identity is deliberately wider than the prompt. The same token IDs under a different RoPE-scaling rule, cache ABI, processor order, EOS set, or chat template are not the same request contract. Conversely, wall-clock timestamps and allocator addresses do not belong in this semantic hash; they are runtime observations.

The following object is an illustrative semantic payload, not by itself a canonical wire format or an executable request:

{
  "schema": "numinens.request.v1",
  "artifact": {
    "release_id": "model-r17",
    "checkpoint_sha256": "...",
    "architecture": "decoder_only"
  },
  "text_interface": {
    "tokenizer_sha256": "...",
    "template_sha256": "...",
    "bos_id": 1,
    "eos_ids": [2],
    "pad_id": 0
  },
  "prompt": {
    "input_ids": [1, 314, 1592],
    "attention_mask": [1, 1, 1]
  },
  "position_visibility": {
    "mode": "causal",
    "position_rule": "valid_token_cumsum",
    "cache_position_start": 0,
    "max_context": 4096
  },
  "policy": {
    "family": "multinomial",
    "ordered_transforms": ["repetition", "temperature", "top_p"],
    "temperature": 0.8,
    "top_p": 0.95
  },
  "rng": {"algorithm": "philox", "seed": 42, "counter": 0},
  "runtime": {
    "framework_revision": "pinned-revision",
    "compute_dtype": "bfloat16",
    "logit_dtype": "float32",
    "cache_family": "dynamic",
    "cache_abi": "per_layer_compact_kv"
  },
  "termination": {"max_new_tokens": 64, "eos_ids": [2]},
  "output": {
    "return_token_ids": true,
    "return_logprobs": true,
    "final_cache": "processed_inputs_only"
  }
}

No-image fallback: validate and canonically bind schema/version; artifact; M2.1 text interface and prompt IDs; M2.2 start-state interface; positions/visibility; M3.8 policy; RNG; runtime/state profile; termination; and output promise. Their digest identifies the request. Separate mutable state then tracks output, pending input, processed model-state lanes, branch/constraint/RNG/decoder/stream state, and finish status until a typed terminal result.

Immutable identity versus mutable request state

The envelope defines the transition system. It is not the transition state. After execution begins, these objects mutate:

committed token sequence and generated-token count
next model-input slice
per-layer logical state frontier plus any implementation cache handle/generation
current position/cache coordinates
beam hypotheses and parent map
constraint/parser state
RNG draw counter or generator state
incremental decoder and stop-matcher suffix
streamed/withheld output frontier
finish reason and cancellation state

A resume snapshot must bind the immutable identity and enough mutable state to continue exactly, including the complete model-state-lane roster declared by the architecture profile. Restoring token IDs while resetting RNG, parser, stream suffix, branch ancestry, finish state, or positions is a different continuation. Changing any identity field requires a new request identity; it must not be disguised as progress on the old request.

The validation order should fail before model execution:

def validate_request_envelope(raw_bytes):
    require_utf8_size_and_nesting_bounds(raw_bytes)
    reject_duplicate_keys_unknown_fields_and_invalid_scalars(raw_bytes)
    m = parse_closed_schema(raw_bytes, schema="numinens.request.v1")
    require_hashes(m.artifact, m.text_interface, m.runtime)
    require_equal_length(m.prompt.input_ids, m.prompt.attention_mask)
    require_declared_token_pattern_beam_and_output_bounds(m)
    require_nonempty_or_start_state(m.prompt, m.text_interface)
    require_ids_in_vocabulary(m.prompt, m.text_interface)
    require_special_id_agreement(m.text_interface, m.termination)
    require_policy_fields(m.policy)
    require_cache_abi_compatible(m.artifact, m.position_visibility, m.runtime)
    require_output_contract_implementable(m.output, m.runtime)
    canonical = serialize_canonically(m)
    return m, domain_separated_sha256(canonical)

Sensitive prompts, decoded text, candidate lists, and traces need a separate classification/redaction/access/retention/deletion policy. A semantic request digest proves identity; it is not permission to retain or expose its payload.

The last check matters. A runtime that returns cache state only for processed inputs cannot truthfully promise that the final selected-but-unprocessed token already has committed K/V.

Authoritative phase and commit spine

Keep four events separate even when one implementation fuses their code:

1. MODEL-CALL COMPLETION
   all declared compute stages and all advancing model-state lanes complete
   under the same request, row, coordinate, and generation identity

2. M3.8 DECISION
   one authenticated score row produces one pending token/beam decision receipt

3. M3.10 LOGICAL COMMIT
   consume that receipt exactly once; append output, advance policy/RNG/parser/
   stop/stream state, and tag the selected suffix as pending or terminal

4. LATER MODEL PROCESSING
   only a later valid call can process the pending token into hidden/K/V or
   other architecture-qualified state; persistent publication remains M5.1

The state names used below are:

O: committed output-token count
U: committed output suffix not yet processed, tagged EMPTY,
   PENDING_NEXT_CALL, or CLOSED_UNPROCESSED
P: complete logical processed-input frontier
K: conditional cached-baseline logical frontier only when that profile applies
P_dyn and F_lane: generic dynamic frontier and each required state-lane frontier

For the ordinary decoder-only cached profile, $P+|U|=L_{prompt}+O$. A physical buffer length, allocated capacity, durable cache record, reader-visible stream byte, and logical frontier are not aliases.

One request as a transition system

At decision step $t$, define request state

$$S_t=(I_{request},X_t,A_t,P_t,C_t,R_t,F_t,U_t),$$

where:

I_request: immutable request-envelope identity
X_t: committed token IDs / prompt representation
A_t: attention and segment visibility metadata
P_t: position IDs and cache positions
C_t: per-layer cache state
R_t: random/policy/constraint state
F_t: finished and stop-matcher state
U_t: tagged committed output suffix not yet processed by the model

The model and policy produce a token decision

$$z_t=\operatorname{ModelStep}(S_t), \qquad u_t=\operatorname{Policy}(z_t,R_t),$$

and, only after authenticating that receipt and its expected prior-state identity, the controller applies

$$S_{t+1}=\operatorname{CommitOnce}(S_t,u_t,I_{decision}).$$

The split between decision and commit is essential. A token may be proposed or sampled before its own K/V row exists. Cache state is produced when that token is later processed as model input, unless an explicit append/repair forward computes it immediately.

The first-output off-by-one

For prompt tokens $x_0,\ldots,x_{L-1}$, prefill processes all $L$ inputs and returns the next-token distribution after $x_{L-1}$. If policy selects $y_0$:

after prefill:
    committed output text includes y_0 after selection
    cache contains prompt rows x_0 ... x_(L-1)
    cache does not yet contain y_0

next decode call:
    input token is y_0 at position L
    model appends K/V for y_0
    returned logits predict y_1

For the ordinary cached decoder-only profile and only when $G\ge1$, stopping immediately after selecting the last output leaves the complete logical processed frontier at $P=L+G-1$ and the final token tagged CLOSED_UNPROCESSED. It becomes $P=L+G$ only if a separate valid append/repair call processes that token. G=0 is a typed union: the request may reject before state construction, prohibit every call, or perform a prefill-only call and select no output. Blind substitution into $L+G-1$ is invalid. Physical cache length or durable publication requires the selected cache profile and M5.1 contract; it is not proved by this logical equation.

emitted token count != necessarily cached generated-token count

From chain rule to repeated model calls

For a prompt or source object $C$ and output tokens $y_{1:G}$, the chain rule gives

$$p_\theta(y_{1:G}\mid C) = \prod_{t=1}^{G}p_\theta(y_t\mid C,y_{<t}).$$

Taking logs converts the sequence score into an additive trace:

$$\log p_\theta(y_{1:G}\mid C) = \sum_{t=1}^{G}\log p_\theta(y_t\mid C,y_{<t}).$$

The identity alone does not require slow training. M3.9 materializes all gold prefixes in one causal tensor because $y^*_{1:G}$ is already known. In free running inference, $y_t$ is not known until policy consumes the preceding score row. Therefore the input for decision $t+1$ cannot be finalized before decision $t$:

$$y_t = \pi\!\left(z_{t-1},R_{t-1}\right), \qquad z_t = f_\theta(C,y_{\le t};A_t,P_t,C_t).$$

Here $\pi$ is the declared M3.8 policy, while $A_t$, $P_t$, and $C_t$ are visibility, position, and cache state. This is a data dependency, not merely a Python for loop. A compiler can fuse work inside one call; a runtime can batch unrelated requests; M7 can propose multiple candidates. None may make a committed token depend on an output decision that does not yet exist.

No-image fallback: encode the fixed source; start the target with its declared start token; decode the current target prefix; project the semantic last target row to vocabulary scores; select one token; append it logically; then run a later decoder call on the enlarged prefix. Each selection is causally required before the next prefix exists.

The A5 figure is conceptual. Its repeated full-prefix boxes correspond to the uncached schedule. A cached decoder supplies only unprocessed target rows plus retained state, but produces the same logical conditional distributions within the declared numerical tolerance.

Source-to-tensor trace before the first decision

M2.1 owns serialization, tokenizer/vocabulary identity, special tokens, and ordered token IDs. M2.2 owns the admitted initial model-width state and any embedding/output-tie eligibility. M3.10 records those separate exports and traces them to the first score row:

raw conversation / source object
    -> pinned template and normalization
serialized bytes/text
    -> pinned tokenizer and special-token policy
input_ids [1,L]
    -> valid-token and architecture metadata
attention_mask [1,L], position_ids [1,L], cache_position [L]
    -> M2.2 initial states + Transformer blocks under declared runtime contract
last_hidden_state [1,L,d_model]
    -> explicit semantic output-owner projection
decision_hidden [1,d_model]
    -> M3.8 output-head interface
raw_logits [1,V]
    -> ordered local policy + RNG/constraint state
selected y_0 [1,1]

For a decoder-only prompt with IDs $x_{0:L-1}$, the call and row ownership are:

Call	Model-input rows	Existing cache before	Score row means	Newly selected output
prefill	$x_0,\ldots,x_{L-1}$	0	$p(y_0\mid x_{0:L-1})$	$y_0$
decode 1	$y_0$	$L$	$p(y_1\mid x_{0:L-1},y_0)$	$y_1$
decode 2	$y_1$	$L+1$	$p(y_2\mid x_{0:L-1},y_{0:1})$	$y_2$

The input and output indices differ by one because row $i$ is a representation of the token at $i$ and predicts the token after $i$. The selected token exists first as output-policy state. It becomes hidden/K/V state only when a later valid call processes it.

For an encoder-decoder, source and target address spaces remain separate:

source IDs [1,S]
    -> encoder memory [1,S,d_model]             # fixed for the request

decoder-start ID [1,1]
    -> decoder hidden [1,1,d_model]
    -> target logits [1,V_target]
    -> y_0

y_0 is appended to the target prefix or processed against target self-cache;
encoder memory and source-indexed cross-attention state remain unchanged.

The first target input may be BOS, a model-specific decoder-start ID, a forced language token, or another architecture-defined start state. Treating these as interchangeable can preserve every tensor shape while changing the generated language or output distribution.

Output, pending-input, and processed-state frontiers

Let:

L_prompt: valid decoder-side prompt-input count under the admitted profile
O_t:      number of generated output tokens selected and logically committed
U_t:      tagged committed-output suffix not yet processed by the model
P_t:      complete logical processed-input frontier
K_t:      conditional cached-baseline logical frontier, when the profile has one

For ordinary decoder-only generation immediately after selection:

$$O_t=t+1, \qquad P_t=K_t=L_{prompt}+t, \qquad |U_t|=1,$$

because $y_t$ has been selected but not yet processed. Immediately after the next valid append call completes every required compute stage and advancing state lane for $y_t$:

$$P_{t}^{+}=K_t^{+}=L_{prompt}+t+1= L_{prompt}+O_t, \qquad |U_t^{+}|=0.$$

The just-completed call may now expose a consumable score only if two closed rosters agree: every declared compute stage through final export/vocabulary projection completed, and every architecture-required advancing state lane completed at the same logical frontier. A missing lane selects ALL_OLD, yields no consumable score, and permits no decision or logical commit. For recurrent, state-space, hybrid, windowed, or no-cache profiles, use P_dyn plus explicit F_lane values rather than inventing a physical K/V length.

sequence_length, generated_length, logical processed frontier, physical cache length, reserved capacity, and persistent committed length may therefore name different integers at the same controller boundary. Every API and metric must state which frontier it reports.

Likewise, retained-window length, allocation high-water mark, largest valid logical slot, maximum address ever touched, and maximum physical address are separate quantities. None can be substituted for $P$ or for another lane’s frontier merely because the values coincide in a contiguous baseline.

0x01 Minimal encoder-decoder greedy loop

The shortest useful encoder-decoder inference loop is the one already hiding inside the source articles:

def inference_test():
    test_model = make_model(11, 11, 2)
    test_model.eval()

    src = torch.LongTensor([[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]])
    src_mask = torch.ones(1, 1, 10)

    memory = test_model.encode(src, src_mask)
    ys = torch.zeros(1, 1).type_as(src)

    for i in range(9):
        out = test_model.decode(
            memory,
            src_mask,
            ys,
            subsequent_mask(ys.size(1)).type_as(src.data),
        )
        prob = test_model.generator(out[:, -1])
        _, next_word = torch.max(prob, dim=1)
        next_word = next_word.data[0]
        ys = torch.cat(
            [ys, torch.empty(1, 1).type_as(src.data).fill_(next_word)],
            dim=1,
        )

This is not a generic “Transformer outputs words” story. It is a concrete object trace:

Code object	Shape	Meaning
src	[B,S]	source token IDs
src_mask	[B,1,S]	source padding visibility
memory = encode(src, src_mask)	[B,S,d_model]	source-side encoder states
ys at prefix length t	[B,t]	generated target prefix, including start token
subsequent_mask(t)	[1,t,t] or [B,t,t]	target causal visibility
out = decode(...)	[B,t,d_model]	decoder hidden states for the whole visible prefix
out[:, -1]	[B,d_model]	last prefix state used for the next-token distribution
generator(out[:, -1])	[B,V_tgt]	vocabulary scores or log-probabilities
next_word	[B] or scalar in toy code	selected next token ID
cat([ys,next_word])	[B,t+1]	next decoder input prefix

The source side is computed once:

memory = test_model.encode(src, src_mask)

In M3.2 terms, memory is still source-indexed:

$$M \in \mathbb{R}^{B \times S \times d_{\text{model}}}$$

The decoder can read that memory at every step through cross-attention. The target side, however, changes after every selected token:

$$ys_t \in \mathbb{N}^{B \times t}$$

No-image fallback: source IDs and source mask produce encoder memory once. Every target step keeps that source-indexed memory fixed, while decoder self-state follows the decoder-start token and selected target tokens. Optional cross-attention K/V stays source-indexed; target self-state advances only when a selected target token is later processed.

The important part of this figure is the feedback edge on the target side. The encoder output can stay fixed for a fixed source sentence, but the decoder input prefix must be rebuilt or extended after each selected token.

The loop can be read as a table:

Step	ys before decode	selected from generator(out[:, -1])	ys after append
0	<bos>	I	<bos> I
1	<bos> I	ate	<bos> I ate
2	<bos> I ate	an	<bos> I ate an
3	<bos> I ate an	apple	<bos> I ate an apple
4	<bos> I ate an apple	<eos>	<bos> I ate an apple <eos>

If step 1 selects like instead of ate, the following steps condition on <bos> I like .... There is no gold tgt_y to repair the prefix.

Encoder-decoder cached state has two lifetimes

In a cached encoder-decoder, each decoder layer can own:

self-attention K/V:
    target-prefix indexed
    grows by one row per processed generated token

cross-attention K/V:
    source indexed
    derived from fixed encoder memory
    reusable at every target step

For layer $l$:

$$K^{self}_l,V^{self}_l\in\mathbb R^{B\times H_{kv}\times T\times d_h},$$ $$K^{cross}_l,V^{cross}_l\in\mathbb R^{B\times H_{kv}\times S\times d_h}.$$

An implementation may cache cross-attention projections once or recompute them from encoder memory. Recomputing can remain mathematically correct but repeats source projection work. Mutating source memory or cross K/V during ordinary target decode is a correctness bug.

Decoder-only, prefix-decoder, and encoder-decoder visibility

No-image fallback: a causal decoder lets each continuation row read permitted earlier-or-equal combined-stream rows; a prefix decoder may allow bidirectional visibility inside a fixed prefix but causal visibility in the continuation; an encoder-decoder combines causal target self-attention with cross-attention to fixed source states. All still require earlier output decisions before later free-running inputs exist.

causal decoder:
    every new row reads the entire preceding combined prompt/output stream

prefix decoder:
    prefix rows may have bidirectional visibility;
    generated continuation remains causal

encoder-decoder:
    decoder causal self-attention + cross-attention to fixed encoder states

A cache implementation must match the trained visibility contract. Reusing a causal-decoder mask for a prefix decoder can make prefix representations depend on less context than training; letting generated rows modify source memory changes the encoder-decoder contract.

Architecture family changes the carried state

The repeated decision/commit loop is shared, but model state is not always a decoder-only K/V tuple.

Family	Fixed request state	State that grows or transitions	First decision row
decoder-only causal LM	artifact and serialized prompt identity	self-attention K/V or equivalent history state; output/stop/policy state	last valid prompt row
prefix LM / unified seq2seq	bidirectional prefix boundary and segment roles	causal continuation state after the prefix	last prefix row under the trained prefix mask
encoder-decoder	source IDs/mask, encoder memory, optionally projected cross K/V	target self-attention state and generated target prefix	decoder-start row conditioned on source memory
recurrent decoder	source context if any	recurrent hidden/cell state plus previous token	current recurrent output state
state-space or attention hybrid	layer-specific static metadata	recurrent/convolution states for some layers and K/V for attention layers	newest valid model output row

For a decoder-only model:

$$S_t^{dec}=(x_{0:L-1},y_{<t},C_t^{self},P_t,R_t,F_t).$$

For an encoder-decoder:

$$S_t^{encdec} = (M_x,C^{cross},y_{<t},C_t^{self},P_t,R_t,F_t),$$

with

$$M_x=\operatorname{Encoder}(x,A_x), \qquad C^{cross}=\operatorname{CrossKV}(M_x)$$

when cross projections are cached. $M_x$ and $C^{cross}$ are invariant for a fixed source request; $C_t^{self}$ advances with processed target inputs. A runtime may omit $C^{cross}$ and project from $M_x$ repeatedly. That changes work, not the source-state identity.

No-image fallback: target inputs enter masked decoder self-attention, then cross-attend to fixed encoder output, then traverse the checkpoint-declared MLP/residual/normalization assembly. The semantic final target row is projected to vocabulary scores for the next decision.

The A4 figure combines training and architecture language. During inference, the bottom-right Outputs (shifted right) object is the decoder-start token plus generated prefix, not the unavailable gold target.

Prefix execution has a phase change inside one sequence

A prefix LM can allow prefix rows to see one another bidirectionally while the continuation remains autoregressive. For prefix length $P$ and total current length $T$, one visibility contract is

$$V(i,j)= \begin{cases} 1,&i<P\ \land\ j<P,\\ 1,&i\ge P\ \land\ j\le i,\\ 0,&\text{otherwise}. \end{cases}$$

The prefix can be computed once because generated continuation rows do not become visible to earlier prefix queries. Replacing this with a globally lower triangle changes prefix representations. Replacing it with global bidirectionality leaks future generated rows.

prefix_length, segment roles, and any prefix-specific positions are therefore immutable request fields. The decode cache alone cannot reconstruct the mask that produced its prefix rows unless that metadata is retained.

Recurrent and hybrid state generalize the cache frontier

For a recurrent layer with state $r_t$:

$$(h_t,r_{t+1})=F_\theta(e(u_t),r_t).$$

The state size need not grow with token count, so cache_length is not a universal storage dimension. The controller still needs a processed-input frontier $K_t$ to prove which token prefix produced $r_t$. A hybrid model may carry:

attention layer -> K/V rows indexed by retained token positions
state-space layer -> recurrent/convolution state at processed frontier K_t
cross-attention layer -> fixed source-indexed K/V

All components must represent the same logical processed prefix before the next decision. Branching requires architecture-specific fork/copy semantics; blindly calling reorder_cache on only attention K/V leaves recurrent layers on the wrong parent.

Architecture-qualified conformance tests

Family	Minimum one-request equivalence check
decoder-only	full-prefix last logits versus cached incremental logits
prefix LM	trained prefix-mask one-shot result versus cached continuation from the same prefix state
encoder-decoder	fixed encoder memory/cross state across target steps; cached target self-state versus full target-prefix decode
recurrent/state-space	token-by-token state versus supported chunked state transition
hybrid	every layer-family state advances from the same processed-input frontier

Passing one family’s test does not validate another. In particular, K/V length equality says nothing about a recurrent state that was not reordered or a cross cache projected from the wrong encoder memory.

0x02 One step as a shape ledger

At prefix length t, the decoder returns one hidden state per prefix position:

$$out \in \mathbb{R}^{B \times t \times d_{\text{model}}}$$

The next token is not selected from all rows. It is selected from the last valid prefix row:

last = out[:, -1]              # [B,d_model] in the simple no-padding case
scores = model.generator(last) # [B,V_tgt]
next_token = select(scores)    # [B] or [B,1]
ys = torch.cat([ys, next_token.reshape(B, 1)], dim=1)

For a no-padding single-row example, out[:, -1] means “the hidden state of the final prefix token.” That hidden state has seen the available target prefix through masked self-attention and, in encoder-decoder models, the source memory through cross-attention.

The exact one-step trace is:

Object	Shape	Contract
prefix IDs	[B,t]	visible generated target tokens so far
prefix hidden states	[B,t,d_model]	one decoder state per prefix row
last hidden state	[B,d_model]	one state per batch row for next-token prediction
vocabulary scores	[B,V]	one score per vocabulary item
selected token IDs	[B] or [B,1]	concrete token IDs chosen by the decoding policy
updated prefix	[B,t+1]	next loop input

This is also the boundary between M3.9 and M3.10. Training loss uses every valid target row after flattening. Inference usually selects from one row per active hypothesis.

Why logits at row i predict token i+1

Under causal LM training, hidden row $h_i$ sees tokens $x_{\le i}$ and its output head is supervised against $x_{i+1}$:

$$z_i=\operatorname{Head}(h_i), \qquad p(x_{i+1}\mid x_{\le i})=\operatorname{softmax}(z_i).$$

At inference, the prompt’s last valid input row therefore predicts the first generated token. Feeding that generated token produces the next hidden row and the next distribution. Appending a selected token to a token array does not retroactively create its hidden state.

This alignment also explains empty-prompt handling. A model still needs a valid initial input convention, such as BOS or a structured prefix. There is no hidden row from which to predict if the implementation supplies zero input positions and has no special start-state path.

A bounded cached profile—not a complete controller

The following sketch keeps the ordinary decoder-only, batch-size-one decision/commit/state timing explicit. It deliberately delegates schema validation, text/structured stops, cancellation, errors, and stream holdback to named interfaces; calling the shorter loop “complete” would erase those branches.

@torch.inference_mode()
def generate_one(
    model,
    prompt_ids,
    *,
    policy,
    max_new_tokens,
    eos_ids,
    streamer=None,
):
    # prompt_ids: [1,L], L >= 1
    validate_bound_request(prompt_ids, policy, max_new_tokens, eos_ids, streamer)
    if max_new_tokens == 0:
        # This profile chooses NO_MODEL_CALL. Other typed profiles may prefill.
        return GenerationResult([], "length", None, pending_input=None)

    out = model(
        input_ids=prompt_ids,
        use_cache=True,
    )
    cache = out.past_key_values          # K/V for prompt inputs only
    scores = out.logits[:, -1, :]        # predicts first output

    state = initial_mutable_state()

    for step in range(max_new_tokens):
        receipt = policy.decide(scores, state.policy_rng)  # M3.8 receipt
        state = apply_output_transaction_once(
            state,
            receipt,
            eos_ids=eos_ids,
            at_length_limit=(step + 1 == max_new_tokens),
            streamer=streamer,
        )
        if state.terminal:
            break

        # Only now does the selected token become a model input and acquire K/V.
        candidate = run_bound_model_call(
            input_ids=state.pending_next_input,
            old_state=cache,
            request_identity=state.request_identity,
        )
        cache, scores = select_all_new_only_if_rosters_complete(candidate, cache)

    if streamer is not None:
        streamer.finish(state.finish_reason, state.safe_stream_frontier)

    return GenerationResult.from_state(state, cache)

This profile must close every terminal and failure branch explicitly:

Branch	Required result before another call
malformed, oversized, empty-without-start-state, or incompatible request	typed rejection; no model, policy, RNG, cache, or stream mutation
zero allowed new tokens	declared no-call or prefill-only profile; zero output and a terminal reason
ordinary selection	exactly one authenticated M3.8 receipt consumed; aligned token/logprob and policy/RNG state
EOS, length, stop text, or structured completion	one precedence decision; safe stream flush; pending token tagged CLOSED_UNPROCESSED unless repaired
constraint/parser error	typed terminal error with no unsupported next call
cancellation/timeout before launch	no launch and no logical mutation
cancellation/timeout or device error in flight	await/abort under the declared runtime contract, select ALL_OLD, and expose no partial token/state
streamer failure	apply the declared output-transaction policy; never claim an externally delivered chunk without its token metadata
successful nonterminal transition	one PENDING_NEXT_CALL suffix plus complete old/new identity receipts

If stop strings or structured completion can fire after a non-EOS token, evaluate those conditions after incremental decode and before the next model call. Then the final output token can be logically committed but absent from processed model state, exactly like an EOS- or length-stopped final token. A pre-selection support mask and the prospective post-append input mask are different revisions; one must not be reused as the other without derivation.

If an API promises a cache including every returned token, it must run an additional append-only forward for the final token or define a different model output/cache contract. That extra call has cost and can return logits that are deliberately discarded.

Prompt logprobs versus generated-token logprobs

Prefill returns one logit row per prompt input row. For prompt tokens $x_0,\ldots,x_{L-1}$:

logits at input row 0 score x_1
logits at input row 1 score x_2
...
logits at input row L-2 score x_(L-1)
logits at input row L-1 score first generated token

There is no ordinary model score for $x_0$ without an earlier BOS/context row. Prompt logprob APIs must define whether they return None, omit the first token, or score it under an injected start token.

For generated token $y_t$ selected from score row $z_t$:

$$\log p(y_t)=z_{t,y_t}-\operatorname{logsumexp}(z_t).$$

Record this value before advancing the model. The next decode call’s logits score $y_{t+1}$, not the token just fed as input.

0x03 Decoder-only serving loop

Decoder-only inference removes the separate source encoder. The prompt, retrieved context, previous conversation, tool traces, and generated answer all live in one causal stream:

system instruction
user message
retrieved text
partial assistant output

The same last-position dependency remains. A no-cache decoder-only loop can be written as:

for step in range(max_new_tokens):
    hidden = model(input_ids, attention_mask=attention_mask)  # [B,L,d_model]
    last = gather_last_valid(hidden, attention_mask)          # [B,d_model]
    scores = lm_head(last)                                    # [B,V]
    next_token = select(scores)                               # [B]

    input_ids = append(input_ids, next_token)                 # [B,L+1]
    attention_mask = append(attention_mask, 1)                # [B,L+1]
    update_finished_rows()

The LLaMA-style generate() path in the source articles makes the serving objects more concrete:

prompt_tokens: List[List[int]]
tokens = torch.full((bsz, total_len), pad_id, dtype=torch.long, device="cuda")
input_text_mask = tokens != pad_id

prev_pos = 0
eos_reached = torch.tensor([False] * bsz, device="cuda")
stop_tokens = torch.tensor(list(self.tokenizer.stop_tokens))

for cur_pos in range(min_prompt_len, total_len):
    logits = self.model.forward(tokens[:, prev_pos:cur_pos], prev_pos)

    if temperature > 0:
        probs = torch.softmax(logits[:, -1] / temperature, dim=-1)
        next_token = sample_top_p(probs, top_p)
    else:
        next_token = torch.argmax(logits[:, -1], dim=-1)

    next_token = next_token.reshape(-1)
    next_token = torch.where(
        input_text_mask[:, cur_pos],
        tokens[:, cur_pos],
        next_token,
    )
    tokens[:, cur_pos] = next_token

    eos_reached |= (~input_text_mask[:, cur_pos]) & (
        torch.isin(next_token, stop_tokens)
    )
    prev_pos = cur_pos

    if all(eos_reached):
        break

That line is a pinned historical-source observation, not a portable tensor contract. Before torch.isin, a generic implementation must bind stop IDs to the compared token tensor’s device and integer dtype:

stop_tokens = torch.as_tensor(
    sorted(validated_stop_ids),
    device=next_token.device,
    dtype=next_token.dtype,
)

The validated set must be nonempty or follow an explicit empty-set policy, and every ID must lie in the admitted vocabulary.

The variables are the article’s real serving objects:

Object	Shape / type	Meaning
prompt_tokens	list of ID lists	uneven prompt inputs before rectangular batching
tokens	[B,total_len]	prompt plus generated slots
input_text_mask	[B,total_len]	which positions belong to the original prompt
prev_pos	scalar	first not-yet-processed position for the next forward chunk
cur_pos	scalar	position being filled or preserved
logits[:, -1]	[B,V]	scores for the next position
next_token	[B]	selected candidate, unless prompt token must be preserved
eos_reached	[B]	finished-row state

The line below is easy to miss:

next_token = torch.where(input_text_mask[:, cur_pos], tokens[:, cur_pos], next_token)

It prevents generation from overwriting prompt tokens while rows with shorter prompts have already entered generated-token territory. That is not an architecture detail. It is serving-loop bookkeeping.

At the probability level, decoder-only generation still follows:

$$p_\theta(y_1,\ldots,y_m \mid C) = \prod_{t=1}^{m} p_\theta(y_t \mid C,y_{<t})$$

where $C$ is the context before the answer begins.

0x04 Score conventions and the generator boundary

At each step, the model has a hidden state before it has a token:

last hidden state
    -> Generator / LM head
        -> vocabulary-width scores
            -> decoding rule
                -> selected token ID

The Harvard Generator is:

class Generator(nn.Module):
    "Define standard linear + softmax generation step."

    def __init__(self, d_model, vocab):
        super(Generator, self).__init__()
        self.proj = nn.Linear(d_model, vocab)

    def forward(self, x):
        return log_softmax(self.proj(x), dim=-1)

So the shape change is:

$$h_t \in \mathbb{R}^{B \times d_{\text{model}}} \quad\rightarrow\quad s_t \in \mathbb{R}^{B \times V}$$

No-image fallback: one admitted semantic hidden row of width d_model enters the learned output head and produces one row of width V. Raw logits, log-probabilities, and probabilities are distinct score kinds. Only an authenticated M3.8 policy turns the row into a pending token or beam decision.

The figure is useful because it shows the dimensional jump from hidden width to vocabulary width. The exact tensor name depends on the implementation:

Name	Meaning	Can be used with
logits	raw linear scores	argmax, softmax, temperature, top-k filtering
log-probs	log_softmax(logits)	argmax, NLL-style scoring, log-prob beam sums
probabilities	softmax(logits)	multinomial sampling, top-p cumulative probability

The toy Harvard code calls the tensor prob, but its Generator returns log_softmax. That distinction matters. torch.max or argmax gives the same token for logits and log-probs, but multinomial sampling expects non-negative probability weights.

No-image fallback: vocabulary IDs form the candidate axis. The policy may take an argmax over compatible scores or transform admitted raw logits into normalized probabilities before sampling. Candidate indices must map back through the authenticated vocabulary/token map; no displayed word has been committed yet.

The model has not emitted the displayed word yet. It has emitted a vocabulary-width score row. Call it a distribution only after the declared normalization. Also keep it separate from the attention score tensor $QK^\top/\sqrt{d_h}$: attention scores have shape [B,H_q,Q,K], whereas head-owned vocabulary logits have shape [R_head,V]. Both are sometimes called scores, but their axes, support, normalization, and consumers are unrelated. The M3.8 decision receipt binds score kind, vocabulary axis/map, support, ordered transforms, parameters, tie rule, and RNG address before M3.10 accepts a token.

0x05 Selection policies over [B,V]

Every decoding rule in this section starts from a vocabulary-width row:

scores: [B,V]

Greedy decoding chooses the highest-scoring vocabulary item:

next_token = torch.argmax(scores, dim=-1)  # [B]

This is deterministic and cheap. It also commits early. Once a weak token is appended, later steps condition on it.

Temperature changes logits before sampling:

probs = torch.softmax(logits[:, -1] / temperature, dim=-1)

Top-k filtering keeps only the k highest-scoring candidates, sets the others to -inf, and samples after softmax. Top-p, or nucleus sampling, keeps the smallest probability-sorted candidate set whose cumulative mass crosses a threshold.

The LLaMA-style top-p code is a good implementation object:

def sample_top_p(probs, p):
    probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True)
    probs_sum = torch.cumsum(probs_sort, dim=-1)

    mask = probs_sum - probs_sort > p
    probs_sort[mask] = 0.0
    probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True))

    next_token = torch.multinomial(probs_sort, num_samples=1)
    next_token = torch.gather(probs_idx, -1, next_token)
    return next_token

There are two important boundaries here:

1. Top-p operates on probabilities, not raw logits.
2. The sampled index is initially in sorted-candidate space, so it must be mapped back through probs_idx.

Beam search keeps multiple prefixes instead of one:

tokens: [B * beam, L]
scores: [B, beam, V]
candidate space per row: beam * V

At each step, a beam implementation expands candidates, selects the best (parent_beam, token) pairs, gathers the matching prefixes, gathers or reorders any cache state, and stores completed hypotheses separately from active ones.

No-image fallback: at each step, expand each live beam over vocabulary candidates, rank the combined beam-by-vocabulary space, move terminal candidates to a completed set, and retain a bounded live frontier. Every retained child must inherit one parent's complete token, score, length/coverage, model state, position, constraint, RNG, decoder/stream, status, and ancestry state before its new token is appended.

Beam search is therefore a bookkeeping algorithm over prefixes, scores, and cache rows. It is not a different Transformer block.

Applying one beam decision to request state

M3.8 returns:

next_scores: [B,K]
parent_ids:  [B,K]
token_ids:   [B,K]

M3.10 applies those parent IDs to every beam-owned object:

flat_parent = (
    torch.arange(B, device=parent_ids.device)[:, None] * K
    + parent_ids
).reshape(-1)

tokens = tokens.index_select(0, flat_parent)
attention_mask = attention_mask.index_select(0, flat_parent)
position_state = position_state.index_select(0, flat_parent)
constraint_state = reorder_constraints(constraint_state, flat_parent)
cache = cache.reorder_cache(flat_parent)

tokens = append(tokens, token_ids.reshape(B * K))
beam_scores = next_scores

The selected token has not yet acquired its own cache row. After reorder, the next model call processes that token under the chosen parent’s cache. Reordering a branch cache that already includes candidate-token K/V requires an implementation-specific verified tree/beam path; ordinary beam search normally reorders parent cache then appends the selected input.

Finished hypotheses should be copied into a finished set with their score and length metadata. Live beam slots are refilled from non-finished candidates according to the search contract. A finished hypothesis’s cache need not be extended unless the API deliberately keeps rectangular dummy rows.

A beam is a coupled branch-state vector

For request $b$, beam $k$, and decision step $t$, define

$$\mathcal B_{b,k,t} = (X,S,L,C,P,Q,R,D,F,\Pi,W),$$

where:

X: token prefix and visible token spans
S: cumulative and normalized search-score state
L: generated length and any coverage/length statistics
C: per-layer parent cache or branch-cache reference
P: position, cache-position, and visibility state
Q: constraint/grammar automaton state
R: RNG state or deterministic child-stream rule
D: incremental tokenizer decoder and stop matcher state
F: active/completed/invalid status and finish reason
Pi: ancestry/provenance needed for returned scores or paths
W: authenticated search-plan/policy revision and decision-consumption status

index_select over token rows is correct only if the same parent map is applied, or translated, across every component. A checksum makes the coupled invariant testable:

$$I_{branch}(b,k,t) = H(I_{request}\Vert b\Vert k\Vert t\Vert X\Vert S\Vert L\Vert C\Vert P \Vert Q\Vert R\Vert D\Vert F\Vert\Pi\Vert W).$$

After reorder, every field for child $(b,k)$ must descend from the parent named by parent_ids[b,k], except fields explicitly initialized from the new token. The identity includes cumulative/normalized score, length/coverage statistics, active/finished/invalid status, finish reason, search plan, and whether the pending decision has already been consumed. Omitting any of those can preserve shape while changing ranking, termination, or exactly-once behavior.

No-image fallback: for each selected child (b,k), use parent_ids[b,k] to gather the same parent's X,S,L,C,P,Q,R,D,F,Pi,W fields. Fork that parent state before appending the new child token. The new token is pending input and has no model-state row yet. Any omitted or differently mapped lane invalidates the branch even if its tensor shape and final text match.

Fork parent state before appending the child token

One parent can produce several surviving children. With beam width two:

selected child 0: parent 0 + token A
selected child 1: parent 0 + token B

Both children share the same committed parent prefix. Their new tokens differ and have not yet been processed. A dense implementation duplicates parent cache rows with index_select; a reference-counted implementation can fork a logical prefix and allocate separate append state later. Either must preserve:

child cache before append == parent cache
child pending input        == selected child token
child position             == parent committed length

Mutating one shared cache in place before the fork closes can cause sibling A to observe sibling B’s token. Copy-on-write or generation-qualified ownership is therefore a correctness mechanism, not only a memory optimization.

Active and completed sets have different lifetimes

A completed hypothesis is a candidate result, not an active model input. On EOS or another terminal rule:

snapshot output tokens, score statistics, finish reason, and ancestry
move the hypothesis to the completed set
do not spend another ordinary model call on it
refill the live frontier according to the search contract

Beam-level termination is separate from token-level completion. Common contracts include:

Search termination	State rule	Limitation
fixed complete count	stop when enough completed hypotheses exist	can stop before a live prefix loses any chance to beat the result
heuristic early stop	compare completed score with a runtime bound/heuristic	result depends on the exact score and bound implementation
exact relative to a declared finite-beam search trace	stop only when no still-live hypothesis in that trace can beat the best complete result	requires an admissible bound for the declared score, fixed beam width, deterministic tie/pruning rules, and the complete prior pruning history; it is not global exact search
hard length/context cap	close all remaining hypotheses at the cap	truncation is a finish reason, not EOS

The score and bound mathematics belong to M3.8. M3.10 owns the fact that live and completed sets, their cache state, and their finish metadata must not be collapsed.

Search procedures change required runtime state

Procedure	Frontier state beyond one ordinary path	Execution consequence
greedy or multinomial	one prefix, one cache, one stop/RNG state	no branch reorder
beam search	fixed-width live prefixes plus completed set	coupled parent gather and cache fork/reorder
diverse/group beam	group-local frontier plus diversity statistics	group order and diversity state affect later selection
lexically constrained/grid beam	automaton/grid coverage state per branch	constraint state must fork/reorder with tokens
best-first or A*-style search	variable priority queue of partial hypotheses	cache snapshots/recomputation and queue tie-breaking become state
contrastive search	candidate hidden states and degeneration score against prefix representations	candidate forward state must be committed or discarded consistently

These are not interchangeable switches over one loop. They request different model rows, retain different branch state, and use different stopping proofs. Speculative decoding is still outside this table’s ownership: M7 proposes and verifies multiple future tokens to reduce target-model steps, whereas these procedures search or rank output hypotheses.

Paper contracts should not be flattened

Source contract	Search contribution	M3.10 integration obligation
GNMT	length normalization and source coverage penalty for NMT beam search	preserve source-attention coverage statistics with each beam and qualify score-dependent stopping
Optimal Beam Search	termination proof modulo beam size for a declared score, plus bounded length reward	use its bound only with the score assumptions it proves
Diverse Beam Search	group-wise diversity-augmented selection	carry group identity and diversity history
Grid Beam Search	hard lexical-constraint progress in a grid	reorder lexical coverage state with every branch
Best-First Beam Search	priority-queue schedule with replaceable search components	cache/recompute policy and queue tie-breaking become reproducibility state
Contrastive Search	combine model confidence with representation degeneration penalty	request candidate hidden states; do not treat it as score-row-only sampling

Reported quality or speed results remain source-reported and task-specific. The durable point is the changed state contract.

0x06 Stop criteria and active rows

The loop needs a stop rule. Common stop objects include:

EOS token
maximum new-token limit
stop token set
stop string after detokenization
complete JSON / structured object
complete tool-call-like output
external controller stop

In a batched loop, stop state is usually row-wise:

eos_reached: [B]
active:      [B]

If row 1 has already emitted <eos> but row 2 is still generating, the batch may continue. The finished row must be masked, forced to emit padding/EOS, compacted away, or otherwise kept out of future selection. Otherwise it will generate tokens after completion.

Stop strings add a second problem: text boundaries are not token boundaries. A stop string may be split across several tokens, so checking only the last token is insufficient. If logprobs are returned, trimming must keep the token list and the logprob list aligned:

for stop_token in self.tokenizer.stop_tokens:
    try:
        eos_idx = toks.index(stop_token)
        toks = toks[:eos_idx]
        probs = probs[:eos_idx] if logprobs else None
    except ValueError:
        pass

The details vary by serving stack, but the invariant is simple: the sequence rows, score rows, cache rows, active flags, and returned metadata must refer to the same hypotheses.

Token stops, text stops, and structured stops

These stop mechanisms observe different representations:

EOS/token-set stop:
    evaluated on selected token IDs

stop-string stop:
    evaluated on incrementally decoded text/bytes

structured stop:
    evaluated on parser state, for example complete JSON or a tool call

length stop:
    evaluated on generated-token count or total context length

A stop string may span token boundaries. Incremental checking needs a suffix buffer at least long enough to cover the largest stop pattern in the tokenizer’s decoded byte/text representation. Re-decoding each token independently can be wrong because byte fallback, whitespace cleanup, or tokenizer prefix rules may combine tokens nonlocally.

If a matched stop string should be excluded from returned text, trim at the text/byte boundary and preserve a mapping to token/logprob offsets. The match can end inside a token’s decoded string, so token-aligned trimming may retain extra characters or drop too much unless the API explicitly defines token-level behavior.

Structured generation is similar. A parser can report:

incomplete but valid prefix
complete object
invalid prefix

The controller should not equate a closing brace token with semantic completion if it occurs inside a string or nested object.

Bound parsers, matchers, traces, and sensitive data

Correct state transitions are not a resource or privacy policy. Admission must freeze finite limits before parsing or generation:

raw request bytes and decoded code points
prompt tokens, total context, and maximum new tokens
stop-pattern count and maximum encoded-pattern length
grammar size, parser nesting depth, node count, and transition budget
beam/candidate count, retained ancestry, and per-step metadata
incremental decoder/matcher suffix and total returned bytes
trace probes, tensor/candidate capture size, and diagnostic event count
wall/device budget plus cancellation checkpoints

Exceeding a bound produces a typed rejection or terminal resource reason; it must not silently truncate an identity-bearing field. Prompt bytes, token IDs, decoded fragments, candidate lists, logprobs, parser state, and snapshots can contain user content or secrets. A separate trace passport therefore binds data classification, field-level redaction or omission, access roles, encryption, retention/deletion deadlines, export policy, and audit purpose. Prefer opaque request/artifact digests and aggregate counters when raw content is unnecessary. Neither a hash nor a schema is consent to log the underlying value.

Streaming has a commit horizon

Streaming emits text before the request completes. Some token bytes cannot be safely emitted immediately because a future token or stop match may alter the decoded suffix. A robust streamer keeps a small uncommitted suffix:

committed text:
    safe to expose and never retract

pending suffix:
    enough bytes/tokens for tokenizer cleanup and stop-pattern overlap

Logprobs, token IDs, character offsets, and finish reason should be emitted under the same commit decision. Returning text before its associated token metadata is finalized creates irreconcilable alignment.

RNG state belongs to the request policy

For stochastic sampling, request state includes a pseudorandom stream or counter. One global mutable generator makes results depend on whether other requests draw before this one. Single-request semantics can define:

seed + draw counter
counter-based RNG keyed by request and step
saved generator state

Beam sampling may require one stream per beam or a deterministic mapping from parent beam to child draws. Parent reorder must then include RNG state or a reproducible derivation of it.

One selection is an output-state transaction

Stopping is not one boolean after sample(). A selected token changes several representations, and each observer sees a different one:

1. policy selects token ID u_t and records the score/logprob row that owns it
2. append u_t to the logical output-token ledger
3. feed u_t through the incremental tokenizer decoder
4. update byte/text stop matcher and structured parser
5. resolve EOS, length, text, grammar, timeout, or cancellation precedence
6. advance only the RNG state actually consumed by this decision
7. publish only the text and metadata behind the stream commit horizon
8. if unfinished, expose u_t as the next model input; otherwise close output

An implementation may combine operations, but changing their semantic order changes results. min_new_tokens, for example, normally modifies the allowed EOS support before selection. A text stop is discovered after decoding the selected token. A timeout observed during a model call may close after the call rather than interrupting and partially publishing model state.

Define a transition result:

$$T_t=(u_t,\ell_t,D_t,Q_t,R_{t+1},F_t,U_{t+1}),$$

where $\ell_t$ is aligned token metadata, $D_t$ is full decoded state, $Q_t$ is the safely publishable stream prefix, $R_{t+1}$ is policy/RNG/constraint state, $F_t$ is finish state, and $U_{t+1}$ is the next model-input suffix. If $F_t$ is terminal, ordinary execution leaves $U_{t+1}$ unprocessed.

No-image fallback: bind the decision's score/logprob row; append its token ID and metadata; advance the incremental decoder, byte/text matcher, structured parser, and only the RNG draws actually consumed; resolve cancellation, timeout, parser error, stop string, EOS, and length using the declared precedence; publish only the safe text/metadata prefix; then expose the token as PENDING_NEXT_CALL or close it as CLOSED_UNPROCESSED.

Deriving the stream holdback

Let $D_t$ be decoded bytes after token $t$, and let $\mathcal P$ be the set of nonempty stop byte strings. A future token can complete a stop only through a suffix of $D_t$ that is a proper prefix of some stop pattern. Define

$$h_t= \max\left( \{0\}\cup \left\{r: \operatorname{suffix}_r(D_t) \in\operatorname{ProperPrefix}(\mathcal P) \right\} \right).$$

Ignoring additional tokenizer-decoder pending bytes for one line, bytes before $|D_t|-h_t$ are safe to stream. For stop pattern <STOP>:

decoded bytes     suffix that can still become <STOP>    safe to publish
hello<            <                                  hello
hello<ST          <ST                                no additional bytes
hello<STOP>       full match                         finish; trim/include by contract

The tokenizer’s incremental decoder can require more holdback than the stop matcher. UTF-8 code points, byte fallback, whitespace cleanup, and normalizer state can span token boundaries. The safe stream frontier is therefore the minimum frontier accepted by both decoder and stop matcher.

Token-aligned and text-aligned outputs can diverge

A stop match can begin or end inside one token’s decoded byte string:

token 101 -> b"hello<"
token 102 -> b"ST"
token 103 -> b"OP>"
stop      -> b"<STOP>"

Excluding the stop yields returned text hello, but there is no token boundary between hello and < inside token 101. An API must choose and document one of these contracts:

Output contract	Consequence
token-aligned	keep/drop whole tokens; returned text may include extra stop bytes or lose preceding text
text-aligned	trim exact bytes/chars; returned token IDs no longer decode exactly to returned text
span-aligned	retain tokens and publish byte/character spans marking visible and trimmed ranges

Token logprobs still belong to selected token IDs. They cannot be split into an exact logprob for the visible prefix hello and the trimmed suffix < without an additional model/tokenization definition.

Constraints before selection, completion after selection

A grammar or finite-state constraint can play two roles:

pre-selection support constraint:
    parser state -> allowed next token set -> M3.8 policy support

post-selection completion detector:
    selected token bytes -> parser transition -> complete/incomplete/error

The first changes the sampling distribution and must execute in the declared processor order. The second decides whether the accepted output is complete. A parser state that says complete must be coupled to the exact token/byte prefix and reordered with its beam. A closing brace alone is not proof of JSON completion.

Finish reason and RNG precedence

Multiple conditions can become true at one selection. Freeze a precedence rule, for example:

cancelled > timeout > structured_error > stop_string > eos > length

This ordering is an API decision, not a universal standard. Record every trigger for diagnostics and publish one canonical finish reason. RNG advances only for stochastic operations that actually execute. Forced prompt tokens, greedy argmax, masked-out finished rows, and rejected pre-validation paths must not silently consume draws if exact replay is promised.

Cancellation also has a commit boundary. If cancellation arrives during a forward call, the runtime may let the call finish and then discard its result, or may abort under a documented recovery contract. It must not return text whose token/logprob/cache transaction was only partially committed.

0x07 Batched inference and padding-side bugs

The last-position expression:

out[:, -1]

is only safe when the last rectangular column is also the last semantic token for every batch row.

Suppose two prompts have different lengths:

row 1: [A, B, C, D]
row 2: [A, B]

If the batch is right-padded:

row 1: [A, B, C, D]
row 2: [A, B, PAD, PAD]

then out[:, -1] is correct for row 1 but wrong for row 2. Row 2’s last real token is at index 1, not index 3.

No-image fallback: row 1 is [A,B,C,D] with last valid column 3; row 2 is [A,B,PAD,PAD] with last valid column 1. The rectangular last column is therefore wrong for row 2. Validate nonempty mask geometry, find the greatest valid column, and separately verify that it owns the vocabulary output for the current pre-append revision.

For a validated, nonempty, contiguous right-padded mask, count-minus-one is a valid shortcut:

lengths = attention_mask.sum(dim=-1)          # [B]
last_idx = lengths - 1                        # [B]
last = hidden[torch.arange(B), last_idx, :]   # [B,d_model]

It is not a general last-valid gather: left padding maps [0,0,1,1,1] to index 2 instead of 4, a noncontiguous admitted mask can do the same, and an all-zero row produces -1. A rectangular implementation that admits either padding side first validates Boolean mask semantics and rejects empty active rows, then selects the greatest valid absolute column:

valid = attention_mask.to(device=hidden.device, dtype=torch.bool) # [B,T]
if (~valid.any(dim=-1)).any():
    raise EmptyInputRow("active row has no output-owner candidate")
columns = torch.arange(valid.size(-1), device=valid.device)
last_idx = torch.where(valid, columns, -1).amax(dim=-1) # [B]
batch = torch.arange(hidden.size(0), device=hidden.device)
last = hidden[batch, last_idx, :]                        # [B,d_model]

Even that gather is valid only when the chosen row is the profile’s semantic output owner and has completed the final-export/head passport. Packed, prefix-decoder, encoder-decoder, compacted, or row-selective runtimes may need an authenticated row map rather than a mask-derived shortcut. The mask used to choose the current decision row is the pre-append revision; the next call’s prospective mask is a distinct post-append object.

Left padding can also make out[:, -1] match the last real token, which is why many decoder-only generation paths prefer left padding. But padding side is not only a data-loader preference. It must agree with:

attention_mask
position_ids
cache_position
last-position gather

If those objects disagree, the tensor shapes can still look normal while the model samples from a PAD position or assigns the wrong position to a real token.

0x08 Prefill and decode

A naive decoder-only loop recomputes the whole prefix after every generated token:

step 1:
    process [A B C] -> generate D

step 2:
    process [A B C D] -> generate E

step 3:
    process [A B C D E] -> generate F

That repeats K/V projection work for the old prefix. Serving implementations split inference into two phases:

prefill:
    process the prompt/context and write its K/V states

decode:
    process the newest generated token and read cached K/V states

No-image fallback: prefill processes a prompt chunk, creates its declared model-state lanes, and produces a first decision row. An ordinary cached decode call later processes one pending selected token while reading retained history, advances every required state lane, and produces the next decision row. Reuse removes historical projection/re-execution under this profile but not the causal dependency between decisions.

The shape difference is the point:

Phase	Input	Current Q	K/V source	Score shape
prefill	[B,L,H]	[B,h,L,d_head]	prompt K/V	[B,h,L,L]
decode	[B,1,H]	[B,h,1,d_head]	cached prior K/V plus current K/V	[B,h,1,past+1]
chunked decode	[B,seqlen,H]	[B,h,seqlen,d_head]	cached prior plus current chunk	[B,h,seqlen,start_pos+seqlen]

One model call prepares a complete logical state candidate

cache.append(token) is too weak a mental model. Every cache-producing layer must compute a K/V row for the same logical input position, use the current row during that layer’s attention, and return a next-call state whose logical frontiers agree. Architectures with recurrent, convolutional, cross, or hybrid state require those lanes too; K/V completeness alone is insufficient.

For generation state $g$, cached-baseline logical frontier $K_g$, and an input chunk $U_g$ of length $\Delta_g$, define a prepared interval at each layer $l$:

$$\widehat C_{l,g} = C_{l,g}\oplus \operatorname{KV}_l(U_g;P_g,A_g),$$

where $\oplus$ means append or write at the declared cache positions. For this bounded contiguous full-K/V profile, the candidate has the expected logical length only if

$$\forall l\in\{1,\ldots,N_{layers}\}:\quad \operatorname{len}(\widehat C_{l,g})=K_g+\Delta_g.$$

Length agreement is necessary, not sufficient. Bind two closed rosters before launch:

compute-stage roster:
    embedding / admitted initial state
    every declared attention + MLP + residual/wrapper stage
    conditional final export
    vocabulary projection and explicit output-owner row

advancing-state-lane roster:
    every self K/V lane required by the next call
    fixed or validated cross state where applicable
    recurrent/convolution/hybrid lanes declared by the architecture profile
    logical positions, visibility revision, slot/generation, and completion handle

The logical transaction is:

PREPARE
    validate token IDs, positions, visibility, generation, and cache ABI
    compute each layer's current Q/K/V
    let current queries read committed history + valid current-chunk K/V
    stage complete K/V rows for [K_g, K_g + Delta_g)

SELECT LOGICAL STATE
    await the declared synchronous/device completion boundary
    validate both closed rosters and every identity/coordinate join
    select ALL_NEW only when every item agrees; otherwise retain ALL_OLD
    expose a consumable score only with ALL_NEW and its output-owner passport

DECIDE
    accept one authenticated M3.8 receipt for that score row
    consume it once into output/policy/RNG/stop/stream state
    do not claim that selected token is already processed model state

ALL_NEW is a logical selection in this article. It does not prove that mutable bytes were restored on failure, that a cache object was durably published, or that concurrent readers cannot see partial physical mutation. Those persistent object, committed-length, fork/truncate/release, durability, and reader-atomicity contracts belong to M5.1; page/block allocation belongs to M5.3. A future call may consume the logical candidate only through the separately implemented persistent-state contract. On failure, retaining ALL_OLD forbids logical publication but does not authorize reuse of a possibly mutated physical handle without a proven recovery operation.

No-image fallback: prefill completes prompt positions [0,L); selecting y0 makes O=1, U=[y0], and P=K=L. The next valid call processes y0, making U=[] and P=K=L+1, then its score can select y1, restoring a one-token pending suffix. Stopping leaves the last token CLOSED_UNPROCESSED unless an explicit repair call processes it.

Output, input, and cache frontiers

Three frontiers move at different points in the loop:

output frontier O:
    selected tokens accepted for the returned sequence

next-input frontier U:
    selected output suffix not yet processed by the model

processed frontier P:
    complete logical input positions represented by every required state lane

cached-baseline frontier K:
    P's K/V realization only in the admitted contiguous full-cache profile

For a decoder-only prompt of length $L$, ordinary one-token decode in that cached profile maintains

$$P+|U|=L+O, \qquad K=P.$$

Immediately after a selection, $|U|=1$ and $P=K=L+O-1$. Immediately after the next valid append call completes and its logical ALL_NEW state is selected, the pending token moves from $U$ into $P$; that same call produces scores for a new selection, so $|U|$ becomes one again.

Boundary	Output tokens $O$	Pending model inputs $U$	Logical processed $P$; cached baseline $K=P$
after prompt prefill, before first selection	0	empty	$L$
after selecting $y_0$	1	$[y_0]$	$L$
after processing $y_0$, before selecting $y_1$	1	empty	$L+1$
after selecting $y_1$	2	$[y_1]$	$L+1$
stop immediately	2	$[y_1]$ or closed-unprocessed output	$L+1$
optional append/repair of $y_1$	2	empty	$L+2$

The optional repair call is not another output decision. Its returned logits predict a token beyond the closed output and are discarded. The output contract must say whether this extra cost is paid and whether final cache is returned at all. This table says nothing about physical or durable cache publication, and it does not apply unchanged to no-cache, sliding, recurrent, hybrid, compacted, or encoder-decoder state profiles.

Candidate state cannot be relabeled

Branch algorithms make the same rule visible. Suppose a temporary call processed candidate token A, but policy or verification commits correction token B:

temporary K/V row = KV(A, position=p)
committed token    = B

Changing metadata from A to B does not change the row. The only valid choices are:

discard KV(A) and process B through a valid append call
or use a verified runtime operation that had actually computed KV(B)

M7 owns how candidates are proposed and accepted. M3.10 owns this input-token to cache-row identity invariant.

Quantifying projection reuse and retained state

Suppose a valid prompt has length $L\ge1$, the selected ordinary decoder-only profile performs prefill, and generation produces $G\ge1$ output tokens. A no-cache loop then processes prefix lengths:

$$L, L+1, \ldots, L+G-1.$$

The number of token rows sent through each layer’s K projection, and separately through its V projection, is therefore:

$$N_{\text{rows,no-cache}} = \sum_{g=0}^{G-1}(L+g) = GL + \frac{G(G-1)}{2}.$$

With the contiguous full-cache baseline, prefill projects the $L$ prompt rows once and later steps project only the generated token fed back at each subsequent call:

$$N_{\text{rows,cache}} = L + G - 1.$$

The -1 is a call-boundary detail under those stated preconditions. The first output is produced by prefill; only the first G-1 generated tokens are fed back to produce the remaining outputs. Immediately after producing output G, the cache therefore contains L+G-1 rows. Feeding that output into one more decode call would append the next row and make the retained length L+G.

G=0 is not covered by either summation range or by L+G-1. It is a declared union: a rejected/no-call request projects zero rows; a prefill-only profile can project $L$ rows and select zero output; another architecture may use a start state with no token-row prefill. Record the branch rather than manufacturing a negative or off-by-one frontier.

For L=2048 and G=128:

no-cache K rows per layer: 270,272
cached K rows per layer:     2,175
projection-row reduction:  124.26x

The same reduction applies to V projection rows. It does not mean decode became constant-cost: each new query still reads a growing key/value history, and that history remains resident.

For N_layers layers, H_kv KV heads, head width d_h, retained length T, and b bytes per stored element:

$$M_{KV}=2N_{\text{layers}}H_{kv}Td_hb.$$

At 32 layers, 8 KV heads, d_h=128, two-byte storage, and T=2175, one request retains 271.875 MiB of K+V payload before allocator metadata or fragmentation. The cache trades repeated projection and old-query work for persistent state and a growing per-step read. M5 later owns admission, paging, and scheduling; M3.10 owns the logical reuse/state requirement. M4 owns physical cost and measurement; M5 owns allocation, persistence, placement, and scheduling.

KV cache does not make future tokens parallel. It removes repeated prefix computation inside each step. The selected token at step t is still an input to step t+1.

If non-text context exists, it enters only through an authenticated architecture profile. X1.2 owns non-text unitization plus encoder/projector/fusion construction, while X1.3 owns mixed-origin visibility, runtime, and cache-state semantics. M2.2 owns admitted initial model-width states; M3.10 neither constructs the non-text path nor assumes that every such state becomes ordinary text-indexed K/V.

0x09 KV cache implementation path

The source articles show two equivalent views of the cache.

The HuggingFace-style conceptual layout is often described as:

past_key_values:
    [num_layers, 2, B, kv_heads, seq_len, d_head]

where the second dimension is Key versus Value. Dynamic cache implementations usually store a list per layer, with each K/V tensor shaped like:

[B, kv_heads, seq_len, d_head]

Static caches reserve the maximum length up front. A per-layer static cache shape is:

[B, num_key_value_heads, max_cache_len, head_dim]

No-image fallback: each cache-producing layer may reserve key and value tensors up to max_cache_len. A call writes only declared logical slots. Attention may read only slots admitted by the current validity/visibility map; zero or stale values in reserved slots are not evidence that they are masked. M5.1/M5.3 own persistent objects and physical allocation.

A StaticCache-style update needs a position object:

cache_position = cache_kwargs.get("cache_position")

k_out = self.key_cache[layer_idx]
v_out = self.value_cache[layer_idx]

if cache_position is None:
    k_out.copy_(key_states)
    v_out.copy_(value_states)
else:
    k_out.index_copy_(2, cache_position, key_states)
    v_out.index_copy_(2, cache_position, value_states)

The LLaMA-style attention code shows the same idea with start_pos:

bsz, seqlen, _ = x.shape
xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)

xq = xq.view(bsz, seqlen, self.n_local_heads, self.head_dim)
xk = xk.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)
xv = xv.view(bsz, seqlen, self.n_local_kv_heads, self.head_dim)

xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)

self.cache_k[:bsz, start_pos : start_pos + seqlen] = xk
self.cache_v[:bsz, start_pos : start_pos + seqlen] = xv

keys = self.cache_k[:bsz, : start_pos + seqlen]
values = self.cache_v[:bsz, : start_pos + seqlen]

In prefill:

start_pos = 0
seqlen = prompt_len
write cache slots [0 : prompt_len]
read cache slots  [0 : prompt_len]

In single-token decode:

start_pos = prompt_len + generated_so_far
seqlen = 1
write cache slot  [start_pos : start_pos + 1]
read cache slots  [0 : start_pos + 1]

After K/V heads are repeated if needed and transposed:

xq = xq.transpose(1, 2)       # [B,n_heads,seqlen,d_head]
keys = keys.transpose(1, 2)   # [B,n_heads,start_pos+seqlen,d_head]

scores = torch.matmul(xq, keys.transpose(2, 3)) / math.sqrt(self.head_dim)

So the score shape is:

$$scores \in \mathbb{R}^{B \times h \times seqlen \times (start\_pos + seqlen)}$$

That shape is the cached version of causal attention. The current query positions attend over previous cache slots plus the current chunk.

Cache/no-cache equivalence test

For committed prefix $x_{0:t}$, an exact cache path and a full recomputation path should produce matching next-token logits within the declared numerical tolerance:

$$\operatorname{LM}_{full}(x_{0:t})[-1] \approx \operatorname{LM}_{cache}(x_t,C_{t-1})[-1].$$

Test every step rather than only final text:

cache = None
for t in range(input_ids.size(1)):
    full = model(
        input_ids[:, : t + 1],
        use_cache=False,
    ).logits[:, -1]

    cached = model(
        input_ids[:, t : t + 1],
        past_key_values=cache,
        use_cache=True,
        cache_position=torch.tensor([t], device=input_ids.device),
    )
    cache = cached.past_key_values

    torch.testing.assert_close(
        cached.logits[:, -1],
        full,
        rtol=rtol,
        atol=atol,
    )

The first cached call may instead pass the entire prompt as one prefill chunk; then compare its last-row logits with full recomputation and begin single-token calls at the prompt length.

Differences can come from kernel tiling and reduced precision, so exact bit identity is not always the contract. Large or growing drift points to position IDs, masks, cache format, rotary application, or stale-slot visibility.

Cache format is part of the model/runtime boundary

Cache entries can store:

rotated keys or unrotated keys
full MHA heads or compact GQA/MQA KV heads
contiguous dynamic tensors or fixed static slots
native dtype or quantized/compressed state
logical token order or paged physical blocks

The consumer must know the format. Applying RoPE again to already rotated keys or repeating GQA heads before storing when the cache expects compact heads can preserve rank while changing values or memory by a large factor.

M5.1 owns persistent request-cache schema, publication, committed-length, fork, truncate, release, and concurrent-reader semantics. M5.2 owns admission and multi-request scheduling; M5.3 owns pages, physical maps, allocation, reference counts, and copy-on-write; M5.4–M5.8 own later retention, placement, offload, and serving representations. M3.10 exports only the single-request logical invariant:

logical slot i names the declared K/V representation of processed input token i
for every required layer under the checkpoint's positional transform;
an explicit adapter maps that slot to any physical address

A cache is an ABI, not an untyped tuple

Two tensors with the same rank are not necessarily interchangeable cache state. Define the one-request cache ABI as

$$\mathcal A_C= (L_c,H_{kv},d_h,\mathcal L_K,\mathcal L_V,D_K,D_V, \rho,P,W,G),$$

where:

L_c:                 cache-producing layer set and order
H_kv, d_h:           compact K/V head count and head width
L_K, L_V:            physical tensor-axis layouts exposed to the model
D_K, D_V:            storage dtypes / representation schemas
rho:                 positional-transform state of stored keys
P:                   logical-position and cache-slot mapping
W:                   retained visibility regime, such as full or sliding
G:                   cache generation / mutation ownership

For RoPE, $\rho$ must say whether keys are stored before or after rotation and which scaling/base configuration produced them. For GQA, $H_{kv}$ must say whether compact K/V heads or query-head-expanded copies are stored. For a quantized cache, $D_K,D_V$ include scales, grouping, and dequantization rules, not merely the word int8.

An append is compatible only when the model call and cache agree on every semantic field required to interpret stored rows:

$$\operatorname{Compatible}(M,U,C) = I_{artifact}\land I_{layer}\land I_{head}\land I_{position} \land I_{visibility}\land I_{representation}\land I_{generation}.$$

The physical address of a buffer can change without violating this contract only if the translation receipt preserves the same logical object. Changing key rotation, head expansion, logical slot mapping, or cache generation cannot.

No-image fallback: map a rectangular input column to a valid semantic row and token position; map that position to the model position ID and visibility relation; map the processed frontier to a logical cache slot and retained window; type each layer's K/V layout, head count, dtype/quantization metadata, rotation state, and generation; then, if persistent storage exists, map the logical slot through an M5.3 page/block/offset revision.

Cache families preserve different contracts

Framework names change over time, so compare the semantic choices rather than assuming that past_key_values identifies one representation:

Family	Allocation/mutation shape	Single-request semantic obligation	Boundary
no cache	recompute full committed prefix	next-row logits are the reference for equivalence	cheapest persistent state, repeated historical work
dynamic full cache	concatenate/grow per-layer K/V	append order and every-layer length remain aligned	shape growth may inhibit compilation or reallocate storage
static full cache	reserve capacity, indexed writes	unwritten/stale slots must remain invisible; overflow fails before overwrite	stable shapes can aid compilation, reserved bytes exceed committed bytes
architecture-defined sliding cache	retain only the declared local history	visibility must match the model’s trained/sliding attention contract	not an exact replacement for arbitrary full attention
offloaded full cache	move retained layers/state across devices	placement changes, logical rows and publication order do not	transfer policy belongs to M5
quantized cache	store K/V plus quantization metadata	dequantized attention stays within a declared quality/numerical contract	quantizer and calibration belong to M8.6

Static storage demonstrates why capacity and commitment differ:

$$0\le K_{committed}\le K_{reserved}\le K_{max}.$$

Attention may read only valid committed slots, even when the backing tensor contains values in every reserved slot. Zero-filled stale slots are not safe by themselves; a finite attention score to an unwritten zero key still changes the softmax denominator.

Dynamic and static full caches should implement the same logical function when they store the same representation. Offload should preserve that function modulo numerical transfer behavior. Sliding and quantized caches need stronger qualification because they can change visible history or represented values.

0x0A Position and cache consistency

RoPE and cache interact because cached K/V states are position-dependent. The RoPE source article shows the serving consequence directly:

freqs_cis = self.freqs_cis[start_pos : start_pos + seqlen]
xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)

The important M3.10 fact is not the RoPE derivation. M3.4 owns that. The inference fact is that Q and K must be rotated with the positions that match the cache slots being written and read.

The following history-plus-current construction is a local explanatory reconstruction:

mask = torch.full((seqlen, seqlen), float("-inf"), device=tokens.device)
mask = torch.triu(mask, diagonal=1)
mask = torch.hstack(
    [torch.zeros((seqlen, start_pos), device=tokens.device), mask]
).type_as(h)

It must not be called byte-identical to the pinned Meta implementation. Keep three evidence lineages separate:

1. this local hstack reconstruction makes the history block explicit;
2. the pinned Meta revision supplies its own square-mask/diagonal and runtime call behavior; and
3. the coordinate proof below independently derives the neutral history and causal current-chunk regions.

Agreement on the admitted example supports the semantic relation, not source identity. The local reconstruction has shape:

[seqlen, start_pos + seqlen]

The zero block corresponds to cached history. The triangular block corresponds to future positions inside the current chunk.

Deriving cached causal visibility from coordinates

Let a current chunk begin at logical position $p$ and contain $c$ rows. Query row $i\in[0,c)$ has global coordinate $q_i=p+i$. The available key coordinates are $k_j=j$ for $j\in[0,p+c)$. Ignoring padding and architectural windows for one line, causal visibility is

$$M_{i,j} = \begin{cases} 0,&j\le p+i,\\ -\infty,&j>p+i. \end{cases}$$

This yields a zero history block of width $p$ and a lower-triangular current block of width $c$. For $p=3,c=2$:

             keys: 0  1  2 | 3  4
query pos 3:       0  0  0 | 0 -inf
query pos 4:       0  0  0 | 0   0

Padding validity, segment boundaries, prefix visibility, or a trained sliding window intersect with this causal relation. They do not replace the global coordinate derivation.

This is why these objects must agree:

visible token sequence
attention_mask
position_ids
cache_position
start_pos
RoPE frequency slice
cache write slice
cache read slice

If cache_position says the token belongs at slot 128 but position_ids or freqs_cis rotate it as position 0, and no explicit translation maps slot 128 to position 0 (a static cache that stores 128 left-padding columns declares exactly that map), the string may look correct while the attention geometry is wrong.

Position fields have related but distinct meanings

position_ids:
    coordinates passed into the model's positional method

cache_position:
    logical slots written/read in a cache API

physical cache address:
    allocator or page location holding that logical slot

sequence length:
    count of committed visible tokens under the request contract

In a simple contiguous cache they may all use the same integer. They diverge under left padding, packed segments, sliding windows, prefix reuse, or paged allocation.

Do not join those spaces by integer coincidence. Each prepared call records an explicit translation receipt:

request identity + sequence/branch identity + generation
rectangular column -> valid semantic row -> semantic token position
semantic position -> model position ID
semantic processed frontier -> logical cache slot + retained window
logical slot -> physical page/block/offset under an M5.3 map revision
input row -> requested output-owner row -> vocabulary-score receipt
pre-call visibility revision -> post-call prospective revision

The inverse or comparison rule must be defined wherever replay, reorder, crop, resume, or equivalence needs it. A matching value such as 17 in two fields is not evidence that they share a coordinate system.

For one left-padded row:

rectangular columns:  0    1    2    3    4
tokens:              PAD  PAD   A    B    C
valid mask:           0    0    1    1    1
position_ids:         0    0    0    1    2   # common valid-token rule
cache slots:          -    -    0    1    2   # if padding is not cached

For a static API that stores rectangular columns, cache slots could instead be 0..4 while only 2..4 are valid. Both choices can be correct. Mixing the position-ID rule from the first with a mask/cache-length rule from the second without an explicit translation is not.

For left-padded prompts, position IDs are commonly derived from valid-token cumulative count rather than rectangular columns:

position_ids = attention_mask.long().cumsum(-1) - 1
position_ids.masked_fill_(attention_mask == 0, 0)

For

[PAD, PAD, A, B, C]

the real tokens receive positions [0,1,2], not [2,3,4], if that matches training. The static cache can still store them in rectangular slots or compact valid-token slots; cache_position must describe which.

ALiBi does not rotate cache entries, but its bias still depends on the query/key coordinate difference. Reused K/V therefore need correct logical key positions. Learned absolute embeddings likewise require the current position ID even though no transform is baked into K/V.

Chunked input equivalence

A prompt can be prefetched in chunks if each chunk’s queries see all previous committed keys and the appropriate causal region inside the chunk. For chunk beginning at $p$ with length $c$:

query positions: p ... p+c-1
key positions:   0 ... p+c-1
mask shape:      [c, p+c]

Chunked prefill should match one-shot prefill last-row logits and cache contents within tolerance. Chunk boundaries must not reset positions, normalization state, or visibility unless the model was trained with such resets.

Logical recovery sockets and branch state

Ordinary single-path decode logically commits one decision at a time. Constraint repair, beam branching, speculative verification, or interrupted structured generation may create temporary state. M3.10 can require the following logical roots and transitions:

accepted logical root:
    complete processed prefix eligible for future accepted work

temporary logical branch:
    candidate state rows not yet selected into the accepted prefix

select ALL_NEW:
    choose only complete state corresponding to actual accepted input tokens

retain ALL_OLD:
    forbid logical exposure of the rejected candidate to future work

If a correction token differs from a rejected candidate, the rejected candidate’s K/V row cannot be relabeled as the correction. The correction must pass through a valid model append path before future decode depends on its cache state.

Selecting ALL_OLD does not prove that an in-place writer erased bytes, released pages, restored allocator metadata, fenced concurrent readers, or durably repointed a cache object. M5.1/M5.3 own those physical and persistent mechanisms. Until their recovery receipt succeeds, a possibly mutated handle is quarantined rather than silently reused.

Numerical and runtime execution contract

model.eval(), disabled gradient recording, reduced precision, deterministic policy, and cache use are independent controls:

Control	Changes	Does not establish
model.eval()	module training flags, disabling dropout and using evaluation behavior for affected modules	disabled autograd, cache use, dtype, deterministic kernels, or decoding policy
torch.no_grad()	reverse-mode gradient recording in its scope	evaluation module behavior or the stronger inference-tensor restrictions
torch.inference_mode()	disables gradient recording plus additional autograd tracking overhead in its scope	model.eval(), deterministic algorithms, or a quality-equivalent dtype
autocast / model dtype	operation and storage precision choices	stable token identity near tied logits
use_cache=True	requests model-specific retained state	a particular cache family, physical placement, or exact equality with no-cache kernels
greedy policy	deterministic selection for fixed score values and tie rule	reproducibility if model scores, kernels, vocabulary order, or tie handling differ

A minimal local wrapper therefore declares both module and autograd state:

model.eval()

with torch.inference_mode(), torch.autocast(
    device_type="cuda",
    dtype=torch.bfloat16,
    enabled=manifest.runtime.autocast,
):
    result = generate_one(...)

The exact context differs by backend. The contract records independent facts: module training/evaluation flags; autograd or inference-mode scope; every functional/custom stochastic path and RNG address; deterministic-algorithm and kernel policy; and compute/logit/cache dtypes. model.eval() changes modules that implement a training/evaluation branch, but it does not disable arbitrary functional randomness or prove deterministic policy, kernels, replay, or distributed execution. A no-grad/inference scope prevents graph recording only within its documented domain; it does not establish module evaluation state.

No-image fallback: bind evaluation/training flags separately from no-grad or inference mode; inventory custom RNG/stochastic paths; bind compute, logit, and every state-lane dtype; bind kernel, compilation, device, topology, and completion-event revisions; require closed compute and state-lane rosters; then compare STATE, TENSOR, POLICY, DECISION, SEQUENCE, and RESUME in order, including the claimed L-infinity precondition before any margin argument.

A logit tolerance is not a token-equality proof

Suppose reference logits $z$ and candidate logits $\tilde z$ satisfy

$$\lVert z-\tilde z\rVert_\infty\le\epsilon.$$

Let $z_{(1)}$ and $z_{(2)}$ be the largest and second-largest reference logits. Greedy selection is guaranteed unchanged only if

$$z_{(1)}-z_{(2)}>2\epsilon.$$

Why: the winner can decrease by $\epsilon$ while the runner-up increases by $\epsilon$. If the margin is no larger than $2\epsilon$, a candidate can pass the tensor tolerance and still flip argmax.

The implication is conditional on actually checking the $L_\infty$ premise. Matching argmax plus a wide reference margin does not imply that the candidate is close. For example, reference [10,0], candidate [100,0], IDs [10,11], and $\epsilon=1$ keep the same winner and satisfy the historical reference margin test, but $\lVert z-\tilde z\rVert_\infty=90$. A valid greedy comparator orders its checks as follows:

same bound request, score kind, vocabulary map, shape, and finite-value policy
compute and require L-infinity error <= finite epsilon, epsilon >= 0
apply the declared tie rule to both rows
if claiming tolerance-guaranteed stability, require reference margin > 2 epsilon
then compare the selected IDs and downstream decision receipt

For stochastic sampling, equal seed is even weaker. Any changed probability row, processor order, multinomial implementation, device generator, or draw count can map the same nominal seed to a different token. Validate at several levels:

state equivalence: tokens, positions, masks, cache frontier, RNG counter
tensor equivalence: selected hidden/logit/cache probes within dtype tolerance
policy equivalence: processed support and probability rows within contract
decision equivalence: greedy IDs or replayed sampled IDs where promised
sequence equivalence: output tokens, text spans, finish reason, metadata
resume equivalence: exported/pending state continues like uninterrupted state

These six layers are ordered: STATE, TENSOR, POLICY, DECISION, SEQUENCE, and RESUME. A later pass is noncurative for an earlier failure. Bit equality can be a useful same-stack gate, but it is not always available across kernels or hardware. Conversely, matching final text alone is too weak to validate intermediate cache and score state, even when the final tokens are identical.

Compilation makes shape and mutation assumptions executable

Static cache shapes can reduce recompilation, but compilation does not infer semantic cache length from nonzero values. The compiled graph still needs:

current input length or valid row count
cache_position / write interval
attention validity over reserved slots
cache generation or mutation ownership
model-specific output/cache structure

Warmup should cover prefill and decode shapes separately. A graph captured for [B,L] prefill is not automatically the graph for [B,1] decode. Dynamic shapes, static capacity buckets, and separate phase graphs are design choices. If a cache buffer is mutated inside a captured graph, reusing it for another request or beam without resetting generation and validity metadata can expose stale rows while every address remains legal.

One request can still be model-parallel

single request does not imply one device. Tensor, pipeline, or expert parallel execution can distribute one model call:

tensor parallel:
    ranks own shards of projections or vocabulary scores;
    collectives reconstruct the next layer or global policy result

pipeline parallel:
    stages own disjoint layer ranges and their corresponding cache state;
    all stages must report completion for the same logical processed-input frontier

expert parallel:
    routed token rows cross expert owners inside the model call;
    M6.2 owns balance/interaction requirements and M6.6 owns physical
    placement, dispatch, topology, precision, communication, and expert runtime

For a fixed contiguous-K/V profile on $R$ declared participants, a necessary logical agreement is

$$K_g^{(0)}=K_g^{(1)}=\cdots=K_g^{(R-1)}$$

under the shared logical request generation, even if each rank stores only its local layer/head state. It is not sufficient: every architecture-qualified state lane and compute stage must complete, and the closed roster must agree on request, generation, position, visibility, output owner, and completion event. A missing participant or lane selects ALL_OLD and supplies no consumable score. How the globally selected token, beam map, finish state, or RNG receipt is communicated is a later runtime mechanism; M3.10 exports the consistency requirement without owning M6.2/M6.6 communication or topology.

One rank exiting while peers enter another collective can hang the request. M3.10 therefore requires a declared participation/completion contract, but does not prescribe the collective, kernel, precision, dispatch, or topology. Those are M4.2+/M6.6 runtime concerns, distinct from M5 multi-request scheduling.

Asynchronous completion and error boundaries

Accelerator launches can return control before kernels finish. A host-side cache pointer or logits tensor being available does not prove the model call completed successfully. Logical candidate selection, timing, and error checks need the declared device event/stream boundary. Persistent publication is a separate M5.1 event. In particular:

TTFT/step timers synchronize or use device events under a stated convention
streaming cannot expose a token before its score/metadata transaction is valid
an asynchronous device fault invalidates staged cache/output state
retry starts from the last proven committed generation, not a reused buffer

M4 owns kernel measurements and M5 owns service clocks. M3.10 only requires a model-call completion boundary before selecting the new logical one-request state.

Pinned official runtime reconciliation

The preceding state machine is an accounting contract, not a transcription of one framework. Real implementations divide ownership differently. The table below was checked against local, hashed snapshots; names and line layouts are revision-specific. The reconciliation is a static requirements synthesis, not runtime or durability evidence.

Pinned implementation	Static source trace	What the observed source supports	What it does not prove
Annotated Transformer debc9fd	encode source once, repeatedly decode the complete target prefix, select the last row	the simplest encoder-decoder recurrence and fixed source-memory lifetime	cached decoder execution, general stopping, streaming, or production search
Transformers generation 4626421	prepare_inputs_for_generation slices the next model-input suffix, forwards cache and aligned position/mask state; the loop updates model kwargs, takes the last logits row in float32, runs processors, selects, appends, streams, then evaluates row-wise stopping criteria	a high-level controller must advance tokens, masks, positions, cache, policy state, and finished rows together	one universal cache layout or identical behavior across model overrides and library revisions
Transformers cache/stop utilities 4626421	cache objects expose update, logical length, reorder, and crop contracts; stop criteria return a Boolean decision per row, and stop-string matching examines token overlaps at the generated suffix	cache and stop state are typed controller objects, not incidental tensors	that every cache family has the same allocation or mutation policy
Meta Llama reference 6c7fe27	a rectangular token buffer preserves forced prompt slots; prev_pos:cur_pos is forwarded, layer caches write at start_pos, and only rows leaving the prompt become policy decisions	explicit prompt/decode frontiers and a static per-layer cache realization	the complete request, streaming, beam, or rollback contract of a serving engine
llama.cpp 6ea215d	the caller submits a llama_batch carrying token or embedding rows, positions, sequence ownership, and a logits-selection flag; llama_decode mutates context state, selected logits rows are read explicitly, and a sampler chain owns policy history	low-level runtimes can expose model rows, sequence identity, persistent state, and policy state as separate caller-managed objects	that llama_get_logits(ctx) means one row, or that sampling and cache mutation form one implicit operation
PyTorch grad-mode source 9dc95fe	inference_mode is thread-local, disables reverse-mode recording and additional view/version tracking, and explicitly does not call model.eval()	module mode and autograd mode are independent manifest fields	deterministic kernels, cache use, or policy reproducibility

These traces reconcile to one implementation-neutral call boundary:

prepare:
    derive the exact input suffix, positions, visibility, sequence IDs,
    cache generation, and requested score rows

execute:
    run every owned model layer and stage complete next-state rows

select logical result:
    expose a consumable score and ALL_NEW state only after both rosters and the
    runtime completion boundary pass; persistent publication remains M5.1

decide:
    apply the M3.8 policy, append the output token, update stop/stream/RNG state,
    and leave that selected token pending until a later valid model input call

The last line is easy to lose when reading framework code. In the pinned Transformers loop, the selected ID is appended to the growing input_ids after the model call. The cache returned by that call represents the input rows just processed, not the selected output row that did not yet enter the model. A library may repair or materialize that final pending row before exporting a resumable state, but it must say so; output length alone cannot establish cache length.

Framework names are observations, invariants are the contract

Map runtime fields semantically before comparing them:

M3.10 semantic object	Transformers observation	Meta Llama observation	llama.cpp observation
model-input suffix	sliced input_ids / decoder_input_ids	tokens[:, prev_pos:cur_pos]	rows in llama_batch
logical coordinates	position/mask inputs and cache logical length	start_pos plus slice offsets	pos, seq_id, and runtime context state
requested decision rows	final model output row in ordinary generation	last row returned by the forwarded slice	batch.logits[i] controls stored output rows
retained model state	model-specific Cache	per-layer preallocated K/V arrays	context/sequence state APIs
branch remap	cache reorder_cache plus controller gathers	not owned by the simple reference loop	explicit sequence/state operations at the low-level API
policy state	generation config, processors, criteria, RNG	top-p/temperature or argmax in controller	sampler chain plus accepted-token history

No column is the canonical vocabulary. A compatibility adapter is correct only when it preserves semantic ownership. Copying a field with a familiar name is insufficient if, for example, one runtime’s length is allocated capacity and another’s is committed sequence length.

The static evidence bundle records each source snapshot by SHA-256 and its reconciliation markers. Those stored reports show only that an author-side checker observed the declared bytes and markers. They do not prove that a tokenizer, model, framework, cache, policy, RNG, stream, or production request executed. A replacement invalidates that source observation; runtime behavior requires separate execution evidence against an admitted physical artifact.

0x0B Shape-correct failure lab

Most inference bugs preserve tensor shapes. The generated text can look plausible while the loop is sampling from the wrong row, reading stale cache, or expanding a finished sequence.

Last-position bugs

right padding plus out[:, -1]
    shape survives:
        [B,d_model]
    semantic break:
        shorter rows sample from a PAD hidden state

last-valid gather with the previous call's attention_mask
    shape survives:
        [B,d_model]
    semantic break:
        the selected row belongs to the previous prefix length

beam expansion without row remapping
    shape survives:
        [B*beam,d_model]
    semantic break:
        scores attach to the wrong hypothesis

Score and selection bugs

argmax over time instead of vocabulary
    shape may survive:
        [B,T,V] -> wrong dimension selection
    semantic break:
        selected index is a time position, not a vocab ID

multinomial over logits
    shape survives:
        [B,V]
    semantic break:
        multinomial expects non-negative weights/probabilities

softmax over exact log-softmax values (finite real logits)
    shape survives:
        [B,V]
    exact result:
        softmax(log_softmax(z)) = softmax(z); the operation is redundant
    finite-precision caveat:
        the extra exponentiation/reduction may round differently

passing log-probs directly as multinomial probability weights
    shape survives:
        [B,V]
    semantic break:
        log-probs are generally negative and are not non-negative weights

normalizing on the wrong axis or under the wrong score-kind/token-map passport
    shape survives:
        [B,V] or another rank-compatible tensor
    semantic break:
        the resulting row is not the declared vocabulary distribution

top-p over logits
    shape survives:
        sorted [B,V]
    semantic break:
        cumulative sums over logits are not probability mass

top-p without gather back through sorted indices
    shape survives:
        [B,1]
    semantic break:
        sorted-space index is treated as the vocabulary ID

Stop and active-row bugs

finished rows keep expanding after EOS
    shape survives:
        rectangular batch continues
    semantic break:
        tokens appear after logical completion

stop string split across tokens
    shape survives:
        token-level loop runs
    semantic break:
        text-level stop is missed or triggered too early

stop token removed from tokens but not logprobs
    shape may survive separately:
        token list and logprob list
    semantic break:
        metadata no longer aligns with returned text

Beam bookkeeping bugs

parent beam IDs not gathered
    shape survives:
        [B,beam,V] candidate scores
    semantic break:
        new tokens attach to the wrong prefix

completed beams expanded again
    shape survives:
        active beam table remains full
    semantic break:
        hypotheses continue after EOS

KV cache not reordered with beams
    shape survives:
        cache still has [B*beam,...]
    semantic break:
        token prefix and cached prefix are different hypotheses

Cache and position bugs

cache_position reset after prefill
    shape survives:
        static slots exist
    semantic break:
        generated K/V overwrite prompt K/V

write [start_pos:start_pos+seqlen] but read only [:seqlen]
    shape may survive:
        current-token attention works
    semantic break:
        cached history is invisible

read stale static-cache slots
    shape survives:
        max_cache_len is allocated
    semantic break:
        unfilled positions become visible keys/values

cache_position and position_ids violate the explicit translation
    shape survives:
        positions and slots are both valid tensors
    semantic break:
        token is rotated for one position but stored in a slot that translates to another

RoPE applied to Q but not K before cache write
    shape survives:
        K has the expected dimensions
    semantic break:
        cached keys are in the wrong coordinate system

Encoder-decoder memory bugs

recomputing memory every target step
    output may remain correct
    semantic break:
        not wrong mathematically, but it wastes encoder work

mutating memory inside the decoder loop
    shape survives:
        [B,S,d_model]
    semantic break:
        cross-attention no longer reads stable source states

dropping src_mask in inference
    shape survives:
        memory is still [B,S,d_model]
    semantic break:
        source PAD positions become visible

Runtime-mode bugs

running inference with model.train()
    shape survives:
        same tensors
    semantic break:
        dropout and training-mode behavior remain active

serving while autograd recording is enabled
    output may remain correct
    conditional risk:
        a graph can be recorded; it is retained only while references survive,
        and any memory/latency effect depends on the workload and must be measured

assuming KV cache parallelizes future tokens
    shape language sounds plausible
    semantic break:
        cache removes repeated prefix computation, not the dependency on selected previous tokens

Commit and streaming bugs

compute path reaches vocabulary logits but one advancing state lane is missing
    score shape survives:
        [B,V]
    semantic break:
        score is not consumable and ALL_NEW is forbidden; retain ALL_OLD

pre-append decision mask reused as the next call's post-append mask
    shape may survive:
        one extra reserved column already exists
    semantic break:
        row ownership or visibility belongs to the wrong revision

committing selected final-token K/V without processing that token
    output text looks correct
    semantic break:
        cache length or final cache row claims state that was never computed

relabeling rejected candidate K/V as a correction token
    shape survives:
        one row is still present
    semantic break:
        row values belong to another token and future logits condition on it

retaining the old logical root after an in-place write and calling it rollback
    logical metadata looks old:
        generation/frontier points to the prior state
    physical break:
        mutated bytes or aliases may remain; quarantine until M5 recovery proves reuse

emitting bytes before stop matching has a safe suffix
    low-latency stream works usually
    semantic break:
        matched stop text may already be irreversibly exposed

trimming returned text without trimming token/logprob offsets
    arrays remain individually valid
    semantic break:
        metadata no longer describes the returned string

Cache equivalence bugs

cache test compares only sampled final text
    semantic break:
        policy can hide significant intermediate logit drift

one-shot and chunked prefill use different position resets
    shape survives:
        every chunk has valid positions
    semantic break:
        chunk boundary changes model geometry

static cache exposes unfilled slots
    shape survives:
        max length is allocated
    semantic break:
        attention reads stale or zero entries as real history

GQA cache expands KV heads before storage unexpectedly
    model may still run
    semantic break:
        memory contract and downstream head-repeat logic disagree

RNG and policy-state bugs

one global RNG shared by unrelated requests
    local sampling is valid
    semantic break:
        request output depends on interleaving with other requests

beam parent reorder omits constraint or RNG state
    prefix/cache may align
    semantic break:
        child policy state belongs to another parent

generated token logprob read from next-step logits
    vocabulary shape matches
    semantic break:
        score refers to the following token distribution

The practical debugging question is:

Which row produced the score, which token was selected, which row was appended,
which cache slot was written, and which positions can the next query read?

If those five answers line up, the inference loop is probably conditioning on the prefix you think it is conditioning on.

---

0x0BA Requirements for observability and later-owner release

This section exports schemas and gates; it does not report a model run, physical artifact, canary, promotion, rollback, or release. Before a separately authorized harness evaluates an admitted artifact, it freezes an evaluation manifest. Otherwise a passing rerun may have changed the tokenizer, template, policy, runtime, or stopping contract rather than validating the candidate.

evaluation identity:
    artifact/checkpoint hash
    tokenizer files + special-token map hash
    prompt-template and normalization hash
    exact token IDs after templating
    position/visibility/cache ABI
    policy order, parameters, tie rule, and RNG identity
    stop/stream/output contract
    runtime revision, kernels, dtype, compile flags, and hardware
    fixture corpus hash and expected-result schema

An authorized harness would record only the bounded, classified fields needed for each transition—not automatically every raw value:

request digest and redacted/omitted prompt fields under the trace privacy passport
position IDs, cache positions, and attention-mask lengths
prefill last-row logits checksum and top candidates
selected token ID, logprob, policy configuration, RNG counter
cache logical length before and after each model call
finish reason and exact stopping match
incremental text chunks with token/byte offsets
device completion event, peak allocated memory, and per-phase timings

This trace can be large and sensitive. Its admission schema fixes maximum events/bytes/probes and the classification, redaction, access, retention, and deletion policy described earlier. Full tensors or decoded content are captured only with explicit authority; otherwise retain content-addressed identities, bounded summaries, and counters. Hash equality is a useful same-stack debugging signal, not a cross-hardware numerical theorem and not proof that hidden content is safe to store.

No-image fallback: M3.10 exports a candidate identity plus ordered STATE, TENSOR, POLICY, DECISION, SEQUENCE, and RESUME requirements. M8.9 must convert and admit the exact runnable artifact. M8.10 must define offline gates, bounded canary exposure/abort criteria, monitoring, and a fenced durable idempotent promotion. M8.10 with M5 must prove rollback and in-flight/cache compatibility; reversing a pointer alone is not physical state recovery.

Correctness matrix

A separately authorized evaluation owner can run the same admitted artifact under:

Case	Compare
no-cache versus dynamic cache	logits at every step
dynamic versus static cache	logits, token IDs, cache lengths
one-shot versus chunked prefill	last-row logits and cache tensors/checksums
batch size one versus padded batch	per-request logits and token path
left-padded versus unpadded equivalent prompt	valid-token logits under declared position policy
eager versus optimized attention	logits under tolerance
greedy reference versus beam width one	tokens/scores when processor and stopping contracts align
fixed seed replay	token path and RNG counter under same stack

The comparison ladder should stop at the first failed layer:

1. state:       tokens, valid lengths, positions, masks, cache frontier
2. tensor:      selected hidden/logit/cache probes within declared tolerance
3. policy:      processed support, probabilities, ordering, RNG draw count
4. decision:    selected IDs, parent beams, aligned token metadata
5. sequence:    output IDs/text spans, finish reason, final stream flush
6. resume:      next call from exported state matches uninterrupted execution

Final-text equality cannot waive a state or tensor failure. Conversely, a small tensor difference need not be a release failure if the declared downstream decision/quality contract allows it; the release rule must say which level is normative.

Boundary cases

Include:

one-token prompt
prompt at maximum allowed length
immediate EOS
zero allowed new tokens
stop string spanning tokens
multiple stop patterns with shared prefixes
non-ASCII/byte-fallback text
right- and left-padded prompts
beam parent duplication and early EOS
structured output completing at a token suffix

Add adversarial state transitions, not only unusual prompt strings:

model call fails before complete ALL_NEW logical-state selection
stream cancellation arrives while a device call is in flight
static cache is reused with a shorter request
beam parent is duplicated while another parent completes
final output token is selected but never processed as an input
resume occurs with one pending output token ahead of the cache frontier

Latency semantics

For one request, distinguish:

time to first token:
    prompt processing + first policy decision + safe stream emission

inter-token latency:
    selected token append + one decode call + policy + emission

total generation time:
    through finish detection and final stream flush

The final selected token may not require another model call, so dividing total time by output-token count can hide the prefill/decode boundary and off-by-one call count. M5 generalizes these metrics across queues and batches.

For an isolated single request, define completed device-event times:

$$\operatorname{TTFT}_{isolated} = t_{first\ safe\ emission}-t_{admit},$$ $$\operatorname{ITL}_j = t_{emit,j}-t_{emit,j-1}, \qquad \operatorname{E2E}=t_{final\ flush}-t_{admit}.$$

Report warmup, synchronization, repetition count, median, and tail quantiles. Do not mix host enqueue time with synchronized device completion. These are single-request execution measurements. Queue delay, continuous batching, fairness, and service goodput remain M5 ownership.

Sensitivity is a controlled experiment

One aggregate measurement cannot explain an inference implementation. Sweep one variable while preserving the immutable manifest elsewhere:

Variable	State expected to change	State that must remain fixed	Main observation
prompt length	prefill rows and cache footprint	output policy and requested new-token count	prefill time, first-row score, peak state
output length	decode-call count and final frontier	prompt tokens	per-step drift, ITL distribution, stop behavior
cache family	allocation/mutation realization	logical positions and visible prefix	tensor equivalence, memory, compile count
input chunk size	number of prefill calls	token/position/visibility sequence	last-row/cache equivalence and call overhead
compute/cache dtype	numerical error and bytes	model semantics and policy	probe error, margin crossings, token divergence
beam width	branch count and coupled state	score definition and stop proof	branch correctness and retained-state growth
stop patterns	matcher and stream holdback	model scores and RNG draws before forced closure	exact text/token spans and finish precedence

If two variables move together, call the result a configuration comparison, not a sensitivity curve. M4 owns physical cost interpretation, M5 service-load experiments, M8.6 owns live cache-quantization evaluation, M8.9 owns artifact admission, and M8.10 owns release evaluation.

A completed output is not automatically a resumable snapshot

Normal generation may stop immediately after selecting token $u_O$. At that point the output includes $u_O$, while the cache can still end at the preceding processed input. That is a valid terminal result. A resumable snapshot must choose one explicit contract:

pending-suffix snapshot requirement:
    bind the accepted logical P or cached-baseline K frontier
    serialize pending model-input suffix U, often [u_O]
    preserve P + len(U) = prompt_length + output_count in the admitted profile

repaired-state snapshot requirement:
    run a valid model append for U without selecting another output
    select its complete all-lane logical state
    serialize U = [] and P = prompt_length + output_count

Never copy the rejected/candidate row, increment a length field, or relabel a cache slot to manufacture the repaired form. Snapshot state also includes positions, visibility, tokenizer decoder bytes, stop/parser state, policy history, RNG counter, beam lineage, and output/token/logprob alignment as applicable.

Resume equivalence is a testable later-owner requirement:

$$\operatorname{Continue}(\operatorname{Restore}(S_g),n) \equiv \operatorname{Continue}(S_g,n)$$

at every declared comparison level. A future M5/M8 implementation must test both snapshot forms if its API supports both; this article does not claim that serialization, restoration, or continuation executed.

Later-owner release, canary, and rollback requirements

M3.10 stops at a candidate requirement passport. M8.9 owns physical artifact conversion plus loader/kernel/runtime admission; M8.10 owns deployment, evaluation, canary, monitoring, promotion, rollback, and release; M5 owns in-flight service/cache lifecycle. Under those later owners, a release candidate needs a closed compatibility bundle, not only a model file:

candidate bundle:
    artifact + tokenizer + template + special-token IDs
    position/visibility/cache ABI and context limits
    policy/stop/stream/output defaults
    runtime revision, kernels, dtype, and compile artifacts
    conformance corpus, expected results, tolerance policy, and reports
    migration and rollback compatibility

The following is a requirements handoff to M8.10, not an operation performed or authorized here:

M8.9: CONVERT and ADMIT a physical artifact under a content-addressed ID
M8.10: VERIFY declared offline gates against that exact admitted artifact
M8.10: CANARY under a bounded traffic, privacy, exposure, and abort passport
M8.10: PROMOTE with a fenced, durable, idempotent publication transaction
M8.10/M5: ROLL BACK deployment and handle in-flight/cache state only under
           tested compatibility; pointer reversal alone is not state rollback

Any future canary comparison needs semantic counters: finish-reason mix, invalid-output rate, stop leakage, token/sequence divergence on shadow fixtures, cache faults, and latency/memory envelopes. Aggregate text quality alone will not expose a position or cache ABI regression.

Future rollback also has a state-compatibility question. In-flight requests can finish on their admitted runtime generation, be drained, or migrate only if a tested snapshot adapter proves compatibility. Pointing an old binary at a new cache object because tensor shapes match is not rollback.

Failure closure by owned contract

Contract	Requirement exported to its later owner	Blocking example
request identity	content hashes and canonical manifest	tokenizer/template changed under same model name
source-to-tensor	token, mask, position, decision-row trace	right padding selects a PAD row
cache frontier/ABI	no-cache/cache/chunk/resume equivalence	selected final token falsely counted as processed K/V
stop/stream	token/byte spans and commit-horizon fixtures	stop bytes streamed before match resolution
branch state	coupled reorder/fork oracle	cache or RNG omitted from beam gather
architecture	decoder/prefix/encoder-decoder conformance	source memory mutated during target decode
numerical/runtime	layered tolerance and completion events	tolerance passes but near-tie argmax flips
physical artifact admission (M8.9)	conversion, exact loader/kernel compatibility, admission receipt	a source/checkpoint identity is mistaken for a runnable artifact
release lifecycle (M8.10/M5)	immutable bundle, bounded canary evidence, fenced durable publication, compatible in-flight/cache rollback	candidate overwrites the last accepted artifacts or pointer rollback leaves incompatible state

The author-side lifecycle fixture is static construction evidence over synthetic manifests. Its stored report checks its own pending-suffix, repaired-snapshot, and pointer-transition rules while rejecting selected fixture defects. It did not execute a model, admit an artifact, touch live traffic, publish a durable pointer, validate concurrent readers, or perform production rollback.

0x0C Summary and handoff

Inference starts where training loses its labels. There is no tgt_y; there is only the current prefix. The loop is:

context / prefix
    -> bound model call completes every compute stage and advancing state lane
        -> explicit semantic output-owner row
            -> M3.8 vocabulary-score passport and pending decision receipt
                -> M3.10 consumes the receipt exactly once
                    -> append logical output and aligned metadata
                        -> update policy/RNG/parser/stop/stream state
                            -> tag token PENDING_NEXT_CALL or CLOSED_UNPROCESSED
                                -> only a later valid call processes pending input
                                    -> select complete ALL_NEW logical state or ALL_OLD

separate later-owner transitions:
    M5.1/M5.3 persist or physically recover cache/request state
    M8.9 admits a runnable artifact
    M8.10 deploys, canaries, promotes, monitors, rolls back, or releases

For encoder-decoder models, the source memory can remain fixed while the target prefix ys grows. For decoder-only models, the prompt and generated answer are one causal stream. In both cases, the next token comes from the last valid visible position, not from a full known target sequence.

Prefill, decode, and K/V reuse are execution profiles around that dependency. They avoid re-projecting historical rows under the stated profile, but do not let the model know future selected tokens in advance. Physical savings and measurements belong to M4; persistent cache and scheduling belong to M5.

M3.10 owns the logical contract for the repeated inference loop. The local score interface inside that loop has a separate owner:

final hidden row -> vocabulary scores -> selected token or beam candidates:
    M3.8

authenticated decision -> logical output/stop/stream/pending-state transition:
    M3.10 (requirements and exactly-once consumption)

persistent cache publication/lifecycle and pages:
    M5.1 / M5.3

Use M3.8 — Output Head and Decoding Policy for the learned projection and authenticated local token/beam decision. X1.2 owns non-text unitization and encoder/projector/fusion construction; X1.3 owns mixed-origin runtime state. M3.4 owns RoPE’s derivation; M4 owns physical cost and measurement; M5 owns persistent cache, paging, multi-request scheduling, and service lifecycle; M6.2/M6.6 own expert-runtime balance and execution; M7 owns speculative/multi-token execution; M8.6 owns live cache quantization; M8.9 owns physical artifact/runtime admission; and M8.10 owns deployment and release.

0xFF Reference

Source roles and counts are intentionally typed. The historical author manifest contains 15 rows; the corrected substantive primary/official inventory contains 16 records after including Best-First Beam Search. The local pinned-source directory contains 10 members: nine code snapshots plus one metadata README. That directory-member count is not “ten executable sources,” and marker/hash checks remain static evidence. The Zhihu page below returned HTTP 403 during the research audit and is retained only as an unverified historical locator.

• Attention Is All You Need
• The Annotated Transformer
• The Annotated Transformer pinned source (debc9fd)
• Sequence to Sequence Learning with Neural Networks
• Neural Machine Translation by Jointly Learning to Align and Translate
• Google’s Neural Machine Translation System
• Optimal Beam Search for Machine Translation
• Diverse Beam Search
• Grid Beam Search
• Best-First Beam Search
• A Contrastive Framework for Neural Text Generation
• Transformers generation/cache/stop source (4626421)
• Transformers cache documentation
• Meta Llama reference source (6c7fe27)
• llama.cpp source (6ea215d)
• PyTorch grad-mode source (9dc95fe)
• PyTorch inference_mode documentation
• A Survey on Efficient Inference for Large Language Models
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• LLM Inference Unveiled: Survey and Roofline Model Insights
• Fast Inference from Transformers via Speculative Decoding
• Breaking the Sequential Dependency of LLM Inference Using Lookahead Decoding
• LLM inference series 1: introduction
• LLM inference series 2: the two-phase process behind LLM responses
• Several Parallel Mechanisms of LLM — historical locator; not externally verified in the research audit
• Understanding How LLM Inference Works with llama.cpp
• Simon J. D. Prince, Understanding Deep Learning official book site — see the publisher-authorized Chapter 12 materials; no root-local PDF is represented as reader-distributed
• Transformer Series (3) — Data Processing
• Transformer Series (4) — Encoder & Decoder
• Transformer Series (5) — Training & Reasoning
• Transformer Series (15) — Sampling and Output
• Transformer Series (17) — RoPE
• Transformer Series (20) — KV Cache