X1.3 - Multimodal Masks, Training, Runtime, and State

0x00 The visible sequence is not the supervised sequence

After X1.1 and X1.2, a system can produce language-width media states. The remaining work is easy to hide in one arrow:

image + prompt -> model -> answer

At implementation level, that arrow contains several different state machines. A processor binds media objects to placeholders. Placeholder tokens may expand to many feature rows. Packing assigns row ranges and positions. Attention masks decide visibility. Labels decide which logits receive loss. Prefill materializes either prefix K/V or separate media memory. Decode reuses that state. A serving engine schedules and caches each object under a compatibility key.

The owner question is:

Which rows exist, which rows may read one another, which rows are supervised, and which multimodal state may be reused across training and serving steps?

This article intersects three earlier handwritten threads. A3 follows data from dataset through collation into model tensors. A5 explains teacher forcing and the shift between decoder inputs and prediction targets. A11 separates padding masks, causal masks, and packed-example masks at the attention score matrix. Multimodal execution does not replace those mechanisms. It composes them with media-object binding, row expansion, objective-specific visibility, and new persistent state.

These questions must not be collapsed:

validity mask:
    is this row real or padding?

attention mask:
    may query row i read key row j?

loss mask:
    should logits at row i contribute to the objective?

media binding:
    which raw object produced this row range?

cache state:
    which processed or encoded objects remain reusable?

What X1.3 owns and defers

X1.3 owns multimodal processor outputs, placeholder expansion, mixed-origin packing, media visibility, label alignment, multimodal training rows, staged execution, media/prefix cache state, batching, admission, reuse, and failure recovery. M3.9 owns general next-token training. M3.10 owns one-request text generation. M5 owns the general serving scheduler and K/V page lifecycle. This article adds the multimodal state those mechanisms must carry.

0x01 A multimodal chat template is a binding program

A user message may be represented as typed content:

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image_a},
        {"type": "text", "text": "Compare this with"},
        {"type": "image", "image": image_b},
        {"type": "text", "text": "and explain the difference."},
    ],
}]

The template may first emit one placeholder per media object:

<image> Compare this with <image> and explain the difference.

The processor then decodes/resizes media, tokenizes text, and returns objects such as:

input_ids:       [B, L_template]
attention_mask:  [B, L_template]
pixel_values:    model-specific image batch
image_grid_thw:  model-specific media geometry

Current Hugging Face multimodal processor documentation makes this boundary explicit: the Processor combines tokenization with modality processors, and placeholder tokens may be expanded to the pattern expected by a specific checkpoint. The template string alone is therefore not the final model row layout.

As of July 2026, the Transformers documentation recommends applying the multimodal chat template through the checkpoint’s Processor with tokenize=True and return_dict=True. Depending on the model, the returned dictionary may contain input_ids, attention_mask, pixel_values, and geometry such as image_grid_thw. These keys and shapes are model/version contracts, not a portable schema:

batch = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
)
required = set(deployment_contract.required_model_inputs)
assert required <= set(batch)

For video, requested frame count must agree with the checkpoint and available memory. A chat template cannot repair a processor/model disagreement about frame sampling or placeholder expansion.

If placeholder $m$ expands to $N_m$ media rows, and the unexpanded template contains $M$ single placeholders, the expanded length is

$L_{\text{expanded}} =L_{\text{template}}-M+\sum_{m=1}^{M}N_m.$

The -M removes the one placeholder row replaced for each media object; the sum adds the actual feature rows. If a model keeps boundary tokens around media, their count must be added separately.

template:
    [BOS, <image>, compare, with, <image>, EOS]

media rows:
    image 0 -> 4 rows
    image 1 -> 2 rows

expanded:
    [BOS, I0.0, I0.1, I0.2, I0.3, compare, with, I1.0, I1.1, EOS]

The formula gives 6 - 2 + 4 + 2 = 10 rows.

Binding invariants

def validate_media_binding(input_ids, media_items, media_rows, image_token_id):
    placeholder_count = (input_ids == image_token_id).sum().item()
    assert placeholder_count == len(media_items)
    assert len(media_rows) == len(media_items)

    for item_id, rows in enumerate(media_rows):
        assert rows.ndim == 2                  # [N_m, d_model]
        assert rows.shape[0] > 0
        assert torch.isfinite(rows).all()

This count check prevents a common class of wrong-image failures. It still does not prove order. The expansion record must carry (batch_id, media_id, placeholder_position, start_row, end_row, processor_config).

0x02 Packing mixed-origin rows

After expansion, each row should have aligned metadata:

state:        [B, L_expanded, d_model]
valid:        [B, L_expanded]
origin_kind:  [B, L_expanded]       # image, text, audio, video, ...
origin_id:    [B, L_expanded]       # which object or text span
position_ids: [B, L_expanded] or model-specific coordinates
labels:       [B, L_expanded]

A useful packer returns a row table before returning tensors:

@dataclass
class PackedRow:
    state: Tensor
    valid: bool
    origin_kind: str
    origin_id: int
    source_index: int
    supervised_label: int = -100

def pack_interleaved(parts):
    rows = []
    for part in parts:
        if part.kind == "media":
            for source_index, state in enumerate(part.states):
                rows.append(PackedRow(
                    state=state,
                    valid=True,
                    origin_kind=part.modality,
                    origin_id=part.media_id,
                    source_index=source_index,
                ))
        else:
            for source_index, state in enumerate(part.text_states):
                rows.append(PackedRow(
                    state=state,
                    valid=True,
                    origin_kind="text",
                    origin_id=part.message_id,
                    source_index=source_index,
                ))
    return rows

Real code should vectorize this operation, but the row object exposes the contract that vectorization must preserve.

Placeholder replacement is a row rewrite, not token substitution

In a LLaVA-style prefix path, one placeholder token has one text embedding but must be replaced by $N_m$ projected media embeddings. The replacement changes every aligned row tensor after that position. Let placeholder index $p$ divide text embeddings into $X_{<p}$ and $X_{>p}$ . Then

$X_{\text{expanded}} =[X_{<p};Z_m;X_{>p}],$

and labels become

$Y_{\text{expanded}} =[Y_{<p};(-100)^{N_m};Y_{>p}].$

The placeholder’s own text embedding and label are removed. Media rows are inserted with ignore labels. Validity, origin IDs, segment IDs, and positions must be rewritten by the same operation.

def replace_media_placeholders(input_ids, labels, text_embeds,
                               media_by_placeholder, image_token_id):
    rows, out_labels, origins = [], [], []
    media_slot = 0

    for token_id, label, text_state in zip(input_ids, labels, text_embeds):
        if token_id != image_token_id:
            rows.append(text_state)
            out_labels.append(label)
            origins.append(("text", -1))
            continue

        media = media_by_placeholder[media_slot]  # [N_m, d_model]
        for local_row in range(media.shape[0]):
            rows.append(media[local_row])
            out_labels.append(-100)
            origins.append(("image", media_slot))
        media_slot += 1

    assert media_slot == len(media_by_placeholder)
    assert len(rows) == len(out_labels) == len(origins)
    return stack(rows), tensor(out_labels), origins

For more than one placeholder, process them left to right or precompute all final offsets. Mutating a tensor in place while still using original placeholder indices makes later indices stale.

Position choices are model-specific

Possible policies include:

ordinary monotonic positions over all expanded rows
one text position shared by a block of inserted media features
separate temporal/height/width coordinates
modality-local positions plus global segment IDs
cross-attention memory with no language-sequence position for media

Changing the policy at inference without matching training and checkpoint semantics is not a harmless packing refactor.

0x03 Four mask families

Let the packed sequence contain media prefix $M$ , prompt $P$ , and answer-input rows $A$ . A complete mask audit needs at least four objects.

1. Validity or padding mask

For row $j$ :

$V_j=1 \text{ if row }j\text{ is real},\qquad V_j=0\text{ if padded}.$

It usually masks key columns and may also zero outputs for padded query rows.

2. Causal mask

For ordinary decoder-only order:

$C_{ij}=1 \iff j\le i.$

This prevents a row from reading future text or future inserted rows.

3. Media topology mask

A prefix-LM-style media block may allow all text rows to read all media rows, while media rows read only media:

$T= \begin{bmatrix} \mathbf{1}_{N_m\times N_m} & \mathbf{0}_{N_m\times L_t}\\ \mathbf{1}_{L_t\times N_m} & C_{L_t\times L_t} \end{bmatrix}.$

This is not the only valid mask. A decoder-only implementation may apply one ordinary causal triangle to the fully concatenated sequence, making early media rows unable to read later media rows. Another architecture may encode media bidirectionally before insertion, making the language mask over media rows less important. The checkpoint’s trained topology is authoritative.

4. Cross-attention media mask

For language queries and separate media memory:

$R_{ij}=1 \iff \text{text row }i\text{ may read media row }j.$

R includes padding and object binding. In interleaved data it can also enforce image causality: text following image 0 must not accidentally read a later image 1 that had not appeared at that point.

Flamingo uses a stricter media-causal rule than “all preceding images.” Let $\phi(i)$ be the index of the most recent media placeholder before language row $i$ . If media memory is grouped by item $m$ and latent row $q$ , its direct cross-attention mask is

$R_{i,m,q}=1 \iff m=\phi(i) \land V_{m,q}=1.$

Thus each language token directly cross-attends only to the immediately preceding image or video. Earlier visual evidence can still propagate through language self-attention because earlier text states already integrated it. This distinction is easy to lose when “media-causal” is implemented as a simple lower triangle over image IDs.

The M3W training pipeline separately reports a webpage-alignment augmentation: with probability $p_{\text{next}}=1/2$ , text is associated with the immediately following rather than previous image. That changes the assigned media index in the training example; it does not turn the cross-attention mask into visibility over every prior or future image. Inference with ordinary interleaving uses the preceding-media binding shown here.

def flamingo_media_mask(text_media_index, media_valid):
    # text_media_index: [B, L], most recent media id or -1
    # media_valid:      [B, M, Q]
    media_ids = torch.arange(media_valid.shape[1], device=media_valid.device)
    same_item = text_media_index[:, :, None] == media_ids[None, None, :]
    return same_item[:, :, :, None] & media_valid[:, None, :, :]
    # [B, L, M, Q], flatten M*Q only at the attention boundary

Source: Flamingo, Figure 7. The mask binds interleaved text spans to media items while ordinary language self-attention carries prior evidence forward.

The original mask article combines padding and causal visibility before softmax

Source: Transformer Series A11, mask overview. X1.3 extends the same operation with media-origin and fusion-topology constraints.

Boolean versus additive masks

An additive attention bias can be constructed as

$A_{ij}=\begin{cases} 0,&\text{allowed},\\ -\infty,&\text{blocked}. \end{cases}$

and applied before softmax:

$P=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_h}}+A\right).$

In low precision, kernels may use a large negative finite value instead of an actual infinity. The mask dtype, polarity, shape, and broadcast axes are part of the backend contract.

Normalize mask semantics before calling the kernel

Libraries disagree about boolean polarity: one API may use True for “allowed,” while another uses True for “blocked.” They also accept different ranks such as [B,L_k], [B,L_q,L_k], or [B,H,L_q,L_k]. Keep one canonical internal representation, then translate at the backend boundary.

def allowed_to_additive(allowed, *, dtype):
    # Canonical internal convention: True means this edge is allowed.
    assert allowed.dtype == torch.bool
    blocked = ~allowed
    bias = torch.zeros(allowed.shape, dtype=dtype, device=allowed.device)
    return bias.masked_fill(blocked, torch.finfo(dtype).min)

def canonical_attn_mask(valid_q, valid_k, topology):
    # valid_q: [B, Lq], valid_k: [B, Lk]
    allowed = topology & valid_q[:, :, None] & valid_k[:, None, :]
    assert allowed.shape == (valid_q.shape[0], valid_q.shape[1], valid_k.shape[1])
    return allowed[:, None, :, :]             # [B, 1, Lq, Lk]

A fully blocked query row has no valid probability distribution: softmax over all negative infinity can become NaN, while a finite sentinel can produce an arbitrary uniform row. Either remove padded query rows from computation or zero their attention output after applying a mask that leaves at least one numerically valid key. Do not rely on a kernel to infer the intended policy.

0x04 Attention visibility and loss supervision are independent

For next-token training, logits have shape

$\text{logits}\in\mathbb{R}^{B\times L_{\text{expanded}}\times |\mathcal V|}.$

Media rows and user prompt rows can be visible context while contributing no language-model target. Labels therefore commonly use an ignore sentinel:

$\mathcal L =-\sum_{i:\,y_i\ne -100} \log p_\theta(y_i\mid x_{\le i}).$

The summation condition is the loss mask. It does not change which keys the row can attend to.

Take four image states, three prompt tokens, and answer apple <eos>:

position:   0   1   2   3   4    5    6     7     8
input:      I1  I2  I3  I4  what blue word  <bos> apple
label:     -100 ... -100                   apple  <eos>

Prefix-style image QA training row with attention and loss masks

The <bos> row predicts apple; the apple row predicts <eos>. The image and prompt rows are visible and unsupervised. Applying cross-entropy to media rows asks vocabulary logits at a non-text row to predict an arbitrary token.

A5's teacher-forcing diagram exposes all shifted target positions in parallel

Source: Transformer Series A5. X1.3 preserves the shifted-target mechanism but removes media, system, and user targets from the loss rather than from the visible context.

def build_answer_labels(input_ids, answer_start, pad_id):
    # input_ids contain prompt plus shifted answer input.
    labels = input_ids.clone()
    labels[:, :answer_start] = -100
    labels[input_ids == pad_id] = -100
    return labels

The exact shift depends on the model’s causal-loss implementation. Unit tests should use unique token IDs and assert which input row predicts each target.

Conversation roles define the supervision mask

The original LLaVA instruction-tuning recipe computes autoregressive loss on assistant answers and their stop tokens, not on the system prompt, user turns, image rows, or separators that merely format those turns. For a multi-turn conversation, supervision is therefore a union of assistant spans:

$S_i=1 \iff \operatorname{role}(i)=\text{assistant} \land i\text{ is a predicted target row}.$

expanded input origin:  system user image user assistant assistant user assistant
target token role:      system user image user assistant assistant user assistant
loss enabled:             no    no   no    no      yes       yes    no      yes

The row holding an input token predicts the next token, so the role attached to the target determines whether that logit is supervised. Masking by the role of the input row instead creates a boundary error at every role transition.

def shifted_assistant_labels(input_ids, target_role, valid, assistant_id):
    # labels[:, i] is the token predicted from input row i-1 in this explicit form.
    labels = input_ids.clone()
    supervise_target = (target_role == assistant_id) & valid
    labels[~supervise_target] = -100
    return labels

def audit_shift(input_ids, labels, target_role, assistant_id):
    for i in range(1, input_ids.shape[-1]):
        expected = input_ids[i] if target_role[i] == assistant_id else -100
        assert labels[i] == expected

Framework language-model heads often shift logits and labels internally. The test must reflect that implementation rather than applying a second shift.

Attention visibility, validity, and answer supervision are separate row contracts

0x05 Packed training examples without cross-example leakage

Packing several short examples into one physical sequence reduces padding. It also creates a new boundary: example A must not read example B unless the training objective explicitly intends that relation.

Let segment_id[i] identify the packed example. A block-causal mask is

$P_{ij}=1 \iff \operatorname{segment}(i)=\operatorname{segment}(j) \land j\le i,$

with any media-prefix exception applied inside the same segment.

def packed_causal_mask(segment_ids, valid):
    # segment_ids: [B, L]
    same = segment_ids[:, :, None] == segment_ids[:, None, :]
    L = segment_ids.shape[1]
    causal = torch.arange(L)[None, :] <= torch.arange(L)[:, None]
    keys_valid = valid[:, None, :]
    queries_valid = valid[:, :, None]
    return same & causal & keys_valid & queries_valid

A11's block-diagonal packed mask isolates independent text examples inside one physical sequence

Source: Transformer Series A11. X1.3 preserves this segment boundary and then composes each block with that example’s media topology.

Media exceptions must remain segment-local. If segment 0 has a bidirectional media prefix and segment 1 has another, the prefix exception is applied twice inside the diagonal blocks, never across them:

$A_{ij} =\mathbf 1[\operatorname{seg}(i)=\operatorname{seg}(j)] \cdot T^{(\operatorname{seg}(i))}_{ij} \cdot V_iV_j.$

Position IDs are a separate contract. Some packed implementations reset text positions at every segment to match independent examples; others continue monotonically because their kernels or positional scheme require a unique physical index. For media with 2D/3D coordinates, scalar packing position and modality-local coordinates may coexist. Record both rather than inferring one from the other after packing.

def build_positions(segment_ids, origin_geometry, policy):
    if policy == "reset_per_segment":
        scalar = position_within_each_segment(segment_ids)
    elif policy == "physical_monotonic":
        scalar = torch.arange(segment_ids.shape[-1], device=segment_ids.device)
    else:
        raise ValueError(policy)
    return {"sequence": scalar, "media_geometry": origin_geometry}

Multimodal packing additionally requires that each media range stays in the same segment as its placeholder and text. Reordering examples after encoding media but before packing is safe only if the media-origin map is permuted by the same index.

Length-aware batching is now multidimensional

Text-only batching may bucket by token length. Multimodal examples vary by:

text tokens
image count and resolution
video frame count
audio duration
encoder output rows
language-visible media rows

Two prompts with 200 text tokens can have radically different prefill work if one contains a thumbnail and the other contains a minute of video.

0x06 Training state: what receives gradients and why

X1.2 compared LLaVA, Flamingo, and BLIP-2 training programs. X1.3 follows those programs through an actual batch.

def multimodal_train_step(batch, model):
    processed = processor(batch.messages)
    # text IDs/masks plus pixel/audio/video tensors and geometry

    media = model.encode_media(processed)       # trainable or no_grad by stage
    packed = model.bind_and_pack(processed, media)

    out = model.language_forward(
        inputs_embeds=packed.states,
        attention_mask=packed.attention,
        position_ids=packed.position_ids,
    )

    loss = cross_entropy(
        out.logits.reshape(-1, out.logits.shape[-1]),
        packed.labels.reshape(-1),
        ignore_index=-100,
    )
    loss.backward()
    optimizer.step()

The encode_media gradient boundary must be deliberate:

if stage.freeze_encoder:
    with torch.no_grad():
        h_media = encoder(media_input)
else:
    h_media = encoder(media_input)

Freezing removes encoder parameter gradients and optimizer state, but the projector may still need encoder outputs for its backward pass. Caching frozen features can save repeated encoder work only when crop, augmentation, normalization, encoder checkpoint, and selected layer are fixed.

Mixed objectives need separate masks

A system can combine:

language modeling / answer loss
image-text contrastive loss
image-text matching loss
grounding or box loss
OCR or transcription loss
modality reconstruction loss

Each objective owns different rows and targets. A single labels tensor is not enough to describe all of them. Log losses separately before weighting:

$\mathcal L_{\text{total}} =\lambda_{\text{LM}}\mathcal L_{\text{LM}} +\lambda_{\text{ITC}}\mathcal L_{\text{ITC}} +\lambda_{\text{ITM}}\mathcal L_{\text{ITM}} +\lambda_{\text{ground}}\mathcal L_{\text{ground}}.$

Changing a weight changes which bridge behavior is optimized; it is not merely report formatting.

Q-Former objectives change the attention graph

BLIP-2 stage 1 demonstrates why an objective ledger must include masks, not only scalar weights. Let rows be ordered as learned queries $Q$ followed by text $T$ .

For image-text contrastive learning, query and text self-attention are separated:

$A_{\text{ITC}}= \begin{bmatrix} \mathbf 1_{Q\times Q} & \mathbf 0_{Q\times T}\\ \mathbf 0_{T\times Q} & \mathbf 1_{T\times T} \end{bmatrix}.$

For image-grounded generation, queries see queries while text sees all queries and causally preceding text:

$A_{\text{ITG}}= \begin{bmatrix} \mathbf 1_{Q\times Q} & \mathbf 0_{Q\times T}\\ \mathbf 1_{T\times Q} & C_{T\times T} \end{bmatrix}.$

For image-text matching, every query/text row can interact:

$A_{\text{ITM}}=\mathbf 1_{(Q+T)\times(Q+T)}.$

def qformer_self_mask(mode, q, t, device):
    mask = torch.zeros(q + t, q + t, dtype=torch.bool, device=device)
    if mode == "itc":
        mask[:q, :q] = True
        mask[q:, q:] = True
    elif mode == "itg":
        mask[:q, :q] = True
        mask[q:, :q] = True
        mask[q:, q:] = torch.tril(torch.ones(t, t, dtype=torch.bool,
                                             device=device))
    elif mode == "itm":
        mask[:] = True
    else:
        raise ValueError(mode)
    return mask

Source: BLIP-2, Figure 2. Sharing Q-Former parameters does not make these three forward graphs equivalent.

This also changes training runtime. One batch may require distinct forwards or carefully batched mask modes, separate negative-pair construction for ITM, and different reductions: ITC takes a maximum over query-to-text similarities, ITM averages query-level binary logits, and ITG applies causal token loss.

Training telemetry

raw media items and decode failures
preprocessor output shapes
unit, encoder-row, and language-visible-row counts
placeholder expansion counts
valid, visible, and supervised rows
loss by objective and modality
gradient norm by encoder/projector/resampler/LM
gate values for inserted cross-attention
examples truncated or rejected by budget

0x07 Inference state machine

A request moves through more states than text-only generation:

RECEIVED
  -> MEDIA_DECODED
  -> MEDIA_PROCESSED
  -> MEDIA_ENCODED
  -> BOUND_AND_PACKED
  -> PREFILLED
  -> DECODING
  -> COMPLETED / FAILED / CANCELLED

Multimodal serving keeps processor, encoder, fusion, and language cache state under different compatibility keys

Each transition can allocate persistent state:

State	Example object	Reuse scope
Raw media	decoded image/video/audio	request or application cache
Processor state	resized pixels, frame samples, grid metadata	exact X1.1 processor-manifest hash
Encoder state	`H_media [N_e,d_m]`	encoder checkpoint/layer-specific
Bridge state	projected/resampled `Z_media [N_l,d_l]`	projector and fusion-specific
Prefix K/V	language-layer K/V for inserted media/prompt rows	exact model/template/position-specific
Cross-attention K/V	per-inserted-layer media projections	exact cross-attention module-specific
Decode K/V	generated language prefix	request branch-specific

Calling all of these “the multimodal cache” is unsafe. They have different keys, bytes, invalidation rules, and sharing boundaries.

A cache key must name the transformation

For reusable bridge state, include at least:

media content hash or trusted media UUID
canonical X1.1 processor-manifest hash
    decoder, resize, crop, frame sampling, normalization, unitizer geometry,
    flatten order, metadata schema, and processor implementation version
encoder checkpoint and selected feature layer
projector or resampler checkpoint
dtype and output layout
fusion mode and media row policy
instruction/template text when query extraction is instruction-conditioned

The manifest hash is the canonical identity, not a substitute for retaining the manifest itself. Admission compares the hash quickly; an audit expands the manifest to explain which transformation changed. This is the same processor_manifest_hash emitted by X1.1 and bound into X1.2’s encoder contract, so raw-unit, encoded-feature, and bridge caches cannot silently use different unitization policies.

For prefix K/V, add language-model checkpoint, chat template, placeholder layout, position policy, and any adapter/quantization identity. Reusing an image embedding under a different crop is wrong even if the media URL is unchanged.

An application-supplied media UUID is an identity claim, not proof that bytes or preprocessing match. A safe cache either verifies a content hash or trusts the UUID only inside an authenticated namespace. Skipping media payloads on a UUID hit is valid only when the relevant processor/encoder cache is enabled and contains an object under the complete transformation key.

0x08 Prefill and decode state differ by fusion topology

Prefix path

encode media
project/resample
expand placeholders into language sequence
prefill media + prompt rows
retain ordinary language K/V for accepted prefix
decode text one row at a time

Media rows increase prefill length and usually remain represented in every language layer’s K/V cache. The K/V payload follows the same M5 formula with a larger prefix length.

If the inserted media contributes $N_l$ rows to a decoder with $L_{\text{lm}}$ layers, $H_{kv}$ K/V heads, head width $d_h$ , and $b$ bytes per scalar, its incremental persistent K/V payload is

$\Delta M_{\text{prefix KV}} =L_{\text{lm}}N_l\cdot2H_{kv}d_hb.$

The factor 2 stores both keys and values. It is not doubled again for queries, because query vectors are not retained in the autoregressive cache.

Cross-attention path

encode media
resample/project memory
optionally precompute media K/V for inserted cross-attention layers
prefill text prompt
decode text while reusing text K/V and media memory

The language self-attention K/V does not need one row per media state, but separate media K/V may exist for each cross-attention layer. The runtime must route both state families to every decode step.

If cross-attention appears in $K$ language layers and stores projected media keys/values with $H_{xkv}$ heads of width $d_{xh}$ ,

$M_{\text{xattn KV}} =K N_l\cdot2H_{xkv}d_{xh}b.$

Before per-layer projection, one reusable bridge tensor costs

$M_{\text{bridge}}=N_ld_{\text{bridge}}b.$

Do not count only one of these objects when the runtime retains both.

Query bottleneck path

A query resampler adds one pre-language computation:

$H_{\text{media}}[N_e,d_m] \rightarrow R[Q,d_r].$

Only $Q$ rows enter the prefix or media-memory lifecycle. Reusing $R$ is safe only if the resampler and all upstream preprocessing are unchanged. For InstructBLIP-style extraction, $R$ also depends on instruction tokens, so an image-only query cache is invalid even when frozen encoder features are safe to reuse.

A topology ledger, not a universal model comparison

Take an illustrative decoder with 32 layers, 32 K/V heads, head width 128, and BF16 state. Inserting 576 media rows as ordinary prefix state adds

32 * 576 * 2 * 32 * 128 * 2 bytes
    = 301,989,888 bytes
    = 288 MiB per request

For an illustrative cross-attention path with 8 inserted layers and 64 media latents at the same K/V width:

8 * 64 * 2 * 32 * 128 * 2 bytes
    = 8,388,608 bytes
    = 8 MiB per request

This does not claim that prefix and cross-attention architectures of those sizes have equal quality or compute. It isolates how row count, insertion count, and state topology enter persistent bytes. Projection weights, allocator metadata, alignment, temporary scores, encoder state, and text K/V remain additional ledgers.

0x09 Continuous batching and admission

Serving cannot schedule by output-token count alone. A request has at least:

media decode work
processor CPU work
encoder accelerator work
projector/resampler work
language prefill work
decode work
persistent bridge and K/V bytes

An admission estimate can record

$C_r=(N_{\text{media items}},N_e,N_l,L_{\text{text}},B_{\text{bridge}},B_{\text{KV}}).$

Current vLLM multimodal configuration exposes per-prompt modality limits, processor options, media preprocessing caches, and multimodal embedding input. Those controls exist because media count alone is insufficient: frame count, resolution, crops, and encoder activations affect peak work and memory.

The current vLLM multimodal-input documentation is a moving runtime contract, so deployment claims should be pinned to an engine version. Its interfaces include per-modality prompt limits, raw multi_modal_data, optional precomputed multimodal embeddings, content hashing/UUID-based cache identity, and placeholder application inside the model input path. Exact embedding shape is model-specific.

Precomputed embeddings are also a trust boundary. The vLLM documentation warns that malformed or untrusted embeddings can crash execution. Admission should therefore verify dtype, rank, row count, model/checkpoint identity, and finite values before enabling an embedding-input path:

def validate_external_mm_embeds(x, contract):
    assert contract.allow_external_embeddings
    assert x.dtype == contract.dtype
    assert tuple(x.shape[1:]) == contract.embedding_tail_shape
    assert x.shape[0] <= contract.max_media_rows
    assert torch.isfinite(x).all()
    assert contract.checkpoint_hash == loaded_checkpoint_hash

Remote media fetching needs an independent security policy: restrict allowed domains, redirect behavior, byte size, decoded dimensions, media duration, and decode time. These checks address SSRF and decompression/resource-exhaustion risk; they are not substitutes for tensor-budget admission after decoding.

def admit(request, budget):
    estimate = estimate_multimodal_cost(request)
    if estimate.media_items > budget.max_media_items:
        return reject("media count")
    if estimate.language_rows > budget.max_prefill_rows:
        return reject("expanded context")
    if estimate.encoder_bytes > budget.max_encoder_working_set:
        return reject("encoder memory")
    return accept(estimate)

Batching options include:

group by processor shape to reduce padding
batch media encoding separately from language prefill
reuse processed or encoded media by compatible key
cap video frames and image tiles at admission
delay decode admission until media state is ready
avoid head-of-line blocking from one very large media request

The scheduler should log raw units, encoder rows, language-visible rows, and actual stage latency. Otherwise a latency spike appears to be a language-model problem when it came from video decoding or image tiling.

0x0A Cancellation, branching, and safe release

Cancellation separates logical close from physical reclamation, using the M5.1/M5.3 owner and generation-fencing contracts:

request-owned decode K/V:
    close logical publication immediately; reject every late generation result
    reclaim only after readers/kernels/transfers quiesce and ownership is rechecked

request-owned temporary encoder activations:
    stop new use on cancellation; wait for encoder/projector and device work
    to finish or establish cancellation completion before ownership recheck/free

shared immutable processed/encoded media cache:
    drop this request's logical reference; keep other users' ownership intact
    an unreferenced entry may stay cached for later hits; evict it per cache policy
    reclaim only when no owner remains and outstanding users have quiesced

shared prefix cache:
    retain only if exact compatibility key and no mutable request state leaked

A cancellation signal or elapsed timeout is not evidence that a DMA, kernel or reader has stopped. Uncertain completion quarantines allocations rather than recycling them. The reclaimer must recheck ownership/generation after waiting, so a late callback cannot free a reused or still-shared allocation. These are consumed owner obligations, not a second multimodal allocator protocol.

Branching a conversation after a shared image may reuse immutable processed or encoded media. Prefix K/V reuse additionally requires the same template, position layout, model tuple, and prefix token sequence. A new crop, different frame sampling, changed adapter, or altered media order invalidates the bridge state even when the user says “use the same image.”

For interleaved conversations, truncating old text without accounting for its bound media can leave orphaned image rows or references. Context truncation must operate on typed segments, not arbitrary token offsets.

0x0B Training and runtime variant comparison

Contract	LLaVA-style prefix	Flamingo-style cross-attention	BLIP-2	InstructBLIP
Media row location	projected grid in language context	separate Perceiver-resampled memory	Q-Former outputs projected to frozen LLM	instruction-conditioned Q-Former outputs
Main visibility object	expanded causal/prefix mask	last-preceding-media cross mask plus text causal mask	ITC/ITG/ITM masks in stage 1; downstream-LM-specific in stage 2	query/instruction self-attention plus downstream LM mask
Supervised rows	assistant answer and stop-token targets in instruction tuning	ordinary interleaved/paired text targets	ITC pairs, ITM pair labels, ITG/LM text targets	instruction-response LM targets
Paper trainable state	stage 1 projector; stage 2 projector + LLM; vision frozen	Perceiver Resampler and two-gate inserted blocks; towers frozen	Q-Former, then Q-Former + FC projection; towers frozen	Q-Former during instruction tuning; towers frozen
Data/optimization choice	caption alignment then visual instructions	weighted interleaved image-text, paired image-text, and video-text mixture	representation learning then generative bootstrapping	multi-task mixture with square-root dataset sampling
Main persistent serving state	media/prompt language K/V	text K/V + per-inserted-layer media memory/KV	query output plus downstream LM state	instruction-specific query output plus LM state
Variable media length	directly changes context unless merged	64-latent reported resampler bounds memory	32 learned queries in reported model	fixed query count, but query values depend on instruction
Main runtime risk	long prefill, label shift, placeholder mismatch	unsupported inserted graph or wrong media-causal binding	wrong objective mask or decoder-family bridge	unsafe image-only query-cache reuse

The table compares contracts, not quality rankings. Later variants change encoders, projectors, tiling, training data, unfreezing policy, and runtime implementation.

0x0C Failure lab

Failure	Observable symptom	Root contract violation
Placeholder/media count differs	exception or wrong image binding	template and processor disagree
Placeholder order differs	fluent answer about another image	media-origin permutation
Expanded length underestimated	context overflow/OOM	one placeholder treated as one final row
Media row visible but labeled	strange vocabulary loss at media positions	attention and loss masks conflated
Prompt row labeled	model learns to reproduce user text	answer boundary wrong
Input-role rather than target-role mask	loss shifts at every role boundary	causal target alignment is off by one
Padding polarity inverted	NaNs or attention to PAD	mask API mismatch
Fully blocked query passed to softmax	NaN or arbitrary uniform output	padded-query policy unspecified
Packed examples share attention	suspiciously low loss	segment boundary missing
Future image visible	leakage in interleaved data	media-causality mask missing
Flamingo token reads all previous images directly	valid lower-triangular mask	published last-preceding-media rule changed
Q-Former losses reuse one mask	finite aggregate loss	objective-specific information flow collapsed
Position policy changed	quality regression with valid shapes	checkpoint packing contract broken
Encoder feature cache stale	wrong answer after crop/config change	cache key names media, not transform
Instruction-aware query cache keyed by image only	fluent but task-misaligned output	bridge state omits instruction identity
Prefix K/V reused across adapter/model	silent corruption	incomplete compatibility tuple
Cross-attention memory dropped in decode	text-only behavior after prefill	state family not routed per step
Untrusted external embedding accepted	worker crash or corrupt state	rank/dtype/checkpoint trust boundary skipped
One long video stalls batch	tail-latency spike	scheduler ignores encoder/media work
Context truncates half a media segment	invalid row binding	truncation operates on scalar tokens only

Publish assertions

assert placeholder_count == media_item_count
assert expanded_rows == packed_states.shape[1]
assert labels.shape == validity.shape == origin_ids.shape
assert torch.all(labels[media_rows] == -100)
assert supervised_rows_match_target_role(labels, target_roles)
assert no_cross_segment_attention(attention_mask, segment_ids)
assert no_fully_masked_live_query(attention_mask, validity)
assert cache_key.names_processor_encoder_projector_and_fusion()
assert instruction_aware_bridge_implies_instruction_aware_cache_key()
assert cancelled_request_has_no_publishable_logical_state()
assert late_generation_results_cannot_publish()
assert reclaimed_allocations_have_quiescence_and_owner_recheck_receipts()

0x0D Final contract

The multimodal execution contract is not one tensor. It is a synchronized set:

raw-media identity and transformation policy
unit and geometry identity
encoder and bridge checkpoint identity
expanded row layout and positions
validity and attention topology
supervised label ranges
prefill and decode state families
cache compatibility and ownership
scheduler admission estimate

The final invariant is:

A multimodal request is correct only when media identity, row layout, visibility, supervision, and reusable runtime state describe the same transformation graph.

With the parallel track complete, the main reading route returns to M6.1. M6 changes capacity and adaptation state; the multimodal bindings, masks, and cache ownership established here remain runtime constraints rather than being reintroduced there.

0xFF References

Attention Is All You Need
Flamingo
BLIP-2
Visual Instruction Tuning / LLaVA
InstructBLIP
Hugging Face - Multimodal chat templates
vLLM - Multimodal inputs
Transformer Series (3) - Data Processing
Transformer Series (5) - Training and Reasoning
Transformer Series (11) - Mask
Transformer System M3.9 - Training Execution
Transformer System M3.10 - Inference Execution
X1.1 - Unitization and Geometry Metadata
X1.2 - Encoders, Projectors, and Fusion
Transformer System M6.1 - Routed Capacity

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| [Transformer System] X1.3 - Masks, Training, Runtime, and State

X1.3 - Multimodal Masks, Training, Runtime, and State

X1.3 - Multimodal Masks, Training, Runtime, and State

0x00 The visible sequence is not the supervised sequence

What X1.3 owns and defers

0x01 A multimodal chat template is a binding program

Binding invariants

0x02 Packing mixed-origin rows

Placeholder replacement is a row rewrite, not token substitution

Position choices are model-specific

0x03 Four mask families

1. Validity or padding mask

2. Causal mask

3. Media topology mask

4. Cross-attention media mask

Boolean versus additive masks

Normalize mask semantics before calling the kernel

0x04 Attention visibility and loss supervision are independent

Conversation roles define the supervision mask

0x05 Packed training examples without cross-example leakage

Length-aware batching is now multidimensional

0x06 Training state: what receives gradients and why

Mixed objectives need separate masks

Q-Former objectives change the attention graph

Training telemetry

0x07 Inference state machine

A cache key must name the transformation

0x08 Prefill and decode state differ by fusion topology

Prefix path

Cross-attention path

Query bottleneck path

A topology ledger, not a universal model comparison

0x09 Continuous batching and admission

0x0A Cancellation, branching, and safe release

0x0B Training and runtime variant comparison

0x0C Failure lab

Publish assertions

0x0D Final contract

0xFF References