X1.2 - Multimodal Encoders, Projectors, Resamplers, and Fusion

0x00 An image does not enter a language model

The phrase “give the model an image” hides several learned interfaces. The processor first creates units and metadata. A modality encoder converts those units into feature states. A projector or resampler changes feature width, row count, representation space, or all three. A fusion topology finally decides where language states can read the result.

raw media
    -> X1.1 units and geometry
        -> modality encoder
            -> feature states
                -> projector / adapter / resampler
                    -> prefix rows, interleaved rows, or external memory
                        -> language computation

The owner question is:

How do non-text features become usable language-side state, and what changes when fusion uses shared sequence rows, learned bottlenecks, or cross-attention memory?

This question inherits the query/key/value geometry developed in Transformer Series A4, but changes one premise. Self-attention derives all three tensors from one sequence. A multimodal bridge may derive language queries from one model and media keys/values from another model trained under a different objective. Compatible matrix dimensions make the operation executable; they do not make the two representation spaces aligned.

At code level, the answer must identify concrete tensors. If image units enter as

$$U_v \in \mathbb{R}^{B\times N_v\times d_{\text{raw}}},$$

the vision encoder may produce

$$H_v=E_v(U_v) \in \mathbb{R}^{B\times N_e\times d_v}.$$

The language model expects states of width $d_l$:

$$X_t\in\mathbb{R}^{B\times L\times d_l}.$$

The bridge must resolve at least two mismatches:

width mismatch:
    d_v versus d_l

row/topology mismatch:
    N_e media rows versus the number and location of rows exposed to language

Matching only the width is necessary and not sufficient. Two encoders can both emit 1024-wide features while representing entirely different objectives, normalizations, spatial granularity, and layer semantics.

Some libraries fuse X1.1 unitization into the first encoder operator. For example, the public API may pass pixel_values [B,C,H,W] directly to a Conv2d patch projection rather than materializing U_v [B,N_u,P*P*C]. This is an implementation fusion, not the absence of a unitization contract. Kernel, stride, resize/crop policy, flatten order, normalization, geometry, and processor-manifest hash still determine the rows that the encoder sees.

What X1.2 owns and defers

X1.2 owns modality encoders as interfaces, feature-layer selection, linear and MLP projectors, query bottlenecks, Perceiver-style resampling, prefix and cross-attention fusion, training-stage comparisons, and architecture/runtime cost ledgers. X1.1 owns unitization and origin geometry. M2.2 owns the general embedding math. X1.3 owns exact multimodal packing, attention masks, loss masks, cache/state lifecycle, and serving APIs.

---

0x01 The modality encoder is already a modeling choice

For an image, a Vision Transformer makes the feature sequence explicit:

patches:          [B, N_patch, P*P*C]
patch projection: [B, N_patch, d_v]
vision blocks:    [B, N_patch (+ class), d_v]

Source: Simon J. D. Prince, Understanding Deep Learning, Chapter 12, Figure 12.17. X1.1 owns patch identity; X1.2 begins with the encoded feature sequence.

The encoder output is not one universal object. Implementations must choose:

which encoder family and checkpoint
which layer or layers to read
whether to keep or drop class/register tokens
whether to keep the spatial grid
whether features are before or after final normalization
whether the encoder is frozen, partially tuned, or fully tuned

The pretraining objective survives inside the feature geometry

CLIP is a useful example because its image tower is not trained to emit language-token embeddings. Given normalized image and text embeddings $\bar v_i$ and $\bar t_j$, a batch produces the pairwise logits

$$s_{ij}=\frac{\bar v_i^\top\bar t_j}{\tau},$$

and a symmetric contrastive objective

$$\mathcal L_{\text{CLIP}} =\frac{1}{2B}\sum_{i=1}^{B} \left[ -\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ij}}} -\log\frac{e^{s_{ii}}}{\sum_j e^{s_{ji}}} \right].$$

The first term retrieves the paired text from an image; the second retrieves the paired image from a text. This objective explains why CLIP features are globally language-aligned, but it does not imply that every patch row already has the distribution expected by a decoder token embedding. LLaVA therefore reads CLIP grid features and learns an explicit projection into the language model width.

The readout is another design choice:

Encoder use	Typical rows exposed	Information retained	Bridge consequence
retrieval/classification	class or pooled row	global semantics	one-vector interfaces can be sufficient
dense visual reasoning	patch/grid rows	spatially distributed evidence	row count and geometry must survive
multi-layer readout	selected layers or mixtures	different semantic/detail levels	projector checkpoint includes layer choice
video encoder	space-time rows or frame summaries	temporal order and motion	time metadata enters resampling and masks

For the original LLaVA recipe, the selected representation is the CLIP ViT-L/14 visual grid before the final projection used by CLIP itself. The article-level invariant is broader: serialize the complete encoder interface, not only its output width.

encoder_contract = {
    "checkpoint": vision_checkpoint,
    "unitizer_hash": processor_manifest_hash,
    "feature_layer": feature_layer,
    "drop_prefix_tokens": drop_prefix_tokens,
    "post_norm": apply_post_norm,
    "grid_shape": (grid_h, grid_w),
    "feature_width": d_v,
}

Layer selection changes the bridge input

Let the encoder contain layers $H_v^{(1)},\ldots,H_v^{(L_v)}$. The projector may consume the last layer, a penultimate layer, or a learned combination:

$$\widetilde H_v = H_v^{(L_v-k)},$$

or

$$\widetilde H_v = \sum_{\ell\in\mathcal S}\alpha_\ell H_v^{(\ell)}, \qquad \sum_\ell\alpha_\ell=1.$$

Later features can be more semantic and less locally precise; earlier features can preserve texture and geometry while being less aligned with language. A feature-layer choice is therefore part of checkpoint compatibility. It cannot be changed independently of a trained projector.

Encoded rows versus language-visible rows

Use separate symbols:

N_u:
    units emitted by preprocessing

N_e:
    rows emitted by the modality encoder

N_l:
    rows exposed to the language computation

A patch-merging encoder may have $N_e<N_u$. A width-only projector preserves $N_l=N_e$. A learned-query resampler chooses $N_l=Q$ independently of the input resolution. Confusing these counts makes both cost and alignment audits unreliable.

---

0x02 Width-only projectors: what the matrix actually learns

The smallest bridge is a linear map:

$$Z_v = H_vW_p+b_p, \qquad W_p\in\mathbb{R}^{d_v\times d_l},$$

so

$$Z_v\in\mathbb{R}^{B\times N_e\times d_l}.$$

The derivation is dictated by matrix compatibility. Each media row is a row vector in $\mathbb{R}^{d_v}$, while concatenation with text requires a row in $\mathbb{R}^{d_l}$. Right multiplication by a d_v x d_l matrix changes only the last axis. It neither changes row count nor creates cross-row interaction.

Its parameter count is

$$P_{\text{linear}}=d_vd_l+d_l,$$

including bias, and its leading multiply-add work is approximately

$$F_{\text{linear}}\approx2BN_ed_vd_l.$$

The projector is shared across media rows, so it learns a feature-space map, not an image-specific lookup.

The backward path makes the alignment role explicit. If $G=\partial\mathcal L/\partial Z_v$ and the batch and row axes are flattened,

$$\frac{\partial\mathcal L}{\partial W_p}=H_v^\top G, \qquad \frac{\partial\mathcal L}{\partial H_v}=GW_p^\top.$$

When the vision tower is frozen, the second tensor is still the derivative needed to propagate through the projector boundary, but optimizer state and parameter gradients are not retained for the frozen tower. The projector update correlates encoder coordinates with language-side error directions. This is why random width matching is not semantic alignment.

class LinearProjector(nn.Module):
    def __init__(self, d_vision, d_language):
        super().__init__()
        self.proj = nn.Linear(d_vision, d_language)

    def forward(self, h, valid):
        # h: [B, N_e, d_vision]
        z = self.proj(h)                  # [B, N_e, d_language]
        z = z.masked_fill(~valid[..., None], 0)
        return z

A checkpoint loader should reject a dimensionally valid but semantically different encoder/projector pair:

def audit_projector_binding(projector_meta, encoder_contract):
    required = ("checkpoint", "feature_layer", "drop_prefix_tokens",
                "post_norm", "feature_width")
    for key in required:
        assert projector_meta["encoder"][key] == encoder_contract[key], key

    assert projector_meta["d_in"] == encoder_contract["feature_width"]

The masking line matters if padded media rows will be concatenated or cached. Zeroing does not replace an attention mask, but it prevents arbitrary padded features from leaking into later residual paths.

Nonlinear MLP projectors

A two-layer projector adds capacity:

$$Z_v=\sigma(H_vW_1+b_1)W_2+b_2, \qquad W_1\in\mathbb{R}^{d_v\times d_m}, \quad W_2\in\mathbb{R}^{d_m\times d_l}.$$

For row-vector notation with hidden width $d_m$:

$$P_{\text{MLP}} =d_vd_m+d_m+d_md_l+d_l.$$

An MLP can bend the feature space rather than apply one affine alignment. It still treats rows independently unless it contains token mixing. It cannot by itself reduce 4,096 patches to 64 rows or let one projected patch summarize another.

Bridge	Changes width	Changes row count	Mixes media rows	Typical role
Linear	yes	no	no	cheap alignment baseline
Two-layer MLP	yes	no	no	nonlinear alignment
Token pooling/merge	optional	yes	locally or fixed	deterministic compression
Learned-query resampler	yes	yes	yes, through attention	bounded adaptive bottleneck
Cross-attention adapter	language width remains	media stays separate	language queries media	repeated fusion inside LM

---

0x03 Prefix and interleaved fusion: media becomes language context

In a prefix-style design, projected media rows and text embeddings share one sequence:

$$X_{\text{ctx}} =\operatorname{concat}(Z_v,X_t) \in\mathbb{R}^{B\times(N_l+L)\times d_l}.$$

The join is possible because both operands have width $d_l$. The language model then updates them under one attention topology.

def prefix_fusion(pixel_values, input_ids, media_valid):
    h_v = vision_encoder(pixel_values)       # [B, N_e, d_v]
    z_v = projector(select_features(h_v))    # [B, N_l, d_l]
    x_t = token_embedding(input_ids)          # [B, L, d_l]

    x = torch.cat([z_v, x_t], dim=1)          # [B, N_l + L, d_l]
    valid = torch.cat([media_valid, input_ids != pad_id], dim=1)
    return x, valid

Source: Visual Instruction Tuning, Figure 1. LLaVA provides the canonical width-alignment and shared-language-context example.

Why prefix fusion is attractive

one language stack processes both origins
existing causal self-attention provides media-to-text access
projector can be small
media states participate in all later layers
implementation can reuse decoder-only kernels after input construction

Why prefix fusion is not free

If every row participates in dense self-attention, the logical score count per layer changes from $L^2$ to

$$(L+N_l)^2=L^2+2LN_l+N_l^2.$$

The extra terms have distinct meanings. $2LN_l$ covers cross-origin interactions in both directions; $N_l^2$ is media-media interaction inside the language stack. During autoregressive decoding, the accepted media/prompt prefix also contributes persistent K/V state in ordinary decoder-only implementations.

Interleaving multiple images with text adds an origin problem:

<image_0 rows> text about image 0 <image_1 rows> text about image 1

Placeholder order, media order, row ranges, and position IDs must agree. X1.3 turns that requirement into a packing and cache-state contract.

---

0x04 Cross-attention fusion: media remains external memory

An alternative keeps language states $X$ and media memory $M$ separate. A cross-attention sublayer computes

$$Q=XW_Q,\qquad K=MW_K,\qquad V=MW_V,$$ $$\operatorname{XAttn}(X,M) =\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_h}}+A\right)V.$$

Language positions supply queries; media rows supply keys and values. The score matrix is

$$S\in\mathbb{R}^{B\times H\times L\times N_l}.$$

This topology does not require media rows to occupy ordinary language-token positions. It does require inserted modules, media memory storage, and masks that bind each text position to the intended image or video.

Flamingo’s inserted block is more specific than a generic one-gate adapter. It has a gated cross-attention residual followed by a separately gated dense residual:

$$Y=X+\tanh(\alpha)\, \operatorname{XAttn}(\operatorname{LN}(X),M),$$ $$Z=Y+\tanh(\beta)\, \operatorname{FFN}(\operatorname{LN}(Y)),$$

where both scalar gates are initialized to zero. At initialization $Z=X$, so the inserted block preserves the frozen language path. Yet the gate can learn immediately because

$$\left.\frac{\partial Z}{\partial\alpha}\right|_{\alpha=0} =\operatorname{XAttn}(\operatorname{LN}(X),M).$$

The parameters behind a closed gate initially receive a gradient scaled by $\tanh(0)=0$. Optimization first opens the residual path; only then do the cross-attention and dense parameters receive an unattenuated signal.

def flamingo_gated_block(x, media, media_mask, alpha, beta):
    xattn_delta = cross_attn(
        query=layer_norm_xattn(x),
        key=media,
        value=media,
        key_padding_mask=~media_mask,
    )
    y = x + torch.tanh(alpha) * xattn_delta
    z = y + torch.tanh(beta) * dense(layer_norm_dense(y))
    return z

Source: Flamingo, Figure 4. Both gates are scalar tanh gates initialized to zero; reducing the block to one gate removes part of the published mechanism.

Source: Flamingo, Figure 3. The frozen vision encoder and language model remain distinct; trained resamplers and gated cross-attention modules bridge them.

Cost shape

Suppose cross-attention is inserted into $K$ language layers. Ignoring projections, its score work scales as

$$K\,L\,N_l,$$

rather than adding media rows to every self-attention matrix. This can be preferable when $N_l$ is bounded and the base language stack should remain mostly unchanged. It also creates a separate media-memory lifecycle instead of ordinary prefix K/V rows.

The insertion count $K$ is itself a paper-variant choice. In the reported Flamingo family, the gated block appears before every frozen LM block in the 3B model, every fourth block in the 9B model, and every seventh block in the 80B model. Those source-reported configurations trade trainable parameters and media interaction frequency against memory and compute. “Flamingo-style” does not identify one universal insertion schedule.

---

0x05 Learned-query bottlenecks and resamplers

Width projection solves $d_v\ne d_l$ and leaves $N_e$ unchanged. A learned-query bottleneck solves a different problem: produce a bounded number $Q$ of summary rows from a variable or large feature sequence.

Let learned queries be

$$R^0\in\mathbb{R}^{B\times Q\times d_r}.$$

A cross-attention update reads encoder features:

$$R^1=R^0+ \operatorname{Attn}(R^0W_Q,H_vW_K,H_vW_V).$$

The score matrix is $Q\times N_e$, and the output row count is always $Q$. That is why a resampler is not merely a projector: it performs content-adaptive cross-row compression.

class QueryResampler(nn.Module):
    def __init__(self, q, d_resampler):
        super().__init__()
        self.queries = nn.Parameter(torch.randn(q, d_resampler))

    def forward(self, media, media_valid):
        B = media.shape[0]
        q = self.queries[None].expand(B, -1, -1)  # [B, Q, d_r]
        q = q + cross_attn(q, media, media, media_valid)
        q = q + ff(norm(q))
        return q                                  # [B, Q, d_r]

BLIP-2 Q-Former

BLIP-2 uses learned queries in a Querying Transformer to bridge a frozen image encoder and a frozen language model. Its two-stage training contract matters:

stage 1 - vision-language representation learning:
    train Q-Former against frozen image features with
    image-text contrastive, image-text matching, and image-grounded generation

stage 2 - vision-to-language generative learning:
    connect Q-Former output to the frozen LLM and train the bridge for generation

The learned queries are not generic pooling slots trained only by final answer loss. Their pretraining objectives determine what information crosses the bottleneck.

The published Q-Former contains an image transformer and a text transformer that share self-attention parameters. Cross-attention layers are inserted into every other transformer block. The 32 learned queries and text tokens can therefore reuse one BERT-initialized self-attention stack while objective- specific masks decide which information is allowed to flow. The cross- attention layers are randomly initialized; BLIP-2 reports 188 million Q-Former parameters for its 32-by-768 query configuration. Those are source-reported architecture values, not requirements for every query bridge.

The first stage is three mechanisms, not one generic pretraining loss:

1. Image-text contrastive learning (ITC). Queries read the image, text encodes independently, and query/text self-attention leakage is blocked. The image-text score is the maximum similarity across query outputs:

   $$s(I,T)=\max_{q\in\{1,\ldots,Q\}} \frac{z_q(I)^\top z_T(T)}{\tau}.$$

   Taking the maximum lets different learned queries specialize in different visual evidence while one query can support the pair-level match.

2. Image-grounded text generation (ITG). Query tokens see one another but not text tokens. Each text token sees every query plus only previous text tokens. The mask therefore turns the same shared self-attention parameters into a multimodal causal decoder.

3. Image-text matching (ITM). Queries and text interact bidirectionally. A binary classifier is applied to every query output and the logits are averaged; hard negatives are sampled from contrasting pairs.

These objectives cannot be reproduced by summing three scalar losses over one unchanged attention graph. Each objective changes the graph that produced its hidden states. X1.3 derives the masks explicitly.

Source: BLIP-2, Figure 2. The same Q-Former parameters execute three objective-specific information-flow patterns.

Source: BLIP-2, Figure 1. The left and right halves correspond to the two distinct pretraining stages.

The second BLIP-2 stage depends on the language-model family

A fully connected layer maps each query output from Q-Former width $d_q$ to the frozen language model’s input width $d_l$:

$$P=ZW_b+b_b, \qquad Z\in\mathbb{R}^{B\times Q\times d_q}, \quad P\in\mathbb{R}^{B\times Q\times d_l}.$$

For a decoder-only LLM, $P$ acts as a soft visual prefix and the text is trained with language modeling conditioned on that prefix. For an encoder-decoder LLM, the projected query sequence and prefix text feed the encoder while the suffix is generated by the decoder. Calling both paths “prefix tokens” hides different masks, position rules, and cache state.

Source: BLIP-2, Figure 3. The bridge width is shared, but the downstream computation graph depends on the frozen LLM architecture.

InstructBLIP makes the bottleneck instruction-dependent

BLIP-2 can compute image query outputs without knowing the eventual user instruction. InstructBLIP instead supplies instruction tokens to Q-Former as well as to the frozen LLM. Through shared self-attention, the learned queries select image evidence conditioned on the task wording:

$$Z_I=Q_\theta(H_v,T_{\text{instruction}}).$$

Consequently, a cache keyed only by the image is invalid for this bridge. The same image paired with “read the sign” and “count the vehicles” may produce different query states. Either cache the frozen encoder features and rerun the instruction-aware Q-Former, or include the normalized instruction and bridge checkpoint in the cache identity.

Source: InstructBLIP, Figure 3. During instruction tuning, the reported recipe keeps the image encoder and LLM frozen and updates Q-Former.

Flamingo Perceiver Resampler

Flamingo also converts variable visual features into a fixed number of visual latents, then exposes them through gated cross-attention layers. The resampler and Q-Former share learned-query attention as a motif, but they are not the same module or training program.

After adding temporal embeddings and flattening the space-time feature axes, the paper’s resampler update uses its current latent array as queries and concatenates visual features with the latent array for keys and values:

$$Q=X,\qquad K=V=[X_f;X].$$

Including the latent array in $K,V$ lets each latent both read visual evidence and interact with the other latents in one attention operation. Flamingo uses 64 output latents per image or video in the reported architecture. This is a source-reported design point, not a universal optimum.

The reported NFNet-based Resampler adds learned temporal position embeddings but no explicit spatial-grid position embeddings; the authors report no gain from the latter and hypothesize that the CNN features already carry spatial information. A ViT-based reimplementation should treat that as a source- specific result, not a rule that spatial coordinates are never useful.

def perceiver_resampler(features, latents, time_embed, valid):
    # features: [B, T, S, d], latents: [B, Q, d]
    features = features + time_embed[:, :features.shape[1], None, :]
    features = features.flatten(1, 2)          # [B, T*S, d]

    x = latents
    feature_valid = valid.flatten(1, 2)        # [B, T*S]
    latent_valid = torch.ones(B, x.shape[1], dtype=torch.bool,
                              device=x.device)
    for block in blocks:
        kv = torch.cat([features, x], dim=1)   # K = V = [X_f; X]
        x = x + block.attn(
            q=block.norm_q(x),
            kv=block.norm_kv(kv),
            key_valid=torch.cat([feature_valid, latent_valid], dim=1),
        )
        x = x + block.ff(block.norm_ff(x))
    return x                                   # [B, Q, d]

Source: Flamingo, Figure 5. The fixed latent count bounds the language-visible memory while attention makes the compression content-dependent.

Variant	Bottleneck output	Downstream fusion	Frozen components	Training emphasis
Linear/MLP prefix	one row per encoder row	shared LM sequence	design-dependent	feature-to-LM alignment, instruction tuning
Flamingo resampler	fixed visual latents per image/video	gated cross-attention	pretrained vision encoder and LM in the paper	interleaved image-text next-token learning
BLIP-2 Q-Former	fixed learned-query outputs	connected to frozen LLM	image encoder and LLM	representation stage, then generative stage
Deterministic pooling	fixed or reduced grid	prefix or memory	design-dependent	no learned content selection in pooling itself

The bottleneck tradeoff is measurable. Reducing $N_e=4096$ to $Q=64$ cuts the language-visible media rows by 64 times. It does not prove that the 64 rows retain the small OCR span, object count, or coordinate needed by a task.

---

0x06 Early, middle, and late fusion are different computation graphs

The word “fusion” is too broad unless it identifies where interaction occurs.

Early/shared-sequence fusion

Media and text become one sequence before most language layers.

project media -> concatenate with text -> shared self-attention stack

Advantages include direct repeated interaction and simple decoder-only reuse. Costs include longer context, media K/V residency, and strict placeholder/row bookkeeping.

Middle/cross-attention fusion

Media remains separate and selected language layers query it.

text self-attention -> media cross-attention -> FFN

This preserves a frozen or mostly frozen language path and separates memory state. It adds cross-attention modules and their per-layer projections.

Query-mediated fusion

Learned queries summarize media before either prefix or cross-attention fusion.

many media rows -> Q learned rows -> language interface

This stabilizes row budgets while moving information selection into a learned bottleneck.

Late/decision fusion

Separate models combine embeddings, scores, or decisions near the output. It can work for retrieval or classification and does not provide the same token-level generative conditioning as prefix or cross-attention fusion.

The design axis is therefore:

where do modalities first interact?
how often do they interact?
which state persists after interaction?
which parameters receive multimodal gradients?

---

0x07 Training contracts across paper variants

The same forward diagram can hide materially different optimization state.

LLaVA-style staged alignment

The original LLaVA architecture connects a CLIP vision encoder to a language model through a learned projection. A useful implementation decomposition is:

feature alignment stage:
    freeze vision encoder
    freeze language model
    train projector on image-text pairs

visual instruction tuning stage:
    keep vision encoder frozen in the original recipe
    train projector and language model on multimodal conversations

The first stage gives the projector a language-compatible starting point. The second teaches instruction following and answer behavior. A later implementation may unfreeze more components, but that is a different optimizer and memory contract.

The original paper reports a linear projector trained on 595K filtered CC3M image-caption pairs in stage 1, followed by visual instruction tuning where the vision encoder remains frozen and the projector and LLM are updated. In that second stage, only assistant-answer and stop-token positions receive the autoregressive loss. LLaVA-1.5 later reports a two-layer MLP projector and a CLIP ViT-L/14 encoder at 336-pixel resolution. The change is not cosmetic: resolution changes $N_e$, while the MLP changes both trainable parameters and the family of alignment functions.

original LLaVA:
    CLIP ViT-L/14 grid -> linear projection -> Vicuna token space

LLaVA-1.5:
    CLIP ViT-L/14 at 336 px -> two-layer MLP -> language model

Flamingo-style frozen towers with inserted trainable paths

frozen:
    vision encoder
    pretrained language-model blocks

trained:
    Perceiver Resampler
    gated cross-attention dense blocks

Interleaved image-text data is central because language positions must learn which preceding media object they can use. The gate protects the initial text path but can also learn too slowly if optimization never opens it.

The reported training mixture combines M3W interleaved image-text data with ALIGN and LTIP image-text pairs and VTP video-text pairs. Flamingo accumulates their gradients with weighted losses rather than cycling one dataset at a time:

$$\mathcal L =\mathcal L_{\mathrm{M3W}} +0.2\mathcal L_{\mathrm{ALIGN}} +0.2\mathcal L_{\mathrm{LTIP}} +0.03\mathcal L_{\mathrm{VTP}}.$$

Those coefficients are source-reported recipe values. The deeper design point is that interleaved examples teach image-to-text binding, paired examples provide scale, and each data stream has an explicit optimization weight. Training telemetry should therefore report per-dataset loss, mixture rate, and both gate values, not only the aggregate language loss.

BLIP-2-style bridge pretraining

BLIP-2 separates representation learning from generative bootstrapping. This creates more training machinery than a linear projector and avoids end-to-end retraining of both large towers. The Q-Former objectives and attention masks change between sub-objectives; reducing the method to “32 query tokens” loses the mechanism that trained those queries.

In the second stage, the image encoder and LLM remain frozen while Q-Former and the fully connected projection are optimized with language-modeling loss. The architecture has therefore changed objectives and downstream graph without turning either large tower into an end-to-end trainable model.

InstructBLIP-style instruction tuning

InstructBLIP starts from pretrained BLIP-2 components, freezes the image encoder and LLM, and instruction-tunes Q-Former across a mixture of tasks. Its reported dataset sampler reduces domination by large datasets with

$$p_d=\frac{\sqrt{n_d}}{\sum_j\sqrt{n_j}},$$

where $n_d$ is dataset size. This is a design alternative to sampling in direct proportion to dataset size. It increases the relative frequency of smaller tasks while retaining more data-dependent weighting than uniform-over-dataset sampling. The paper also reports manual adjustments for selected datasets, so the square-root equation is the baseline sampler rather than the whole recipe.

Parameter and optimizer ledger

For each stage, record:

trainable parameters by module
frozen parameters resident for forward
parameters requiring backward activations
optimizer-state bytes
precision and gradient-scaling policy
which datasets and losses update each module

Freezing a tower removes its optimizer state and parameter gradients. It does not remove its forward compute, weights, inputs, or necessarily all activations needed to train downstream modules.

Whether frozen-tower activations must remain depends on the cut in the backward graph:

frozen encoder output detached before bridge:
    encoder forward activations can be released after H_v is produced
    bridge parameters still receive gradients from H_v as a constant input

frozen encoder parameters but gradient flows to an earlier trainable input/path:
    backward graph through the encoder is still required

trainable encoder:
    encoder parameter gradients, retained activations, and optimizer state apply

Recipe	Vision tower	Bridge/resampler	Language tower	Objective topology
LLaVA alignment	frozen	train projector	frozen	image-caption language alignment
LLaVA instruction	frozen in original recipe	train projector	train	assistant-only causal LM
Flamingo	frozen	train resampler and gated blocks	frozen base blocks	weighted interleaved/paired next-token LM
BLIP-2 stage 1	frozen	train Q-Former	absent	ITC + ITG + ITM with different masks
BLIP-2 stage 2	frozen	train Q-Former + projection	frozen	decoder-only or encoder-decoder LM
InstructBLIP	frozen	train Q-Former	frozen	instruction-conditioned LM across task mixture

def set_training_stage(model, stage):
    freeze(model.vision_encoder)

    if stage == "projector_alignment":
        freeze(model.language_model)
        unfreeze(model.projector)
    elif stage == "instruction_tuning":
        unfreeze(model.projector)
        unfreeze(model.language_model)
    else:
        raise ValueError(stage)

    trainable = [p for p in model.parameters() if p.requires_grad]
    assert trainable
    return trainable

This sketch is LLaVA-like, not universal. BLIP-2 and Flamingo require different stage/module tables.

---

0x08 End-to-end tensor trace

Consider two images and a text prompt. The processor emits two padded image grids with origin metadata:

pixel_values: [B, M, C, H, W]
media_valid:  [B, M]
input_ids:    [B, L]
source_geometry: per-image transform and patch-grid policy
processor_manifest_hash: exact X1.1 unitizer identity

Here patch extraction is fused into vision_encoder; logically it still creates $N_u$ patch units before producing $N_e$ encoded rows.

A prefix implementation may flatten image and patch axes only after preserving the media origin:

def encode_images(pixel_values, media_valid, processor_manifest_hash):
    B, M, C, H, W = pixel_values.shape
    assert processor_manifest_hash == encoder_contract["unitizer_hash"]
    flat = pixel_values.reshape(B * M, C, H, W)

    h = vision_encoder(flat)                 # [B*M, N_e, d_v]
    h = select_patch_features(h)
    z = projector(h)                         # [B*M, N_l, d_l]
    z = z.reshape(B, M, z.shape[1], z.shape[2])

    # A whole missing image invalidates every row for that image slot.
    row_valid = media_valid[:, :, None].expand(B, M, z.shape[2])
    # Padded slots still pass through the encoder and projector, so their rows are not zero; zero them explicitly.
    z = z.masked_fill(~row_valid[..., None], 0)
    return z, row_valid                      # [B, M, N_l, d_l]

At the bridge boundary, preserve enough metadata to prove that every output row still belongs to the intended media item:

@dataclass
class BridgeOutput:
    states: Tensor          # [B, M, N_l, d_bridge]
    row_valid: Tensor       # [B, M, N_l]
    media_id: Tensor        # [B, M]
    source_geometry: object
    unitizer_hash: str
    encoder_contract_hash: str
    bridge_checkpoint_hash: str

def audit_bridge_output(out, expected_media_ids):
    assert out.states.shape[:-1] == out.row_valid.shape
    assert torch.equal(out.media_id, expected_media_ids)
    assert torch.isfinite(out.states[out.row_valid]).all()
    assert (out.states[~out.row_valid] == 0).all()

The placeholder expansion path then maps each media item to an exact text location. A dangerous shortcut is to flatten all media rows and prepend them, even when the chat template interleaves image placeholders with text. That preserves total shape and changes which sentence refers to which image.

For cross-attention, the same encoded tensor can remain [B,M,N_l,d_bridge]; an image-causal mask selects which media indices are visible to each text position.

---

0x09 Architecture and runtime ledger

Let $K$ be the number of language layers containing cross-attention.

Path	Language self-attention length	Extra interaction	Persistent state tendency
Prefix	$L+N_l$	included in shared self-attention	media rows often become prefix K/V
Interleaved prefix	$L+\sum_mN_l^{(m)}$	shared, origin-sensitive	interleaved prefix K/V
Cross-attention memory	$L$	$K L N_l$ score terms	separate media K/V per inserted layer
Query prefix	$L+Q$	resampler $QN_e$ plus shared self-attention	$Q$ prefix rows
Query memory	$L$	resampler $QN_e$ plus $KLQ$	separate $Q$-row memory
Instruction-aware query memory	$L$	query extraction depends on media and instruction	cache encoder features, or key query state by both inputs

This table omits projection constants and kernel effects. It is a topology ledger, not a latency prediction. Runtime performance also depends on encoder placement, media preprocessing, batching by image shape, cache reuse, and whether the backend has optimized multimodal kernels.

---

0x0A Failure lab

Failure	Why shape checks may pass	Consequence
Wrong vision layer	every layer has compatible width	detail/semantic balance differs from projector training
Class/register token mismatch	row count differs by a small constant	placeholders and geometry shift
Width match treated as alignment	d_v == d_l by coincidence	feature spaces remain incompatible
Linear projector expected to compress	output has correct width	media row count remains large
Resampler over-compresses	fixed Q is valid	small or rare evidence disappears
Media padding not masked	padded rows have vectors	language attends to synthetic evidence
Image order differs from placeholder order	total media count matches	prompt binds to wrong image
Frozen-module assumption wrong	forward still runs	optimizer state and gradients explode memory
Gate stays closed	stable text behavior	media path receives little useful signal
Gate opens abruptly	loss may decrease	pretrained language behavior destabilizes
Cross-attention mask leaks future image	tensor is broadcastable	interleaved examples use unavailable evidence
Prefix cost omitted	output quality looks correct	prefill and K/V budget fail under load
Checkpoint/projector mismatch	dimensions match	normalization and feature semantics differ
Image-only cache reused for InstructBLIP queries	tensor shape and image hash match	query state answers the wrong instruction
Flamingo block implemented with one gate	model still trains	published cross-attention and dense residual controls are conflated
Q-Former objectives share one mask	all losses are finite	ITC leaks text, ITG loses causality, or ITM cannot fuse bidirectionally
Frozen encoder interpreted as free	no encoder gradients exist	forward latency and weight residency remain unbudgeted

---

0x0B Design selection questions

Do not choose a fusion method from its name. Ask:

evidence:
    Is fine localization/OCR required, or is scene-level summary sufficient?

row budget:
    Can media rows enter the language context directly?

training budget:
    Which towers can be frozen, adapted, or fully tuned?

data topology:
    Are examples single-image pairs or arbitrarily interleaved media and text?

runtime:
    Does the serving engine support the fusion graph, media cache, and batching?

state:
    Should media become ordinary prefix K/V or remain separately reusable memory?

compatibility:
    Which encoder layer, normalization, projector, and template were trained together?

The final invariant is:

A multimodal bridge is a trained tensor and state contract, not merely a dimension-matching layer.

X1.3 now follows that contract into packing, visibility, labels, cache state, and serving.

---

0xFF References

• An Image is Worth 16x16 Words
• Learning Transferable Visual Models From Natural Language Supervision
• Show, Attend and Tell
• Visual Instruction Tuning / LLaVA
• Improved Baselines with Visual Instruction Tuning / LLaVA-1.5
• Flamingo
• BLIP-2
• InstructBLIP
• Perceiver
• Transformer Series (2) - Overall Architecture
• Transformer Series (4) - Encoder and Decoder
• Transformer Series (7) - Embedding
• Transformer System M2.2 - Embeddings and Projectors
• X1.1 - Unitization and Geometry Metadata
• X1.3 - Multimodal Masks, Training, Runtime, and State