That path is the part A10 implements as attention(query, key, value, mask=None, dropout=None). A11 supplies the source-local causal, padding, training/inference, and packing cases that are reconciled at the score boundary. A1 supplies the generic read and historical lineage; A2 locates the primitive inside the larger Transformer block.

Owner contract: consumes, owns, and defers

This boundary resolves an apparent mask-ownership conflict. M3.1 does own the attention visibility taxonomy needed to define its score rows, including causal, padding, packed, prefix, window, and cache-shaped examples. It does not own the tokenizer/collator policy that produced lengths and segment IDs, nor the training loop that decides which labels contribute to loss.

Claim status and reading rules

The code spine is readable because it suppresses decisions that a real call cannot suppress. The symbol h does not say whether K/V were stored with the same number of heads as Q. The name mask does not say whether True means legal or blocked. float16 inputs do not determine the accumulation dtype of a fused kernel. An API name such as scaled_dot_product_attention does not by itself identify the kernel that ran.

Before comparing formulas, implementations, or timings, freeze one call as an immutable manifest. “Immutable” means that these fields describe one concrete execution and cannot change halfway through a derivation or benchmark. It does not mean that every model and backend must choose the same values.

Logical axes and effective head ownership

Use distinct symbols even when a toy self-attention call makes several of them equal:

The immutable tensor ledger is:

For ordinary multi-head attention, Hq == Hkv and head_map[h] = h. The code spine’s shared h axis is exactly this case, or a call after K/V have already been expanded to the effective query-head axis. With grouped-query or multi-query attention, M3.2 owns physical projection/storage and the mapping. The M3.1 equation consumes that mapping. A backend may implement it by a view, an expand/repeat adapter, indexed grouping, or native grouped-head execution. Only the logical effective K/V objects are invariant.

This distinction matters for accounting. Repeating K/V into an eager reference tensor is a semantic oracle, not proof that a production kernel stores or moves the repeated tensor.

Storage orientation is not only a shape

The article’s canonical boundary is head-major: [B,H,sequence,feature]. The manifest must additionally record physical shape, stride, device, and dtype for Q, K, and V. Those fields answer different questions:

logical axis order:
    what each index means

physical strides:
    how the current tensor view walks storage

contiguity:
    whether the next operator can consume that view directly

materialization:
    whether an object receives its own storage at all

key.transpose(-2, -1) denotes a logical [B,H,Dk,K] view for the score matmul. It is not a claim that checkpoints or KV caches persist a transposed key. Likewise, scores and probabilities are logical objects even when an online fused kernel never writes their full [Q,K] matrices to HBM. M4.3 owns the schedule and IO proof; M3.1 freezes the values that schedule must preserve.

The manifest therefore distinguishes:

qkv_layout = "BHQD"
key_score_view = "transpose last two logical axes"
q_stride, k_stride, v_stride = observed runtime strides
q_contiguous, k_contiguous, v_contiguous = observed booleans
scores_materialized = true | false | backend_internal
probabilities_materialized = true | false | backend_internal

Two calls with equal shapes but different strides can dispatch differently or pay different copies. Two kernels can be semantically equivalent while only one materializes scores. Shape equality is necessary, not a complete execution identity.

Visibility and finite bias are separate objects

M3.1’s canonical semantic object is visible, a boolean relation where True means the query-key edge exists. It must broadcast to [B,Hq,Q,K]. A separate query_valid relation, normally broadcastable to [B,1,Q,1], states whether an output row will be consumed. Key visibility and query validity are not interchangeable.

position_bias is finite and changes preference among legal keys. Backend adapters may encode illegality as an additive tensor containing 0 and -inf, but that tensor is then a visibility encoding, not the finite position-bias object. The canonical composition is:

Here i indexes a query row and j indexes a key row. In implementation-oriented form:

scores = (query @ key_effective.transpose(-2, -1)) * (Dk ** -0.5)
biased_scores = scores if position_bias is None else scores + position_bias
masked_scores = biased_scores.masked_fill(~visible, -torch.inf)
probabilities = softmax(masked_scores, dim=-1)

Applying authoritative visibility after the finite preference makes the separation obvious: no later finite positive bias can revive an illegal edge. Loop 4 derives causal, padding, packed, prefix, window, and cache predicates in full. Loop 2 only freezes the interface every one of those designs must reach.

API adapters must name their polarity rather than passing an undocumented tensor called mask:

The PyTorch functional SDPA polarity and dropout behavior above are official version-scoped contracts, not universal conventions for every attention API. The all-masked-row policy is also a required manifest field: reject, sanitize_then_zero_invalid_query, or a named backend behavior that has been tested. Leaving it implicit is not a valid call specification.

Numerical state is a ledger, not one dtype label

At minimum, record:

An eager policy may explicitly compute softmax in float32 and cast its result back to the query dtype. The current version-scoped LLaMA eager path used later in this article does exactly that. A fused backend may preserve the same output contract with a different online reduction and without exposing intermediate dtypes. In that case the manifest records a documented backend guarantee or backend_internal; it must not guess float32 merely because the eager oracle uses it.

The same rule applies to arithmetic modes such as TF32, reduced-precision reductions, autocast, and deterministic-algorithm settings. They are execution state, not properties derivable from the attention equation.

Dropout mode includes RNG and returned-weight semantics

dropout_p alone does not identify the operation. Freeze this tuple:

training: true | false
dropout_p: probability in [0,1)
dropout_location: attention probabilities before value mixing
rng_provenance: generator/device plus reproducibility state
returned_weights: none | pre_dropout | post_dropout

For PyTorch’s functional SDPA, a nonzero dropout_p applies dropout even when the surrounding module is in evaluation mode; the caller must pass 0.0 for evaluation. Different kernels are not required to consume random numbers in the same order, so a common seed does not by itself prove bitwise parity. Semantic parity tests either disable dropout or compare the stated stochastic contract rather than one sampled mask.

Backend identity has requested and observed parts

backend = "SDPA" is incomplete because the functional entry point can dispatch among implementations. A reproducible call manifest records:

framework and exact version or source revision
model library and exact version or source revision
entry point
requested backend policy
observed selected backend
device type and architecture
autocast / deterministic / arithmetic-mode state
is_causal flag and external mask kind
dropout mode
return-attention-weights request
compile or graph-capture state

PyTorch documents automatic SDPA backend selection and provides sdpa_kernel to constrain eligible implementations. The manifest keeps requested_backend and selected_backend separate because eligibility is not proof of dispatch. If selection was not observed, record unobserved; do not relabel it as FlashAttention from timing intuition. Loop 8 will compare the runtime paths in detail.

Source-to-tensor lineage

Every local object must have one producer and one consumer contract. The lineage is:

The handwritten A10 code supplies the local Q/K/V score path. A11 supplies the mask creation and application lineage. The official PyTorch and version-pinned Transformers sources qualify API polarity, dropout, dtype, and dispatch claims. None of those sources replaces the manifest: the manifest states which subset of their choices applies to this call.

Concrete manifest schema and validation gate

One auditable call can be serialized as:

attention_call_v1:
  axes: {B, Hq, Hkv, Q, K, Dk, Dv}
  tensors:
    query:          {shape, stride, dtype, device, contiguous}
    key_physical:   {shape, stride, dtype, device, contiguous}
    value_physical: {shape, stride, dtype, device, contiguous}
    head_map:       {shape: [Hq], range: [0, Hkv)}
  visibility:
    semantic_polarity: true_means_legal
    shape: broadcastable_to_[B,Hq,Q,K]
    query_valid_shape: broadcastable_to_[B,1,Q,1]
    all_masked_row_policy: reject | sanitize_then_zero_invalid_query | tested_backend_contract
  bias:
    role: finite_preference_only
    shape: broadcastable_to_[B,Hq,Q,K]
  numerical:
    qkv_storage_dtype
    score_accumulation_dtype
    softmax_accumulation_dtype
    probability_dtype
    value_accumulation_dtype
    output_dtype
  dropout: {training, p, location, rng_provenance, returned_weights}
  backend:
    {framework_revision, model_revision, entrypoint,
     requested_backend, selected_backend, device_arch,
     autocast, deterministic, arithmetic_mode, is_causal,
     external_mask_kind, compile_state}
  materialization: {scores, probabilities}

Reject the manifest before execution if any invariant is missing:

assert query.shape == (B, Hq, Q, Dk)
assert key_physical.shape == (B, Hkv, K, Dk)
assert value_physical.shape == (B, Hkv, K, Dv)
assert query.device == key_physical.device == value_physical.device
assert query.dtype == key_physical.dtype
assert head_map.shape == (Hq,)
assert 0 <= head_map.min() and head_map.max() < Hkv
assert visible.dtype == torch.bool
assert torch.broadcast_shapes(visible.shape, (B, Hq, Q, K)) == (B, Hq, Q, K)
assert torch.broadcast_shapes(query_valid.shape, (B, 1, Q, 1)) == (B, 1, Q, 1)
if position_bias is not None:
    assert torch.broadcast_shapes(position_bias.shape, (B, Hq, Q, K)) == (B, Hq, Q, K)
    assert torch.isfinite(position_bias).all()
assert semantic_polarity == "true_means_legal"
assert 0.0 <= dropout_p < 1.0
assert dropout_location == "probabilities_before_value_mixing"
assert requested_backend is not None
assert selected_backend in observed_backend_or_unobserved

These assertions do not prove the attention implementation correct. They make the object being proved stable. Later loops can now derive scaling, masks, softmax, backward, and backend parity without silently changing axes, polarity, precision, or execution identity between sections.

Three useful readings of the same Q/K/V contract

A10 approached Q/K/V from several directions. They are not competing definitions. Each highlights one invariant and becomes misleading when pushed too far.

Differentiable database reading

The database analogy makes the key-value pairing concrete:

query:
    a request represented in the learned matching space

key:
    an address or descriptor used to decide whether a slot is relevant

value:
    the payload retrieved if that slot receives weight

The source figure uses a shopping-query example and Chinese labels. Its useful claim is structural: candidate keys are scored first, and the values remain paired with those keys. Attention differs from a conventional database in at least four ways:

addresses are learned vectors rather than exact identifiers;
matching is dense similarity rather than equality;
the read normally mixes many values rather than returning one row;
the query, keys, values, and scoring projections train end to end.

The analogy therefore explains the read operation, not the storage engine. A Transformer attention layer does not sort scores, retrieve physical rows by ID, or necessarily materialize a top-k result. Dense attention evaluates every legal query-key pair and uses a differentiable weighted sum.

Sequence-to-sequence alignment reading

In encoder-decoder attention, the decoder state supplies a query while encoder states supply keys and values. Each target row obtains its own alignment over source positions:

The score row S[i,:] is target-indexed and source-normalized. This reading is historically important because it connects Bahdanau/Luong alignment to the Transformer primitive. It does not imply that every learned cross-attention head is a clean lexical word-alignment model. Values may represent syntax, position, discourse, or features that do not have one human label.

Contextual reconstruction reading

For self-attention, the output row is reconstructed from value rows:

This makes the operation a content-dependent linear combination in value space. The coefficients are nonlinear functions of Q and K, so the complete map from input states to output states is not linear. Conditional on a fixed probability matrix P, however, the value path is linear:

That distinction matters for both interpretation and optimization. A heatmap shows the coefficients used for the current input. It does not show the value directions being moved, the output projection applied later, or the complete nonlinear function learned by the block.

The three readings agree on the contract:

Q/K decide where information is read from.
V decides what information crosses the selected edges.
The output remains indexed by queries.

They do not assign immutable semantics to vector coordinates. Query, key, and value are roles in one call, not permanent object types.

Loop 3 contract: compatibility first, scaling second

Every attention family in this section must produce one scalar compatibility for every legal query-key pair. Freeze the unscaled compatibility as u and the multiplicative logit scale as alpha:

Here b is a finite preference such as position bias. Visibility is still the separate boolean relation frozen in Loop 2. This ordering prevents three different design choices from being hidden inside one word such as “score”:

Changing the family changes the model. Moving the same positive scalar between algebraically equivalent locations should not change the real-number model, although finite-precision execution can change. Loop 3 owns that distinction. It does not own the complete softmax algorithm in 0x07, position-dependent bias in M3.3, or context-length-dependent rescaling in M3.5.

Alternative score functions and the Transformer choice

Scaled dot product is not the only differentiable compatibility function. The attention lineage used several choices:

A10 also lists the shorthand w^T[q;k] under concatenation similarity. Taken literally without a nonlinearity, it is not an interactive additive score:

For one query row, w_q^T q is constant over every key. Row-wise softmax is invariant to that constant, so the query-dependent term cancels. Bahdanau’s additive score and Luong’s concat score insert a learned nonlinear hidden layer; that hidden layer is what lets query and key jointly change the ranking. The A10 shorthand is retained as source history, not as the implementation contract.

The alternatives also show what the projections are doing. A bilinear score can absorb W_a into one side. Separate W_Q and W_K let the Transformer learn two comparison spaces before the dot product. This does not make every score family equivalent, but it makes scaled dot product much more expressive than comparing raw input embeddings.

Deriving the score families from their interaction assumptions

The formulas look similar when reduced to a table, but they encode different pairwise interaction structures.

Dot product: coordinate agreement in a learned space

For q,k in R^d, the dot product is:

It rewards dimensions that have the same sign and large magnitude. Learned Q/K projections decide what those dimensions mean. There are no score-specific parameters after projection, and all query-key pairs become one GEMM:

This factorization constrains a single head’s unnormalized score matrix. If Q in R^{Q_len x d_k} and K in R^{K_len x d_k}, then:

Softmax is nonlinear and value/output paths add capacity, so this is not a rank bound on the whole attention module. It is a precise property of the raw logit matrix.

The geometric decomposition separates angle from magnitude:

For fixed query i, its norm multiplies the entire score row and therefore acts like a query-specific inverse temperature. A key norm changes one column relative to other keys. The cosine term carries angular interaction. Calling a dot product “only cosine similarity” drops two trainable magnitude channels.

Bilinear score: learn the interaction metric explicitly

Luong’s general score is:

If W_a is unconstrained, it can rotate, scale, and couple dimensions before the comparison. In a Transformer with independent Q/K projections, part of that freedom can be reparameterized into the projections:

This identity explains why a separate W_a is often unnecessary in the scaled-dot-product design. It does not prove every bilinear parameterization trains identically: initialization, sharing, dimensions, regularization, and factorization rank differ.

For unequal widths Dq and Dk, choose W_a in R^[Dq,Dk]. The batched tensor program is:

query:             [B,H,Q,Dq]
W_a:               [H,Dq,Dk]      # or shared [Dq,Dk]
query_metric:      [B,H,Q,Dk]
key:               [B,H,K,Dk]
score:              [B,H,Q,K]

query_metric = einsum("bhqd,hde->bhqe", query, W_a)
score        = query_metric @ key.transpose(-2, -1)

The score-local parameter count is H * Dq * Dk when every head owns a full metric. Association order matters for work and temporary storage even though (Q W_a)K^T = Q(W_a K^T) in exact arithmetic.

The following A10 figure rewrites projected dot-product attention as a learned QK circuit. It is the bridge between Luong’s explicit bilinear metric and the Transformer’s factorized W_Q W_K^T metric; the causal prefix shown in the figure is one visibility policy, not part of the bilinear identity.

Visual provenance: reused from A10, where it is attributed to Ferrando et al., A Primer on the Inner Workings of Transformer-based Language Models. The semantic job here is the factorization W_Q W_K^T, not an empirical claim.

Additive score: a learned nonlinear pair function

Bahdanau attention uses a small network:

For all pairs, projected query terms broadcast down the key axis and projected key terms broadcast across the query axis:

q_term: [B,Q,1,D_a]
k_term: [B,1,K,D_a]
pair:   [B,Q,K,D_a]
score:  [B,Q,K]

The nonlinear pair tensor gives additive attention a different interaction family, but it also introduces a potentially large [B,Q,K,D_a] intermediate. This is why “similar theoretical complexity” does not mean identical hardware behavior. Dot-product attention maps its central pair operation directly onto matrix multiplication.

With heads and unequal input widths, the full tensor trace is:

query:             [B,H,Q,Dq]
key:               [B,H,K,Dk]
W_q:               [H,Da,Dq]
W_k:               [H,Da,Dk]
v_a, b_a:          [H,Da]

q_term:            [B,H,Q,1,Da]
k_term:            [B,H,1,K,Da]
pair_hidden:       [B,H,Q,K,Da]
score:              [B,H,Q,K]

For independent per-head parameters, the score-local parameter count is:

where the final 2Da accounts for b_a and v_a. An implementation can tile or fuse the pair operation instead of storing the complete five-dimensional tensor, but it cannot silently replace the nonlinear pair function with GEMM and still claim the same score family.

Cosine score: remove vector norm from compatibility

Cosine similarity with an explicit zero-norm policy is:

It bounds raw similarity to [-1,1] when norms are nonzero. A temperature or learned scale is usually needed to give softmax an appropriate logit range. Normalizing Q/K also changes the gradient: radial changes no longer affect the score in the same way as angular changes.

The bounded interval is not automatically sharp enough. If one of K logits is +1 and all others are -1, then without an extra multiplier the largest possible probability is:

At K=128, this is only about 0.055. QKNorm is one concrete design point: it L2-normalizes Q and K and replaces the fixed Dk^(-1/2) multiplier with a learned scale. That is a changed model parameterization, not a different placement of the canonical fixed scale.

One batched implementation oracle for all four families

The following pseudocode makes the common output axis and the family-specific temporary state explicit:

def compatibility(query, key, family, params, eps=1e-6):
    # query: [B,H,Q,Dq], key: [B,H,K,Dk]
    if family == "dot":
        assert query.shape[-1] == key.shape[-1]
        return query @ key.transpose(-2, -1)

    if family == "bilinear":
        # W: [H,Dq,Dk]
        q_metric = einsum("bhqd,hde->bhqe", query, params.W)
        return q_metric @ key.transpose(-2, -1)

    if family == "additive":
        # Wq: [H,Da,Dq], Wk: [H,Da,Dk], v/b: [H,Da]
        q_term = einsum("bhqd,had->bhqa", query, params.Wq)
        k_term = einsum("bhkd,had->bhka", key, params.Wk)
        pair = tanh(q_term[:, :, :, None, :]
                    + k_term[:, :, None, :, :]
                    + params.b[None, :, None, None, :])
        return einsum("bhqka,ha->bhqk", pair, params.v)

    if family == "cosine":
        assert query.shape[-1] == key.shape[-1]
        q_hat = query / norm(query, dim=-1, keepdim=True).clamp_min(eps)
        k_hat = key / norm(key, dim=-1, keepdim=True).clamp_min(eps)
        return q_hat @ k_hat.transpose(-2, -1)

    raise ValueError(f"unknown score family: {family}")

Only the dot-family result can be handed directly to the canonical SDPA primitive. Bilinear may be reducible to transformed Q or K before SDPA. Additive requires a different score kernel or an explicit pair program. Cosine requires normalization and, usually, an explicit scale. Backend availability is therefore part of the design choice, not an afterthought.

What the historical comparison does and does not establish

A10 reused the following table from Britz et al.’s large NMT architecture study. It is useful evidence that additive and multiplicative families can be competitive under a matched recurrent NMT experiment. It is not a universal ranking of Transformer attention mechanisms, and the Mul rows are not the canonical scaled Transformer primitive.

Visual provenance: A10 attributes this table to Britz et al., Massive Exploration of Neural Machine Translation Architectures. The table is source-reported, task-specific evidence. State width, recurrent architecture, training recipe, and measurement uncertainty prevent a general winner claim.

The implementation choice is therefore not just “which formula has fewer symbols.” It chooses:

the pairwise function class;
where trainable parameters live;
which pair tensors are materialized;
which operation accelerators execute efficiently;
how logit magnitude is controlled.

For M3.1, scaled dot product remains the reference because it is the model contract used by the canonical Transformer path. The alternatives are retained to explain the design space, not treated as interchangeable aliases.

The unit-variance case is only the shortest corollary. Suppose instead that Q and K are independent random vectors with means mu_q, mu_k and covariance matrices Sigma_q, Sigma_k. Then:

and:

For zero means this reduces to tr(Sigma_q Sigma_k). If coordinates are isotropic with variances sigma_q^2 and sigma_k^2, it becomes:

so the canonical multiplier leaves variance sigma_q^2 sigma_k^2, not necessarily one. The fixed 1/sqrt(d_k) term removes dimension-driven growth; it does not normalize arbitrary learned Q/K statistics.

If Q and K are statistically dependent, second moments alone generally do not determine the variance of their product sum. One needs cross moments or a stronger distributional assumption. This matters in self-attention because Q and K can originate from the same residual state. The Transformer footnote is an initialization-scale argument under independence, not a theorem about every trained logit distribution.

The assumptions are an initialization-scale explanation, not a claim that trained Q/K components remain independent standard-normal variables. Learned projections, normalization, position mechanisms, and data create correlations. The fixed scale still removes the most direct dimensional growth from summing d_k products.

An executable reference sweep in the Loop 3 evidence package samples independent standard-normal Q/K components. Its expected ledger is:

Sampling error is expected; the analytic formula is the contract. The sweep is a diagnostic that catches an omitted scale, the wrong dimension, or accidental double scaling.

Scale placement and fused APIs

Let alpha > 0 be the effective logit multiplier. These four placements are algebraically equivalent in exact arithmetic:

scores_post = (q @ k.transpose(-2, -1)) * alpha

scores_q = (q * alpha) @ k.transpose(-2, -1)

scores_k = q @ (k * alpha).transpose(-2, -1)

root_alpha = math.sqrt(alpha)
scores_split = (q * root_alpha) @ (k * root_alpha).transpose(-2, -1)

For canonical attention, alpha = d_k^(-1/2). The symmetric multiplier on each operand is therefore sqrt(alpha) = d_k^(-1/4), a detail that is easy to get wrong. Scaling both Q and K by d_k^(-1/2) applies an effective multiplier of d_k^(-1) and changes the model.

The placements are not bitwise equivalent in finite precision. Pre-scaling Q or K rounds the operands before accumulation. Post-scaling rounds the dot-product accumulator. A fused kernel may fold the multiplier into a tiled score/softmax pipeline without ever materializing scores. Parity should therefore compare outputs and gradients under a declared tolerance, while the immutable contract compares the effective real-number multiplier.

PyTorch 2.13’s functional SDPA contract names this multiplier scale:

alpha = d_k ** -0.5

# Canonical default: the API derives alpha from query.size(-1).
out_default = F.scaled_dot_product_attention(q, k, v, scale=None)

# Explicit checkpoint or model contract.
out_explicit = F.scaled_dot_product_attention(q, k, v, scale=alpha)

The keyword is applied before softmax. It is a multiplier, not the denominator temperature. If Q is already pre-scaled by alpha, preserve that model with scale=1.0; leaving scale=None would apply the default multiplier again:

# Correct only when q_scaled already contains the complete alpha.
out = F.scaled_dot_product_attention(q_scaled, k, v, scale=1.0)

# Wrong: effective multiplier becomes alpha * d_k**(-0.5).
out_double_scaled = F.scaled_dot_product_attention(q_scaled, k, v)

The backend may differ while this semantic multiplier remains fixed. Conversely, changing from the canonical multiplier to a learned per-head gamma, a length-dependent multiplier, or a score soft-cap is a model change even if the same kernel can execute it. M3.5 owns context-extension scaling; M4 owns whether a requested placement/backend is actually faster.

Scale-placement validation gate

Before comparing eager and fused results, record and assert:

assert manifest.compatibility_family == "dot"
assert manifest.scale_semantics == "multiplicative_inverse_temperature"
assert manifest.effective_logit_multiplier == configured_alpha
assert manifest.scale_application_count == 1
assert manifest.scale_width == query.shape[-1]  # unless checkpoint overrides it

eager_scores = (q @ k.transpose(-2, -1)) * configured_alpha
eager_out = softmax(eager_scores + bias_or_mask, dim=-1) @ v

fused_out = F.scaled_dot_product_attention(
    q, k, v,
    attn_mask=bias_or_mask,
    dropout_p=0.0,
    scale=configured_alpha,
)

assert_close(eager_out, fused_out, atol=declared_atol, rtol=declared_rtol)

This gate catches the dangerous failures that preserve every tensor shape: using d_model instead of d_k, applying the multiplier twice, interpreting an inverse temperature as a denominator, or comparing two backends under different configured scales.

Loop 4 contract: one support relation, several physical encodings

Freeze the pre-softmax row in three layers:

followed by a Boolean visibility predicate $v_{bhij}\in\{0,1\}$. The model-level probability is not merely “softmax plus a mask.” It is softmax on a restricted support:

This equation fixes four properties before selecting an API:

1. each query row has its own legal key set;
2. an illegal key is absent from both numerator and denominator;
3. finite bias changes probability ratios within the legal set;
4. visibility changes the support itself.

The familiar additive representation is one implementation of the same extended-real contract:

Using a large finite negative constant only approximates this support rule. If one illegal entry has original logit z_bad, receives additive penalty -C, and the legal row denominator is Z_legal, then it still receives

in exact arithmetic. Whether it rounds to zero depends on score range, accumulation dtype, and backend. -inf expresses illegality; -1e9 is a historical implementation sentinel whose behavior must be tested in the actual dtype.

Construction owner versus application owner

M3.1 does not invent document boundaries or decide which labels train. It consumes a structured batch contract and applies the resulting relation at the score boundary:

The boundary is deliberate. M2.1 can prove that token j belongs to document d; M3.9 can prove that output row i predicts a particular target; M3.1 proves whether score edge (i,j) participates in the denominator.

Rectangular causal alignment is part of the contract

For Q=2, K=5, the phrase “lower triangular” is ambiguous. Upper-left alignment produces:

q\k  0 1 2 3 4
 0   1 0 0 0 0
 1   1 1 0 0 0

This is correct when the two query rows have absolute positions [0,1]. It is wrong for a decode chunk representing absolute positions [3,4], which needs lower-right alignment:

q_abs\k_abs  0 1 2 3 4
      3      1 1 1 1 0
      4      1 1 1 1 1

The second relation follows from coordinates, not from a special drawing:

q_pos = torch.arange(q_len, device=device) + past_len
k_pos = torch.arange(k_len, device=device)
visible = k_pos[None, :] <= q_pos[:, None]

PyTorch 2.13 documents that scaled_dot_product_attention(..., is_causal=True) uses an upper-left causal bias for non-square matrices. Its prototype CausalBias API distinguishes causal_upper_left from causal_lower_right. Consequently, is_causal=True is not a generic promise that an arbitrary cache offset will be inferred. A runtime must prove that the API’s alignment matches its query/key coordinates, or pass an explicit bias that does.

The table separates semantic causality from physical extent. A cache tensor can contain allocated but uninitialized slots; no triangle can infer those slots are invalid from shape alone.

Read the rightmost matrix critically. Masking PAD columns excludes meaningless K/V rows from every query denominator. Masking the PAD query rows as well creates rows containing only -inf; ordinary softmax does not turn those rows into a mathematically defined row of zeros. A real implementation therefore needs two objects:

key_valid[b,j]:
    may key/value row j contribute to any query?

query_valid[b,i]:
    will output row i be consumed by residual, pooling, or loss code?

Query validity and key validity are different tensors. In particular, query_valid is separate from the relation normalized by attention; it may govern output zeroing or downstream loss without deleting every key from that physical query row.

For a padded encoder batch, the attention relation can remain

while query_valid separately zeros or ignores outputs at synthetic query rows. Conflating these objects is how an apparently reasonable two-dimensional padding picture becomes an all-masked-row bug.

Boolean and additive conventions

The dangerous part is not the mask idea. It is API polarity.

Same dtype, opposite polarity is a silent correctness bug.

Do not let an untyped tensor named mask cross the model/backend boundary. Convert from one canonical semantic object in a named adapter:

def visible_to_sdpa_bool(visible):
    # SDPA: True participates.
    return visible.bool()

def visible_to_mha_attn_mask(visible):
    # MultiheadAttention binary attn_mask: True is blocked.
    return ~visible.bool()

def key_valid_to_mha_padding_mask(key_valid):
    # MultiheadAttention key_padding_mask: True is ignored.
    return ~key_valid.bool()

def visible_to_additive_bias(visible, dtype):
    bias = torch.zeros(visible.shape, dtype=dtype, device=visible.device)
    return bias.masked_fill(~visible, -torch.inf)

PyTorch’s public contracts also differ in rank:

SDPA attn_mask:
    broadcastable to [B,...,Hq,Q,K]

MultiheadAttention attn_mask:
    [Q,K] or [B*H,Q,K]

MultiheadAttention key_padding_mask:
    [B,K]

If MultiheadAttention receives both attn_mask and key_padding_mask, its documentation requires their types to match. The module expands and merges them internally. That convenience does not erase the semantic distinction: attn_mask represents query-key edges, while key_padding_mask represents key-column validity.

Float masks add values rather than declare a Boolean support. They must match the API’s required dtype and device, and any finite entries are preferences, not authoritative exclusion. A release manifest should therefore record:

canonical polarity: True = visible
adapter/API: SDPA bool | MHA blocked bool | additive float | block metadata
materialized shape and broadcast axes
causal alignment: square | upper-left | lower-right | explicit positions
all-masked-row precondition or policy

Policies for structurally unused query rows

Some batches contain query rows whose outputs will never be consumed, such as PAD query positions. There are three defensible policies, but they must not be mixed implicitly.

Policy 1: reject all-masked rows.

This is the strongest reference behavior:

assert visible.any(dim=-1).all()

It catches malformed examples early and is appropriate when every query row is expected to represent a real token.

Policy 2: give every physical query a legal sentinel key.

A dedicated null/sentinel slot can make the row mathematically defined. The sentinel value and its downstream treatment become learned/model state; this is not equivalent to pretending the row did not exist.

Policy 3: mark unused queries separately and zero their outputs.

Here key visibility and query validity are different tensors:

key_visible = ...   # [B,1,Q,K], relation used by softmax
query_valid = ...   # [B,1,Q,1], whether output row is consumed

A safe reference implementation can temporarily make one key legal for unused queries, normalize, then zero the output:

def attention_with_unused_queries(q, k, v, visible, query_valid):
    visible = torch.broadcast_to(visible, (*q.shape[:-1], k.size(-2)))
    query_valid = torch.broadcast_to(query_valid, (*q.shape[:-1], 1))

    no_key = ~visible.any(dim=-1, keepdim=True)
    assert torch.all(~(no_key & query_valid))

    safe_visible = visible.clone()
    safe_visible[..., 0:1] |= no_key

    scores = (q @ k.transpose(-2, -1)) * (q.size(-1) ** -0.5)
    scores = scores.masked_fill(~safe_visible, -torch.inf)
    probabilities = torch.softmax(scores, dim=-1, dtype=torch.float32)
    output = probabilities.to(q.dtype) @ v

    output = output * query_valid.to(output.dtype)
    return output

The temporary key is an implementation device for unused rows only. It must never make a real query legal when its semantic key set is empty.

Zeroing NaN after an undefined softmax is weaker:

probabilities = torch.nan_to_num(softmax(all_negative_infinity_row))

That can hide a real visibility bug and gives backend-dependent gradients. Choosing a policy before normalization makes forward and backward behavior auditable.

Optimized kernels may define their own all-masked-row result. A backend parity test must include this case explicitly or exclude it by precondition; it should not discover the policy accidentally in production.

Mask construction and mask application are different interfaces

The attention kernel should not have to infer why an entry is illegal. Upstream code constructs visibility from semantic objects; the score path only applies the resulting rule.

def make_causal_visibility(q_len, k_len, past_len=0, device=None):
    q_pos = torch.arange(q_len, device=device) + past_len
    k_pos = torch.arange(k_len, device=device)
    return k_pos[None, :] <= q_pos[:, None]       # [q_len, k_len]

def make_key_padding_visibility(attention_mask):
    # attention_mask: [B, k_len], 1 means a real token.
    return attention_mask[:, None, None, :].bool()  # [B,1,1,k_len]

def combine_visibility(causal, key_visible):
    return causal[None, None, :, :] & key_visible

Application then has one narrow contract:

visible = combine_visibility(causal, key_visible)  # broadcastable to scores
masked_scores = scores.masked_fill(~visible, -torch.inf)

Separating the interfaces makes failures easier to localize:

wrong legal positions:
    construction bug

right legal positions, wrong polarity or dtype:
    application/API bug

right mask, NaN row:
    no legal key or unstable backend path

The distinction also matters for cached decoding. With q_len=1, the causal row must use the absolute query position, not local row zero. Otherwise a current token at absolute position 127 would be allowed to read only key zero.

Packed samples require block-diagonal visibility

Packing several independent examples into one physical token row removes PAD tokens, but it does not make the examples one sequence. A causal mask alone would let a token in the second example read tokens from the first example.

For packed segment IDs:

segment: [0,0,0, 1,1,1,1, 2,2, 3,3,3]
position:[0,1,2, 0,1,2,3, 0,1, 0,1,2]

legal visibility is the conjunction:

same_segment = segment_ids[:, None] == segment_ids[None, :]
causal_local = positions[:, None] >= positions[None, :]
packed_visible = same_segment & causal_local

Packing changes the physical schedule, not the independent-example semantics. For lengths l_1,...,l_n, padding every example to L=max(l_s) presents nL token slots and up to

query-key score pairs to a dense batched kernel. Independent variable-length execution needs only

dense pairs before causal triangular savings. Concatenating all tokens and running one unrestricted dense matrix would instead expose

including cross-document pairs that must not exist. Block-diagonal visibility recovers the independent semantics; a varlen kernel can also avoid materializing or computing the off-block region.

For A11’s lengths [3,4,5,4]:

padded dense pairs:       4 * 5^2 = 100
independent dense pairs:  3^2 + 4^2 + 5^2 + 4^2 = 66
unrestricted concat:     16^2 = 256   # wrong support and extra work

These are logical pair counts, not a claim that every backend materializes the score matrix. M4.3 owns the IO schedule and kernel-level traffic.

The source visual marks each packed sample as its own causal triangle. This is the semantic requirement. A production kernel may represent the same structure with cumulative sequence lengths and avoid materializing the dense mask.

Mask algebra across encoder, decoder, cross-attention, and cache modes

Write visibility as predicates rather than pictures. Let query position be i, key position be j, batch row be b, and real[b,j] indicate a non-padding key.

The omitted head index is intentional only when every head shares topology. The full object is V[B,H,Q,K]; a physical [Q,K], [B,1,Q,K], or [B,1,1,K] tensor is valid only because its omitted axes are declared broadcast-invariant. Head-specific windows or sparsity require an H axis and must not be silently collapsed to one shared mask.

Encoder self-attention normally uses:

Every real query can read every real key. Query padding still needs a separate downstream contract because key masking only controls which values enter a row.

Decoder causal self-attention uses:

Cross-attention from target query i to source key j uses:

unless the application adds an explicit source-memory restriction. There is no generic target causal triangle on the source axis. The target row is already conditioned on legal target history before it queries source memory.

For cached decoding, use absolute positions. Let the current query positions be p_i and cached key positions be r_j:

If the implementation appends exactly one current key after a contiguous past, all provided keys satisfy the predicate and an explicit triangular mask may be redundant. That does not justify removing masks from every decode kernel. Chunked decode, speculative branches, static cache buffers, left padding, and mixed-length batches can expose keys whose validity cannot be inferred from tensor extent alone.

Prefix and window masks are visibility designs, not score biases

A prefix language model may allow all prefix rows to see one another while continuation rows are causal:

A causal sliding window of width w uses:

These masks remove edges from the attention graph. A finite position bias does not: a sufficiently strong content score can overcome a finite negative bias, while a masked edge must remain impossible. That is why ALiBi-like preferences, relative-position biases, prefix visibility, and sliding-window legality should not all be represented by an undocumented float tensor called mask.

One predicate compiler, not one universal dense matrix

A transparent reference compiler can combine declared semantic inputs before choosing a physical encoding:

def compile_visibility(
    q_pos,                 # [B,Q] absolute or segment-local coordinates
    k_pos,                 # [B,K]
    key_valid,             # [B,K]
    *,
    causal=False,
    q_segment=None,        # [B,Q]
    k_segment=None,        # [B,K]
    prefix_length=None,    # scalar or [B]
    window=None,
):
    visible = key_valid[:, None, :]          # [B,1,K]

    if q_segment is not None or k_segment is not None:
        assert q_segment is not None and k_segment is not None
        visible = visible & (
            q_segment[:, :, None] == k_segment[:, None, :]
        )

    if prefix_length is not None:
        P = torch.as_tensor(prefix_length, device=q_pos.device)
        if P.ndim == 0:
            P = P.expand(q_pos.size(0))
        q_in_prefix = q_pos < P[:, None]
        k_in_prefix = k_pos < P[:, None]
        topology = (
            (q_in_prefix[:, :, None] & k_in_prefix[:, None, :])
            | (~q_in_prefix[:, :, None]
               & (k_pos[:, None, :] <= q_pos[:, :, None]))
        )
        visible = visible & topology
    elif causal:
        visible = visible & (
            k_pos[:, None, :] <= q_pos[:, :, None]
        )

    if window is not None:
        distance = q_pos[:, :, None] - k_pos[:, None, :]
        visible = visible & (distance >= 0) & (distance < window)

    return visible[:, None, :, :]            # [B,1,Q,K]

This is an eager semantic oracle, not a recommendation to materialize a dense mask at long sequence lengths. A production adapter may lower the same predicate to an is_causal flag, sequence lengths, cumulative offsets, CausalBias, or block metadata. The lowering is correct only if it preserves every (b,h,i,j) truth value.

The function also exposes incompatible requests. A prefix topology already defines its own causal region; blindly setting both prefix_length and a separate global causal flag can accidentally remove legal prefix-to-prefix edges. Semantic composition must happen once before backend lowering.

Boolean composition and additive composition must agree

With True = visible, boolean rules compose with logical AND:

visible = key_is_real & causal & same_packed_segment

With additive masks, each rule contributes 0 for legal and -inf for illegal:

additive = padding_bias + causal_bias + segment_bias
masked_scores = scores + additive

Because -inf + finite = -inf, either illegal rule remains authoritative. In finite dtypes and mixed precision, constructing several copies of torch.finfo(dtype).min and adding them can overflow to -inf; that is usually semantically fine for illegal entries but can complicate debugging and backend dispatch. Constructing one final boolean visibility relation and converting it once is often easier to audit.

The following truth table should hold before any optimized representation is accepted:

This is the mask’s model semantics. Dense tensors, block metadata, sequence length arrays, and kernel flags are alternative physical encodings.

Dense support, compact metadata, and predicate kernels

The same visibility relation can cross the backend boundary in several forms:

PyTorch FlexAttention makes the last two distinctions explicit. A mask_mod returns whether an edge exists and can be precompiled into BlockMask metadata; a score_mod changes a computed finite score. The separation is the same one established here between visibility and bias. It also has a systems consequence: fully illegal blocks can be skipped, fully legal blocks need no per-element mask test, and only boundary blocks require the predicate.

During cached decoding, a predicate written against local q_idx=0 repeats the same offset bug as a dense upper-left triangle. The official FlexAttention inference example adds the cache offset to the query index before comparing it with kv_idx. Logical coordinates remain part of correctness even when no dense tensor exists.

The mask lifecycle: data policy, model semantics, backend encoding

A11 did more than draw triangular matrices. It followed masks from batch construction through encoder and decoder call sites. That lifecycle is worth preserving because many mask failures happen before the attention function is called.

dataset / tokenizer policy
    -> token IDs, lengths, segment boundaries, PAD convention
    -> semantic visibility predicates
    -> model API masks and position metadata
    -> broadcast / packed representation
    -> backend-specific mask, sequence lengths, or causal flag
    -> score exclusion before row normalization

Each stage owns a different question:

The attention function can verify shapes and legal rows. It cannot reconstruct document boundaries that were discarded by the collator or infer whether the training objective intended prefix-LM rather than causal-LM visibility.

Immutable visibility handoff

A reproducible call should be able to serialize the information from which the relation was compiled, even if the dense relation itself is never stored:

visibility_handoff:
    batch_layout: padded | packed | jagged | cache
    q_positions / k_positions: values or derivation rule
    key_valid: shape, polarity, padding side, initialized cache length
    query_valid: shape and downstream zero/ignore policy
    segment_boundaries: segment IDs or cumulative lengths
    topology: bidirectional | causal | prefix | window | custom
    prefix_length / window / cache_offset: explicit when applicable
    head_sharing: shared | per-head

backend_lowering:
    API and pinned version
    bool/additive/block/length representation
    polarity and broadcast axes
    non-square causal alignment
    all-masked-row precondition

The handoff is immutable for one forward call. Mutating past_len, sequence boundaries, or padding side after Q/K positions were derived creates a new visibility object even if the underlying storage address is reused.

Harvard names, PyTorch names, and semantic objects

A11’s comparison figure records an important source of confusion: libraries split the semantic mask into differently named tensors.

The figure contains Chinese labels, but its central split is readable:

attention mask:
    relation between query positions and key positions
    examples: causal triangle, memory restriction, block-diagonal packing

key padding mask:
    which key columns correspond to real tokens in each batch row

For torch.nn.Transformer, the common public objects are:

These names are API contracts, not six independent mathematical ideas. At the score boundary they become a relation broadcastable to [B,H,Q,K]. A wrapper may combine them; a fused kernel may receive only cumulative lengths and an is_causal flag. Parity means all encodings exclude the same edges.

Teacher-forced training: future rows exist but remain illegal

During decoder training, the target batch commonly contains the whole shifted target sequence:

decoder input:  <bos> y_0 y_1 ... y_(T-2)
labels:          y_0  y_1 y_2 ... y_(T-1)

All input rows can be processed in parallel because row i is prevented from reading rows j > i. The target causal mask and the label shift solve different problems:

causal visibility:
    prevents hidden-state leakage inside the model

label shift:
    chooses which next token each output row predicts

loss padding mask:
    chooses which output rows contribute to the objective

Applying only two of the three can produce plausible loss curves with the wrong task. In particular, masking PAD keys does not automatically remove PAD query rows from cross-entropy.

The handwritten mask article followed the same objects through the encoder and both decoder attention calls:

The operational call graph is:

memory = encoder(src_states, src_visibility)

target_states = decoder_self_attention(
    target_states,
    target_states,
    target_states,
    target_visibility,       # causal AND target-key-real
)

target_states = cross_attention(
    target_states,
    memory,
    memory,
    source_key_visibility,   # source columns, no target triangle
)

When S == T, swapping the last two masks may preserve every tensor shape. The semantics still fail. This is why call-site tests need unequal source and target lengths.

Prefill, one-token decode, and static cache buffers

At inference prefill, a decoder-only model processes a prompt block with a causal relation over prompt rows. After prefill, ordinary one-token decode may have:

Q = 1
K = past_length + 1

If the K/V tensor contains exactly the contiguous legal prefix, every supplied key is legal for the current query and the kernel can omit an explicit dense triangle. That optimization is a consequence of cache extent and absolute positions, not a change in causal semantics.

Explicit validity becomes necessary again when physical extent exceeds legal state:

static cache:
    allocated slots beyond current length must stay invisible

left-padded batch:
    physical columns before the real prefix may be PAD

chunked decode:
    several new queries need a local causal relation inside the chunk

speculative branch:
    candidate rows may have branch-specific ancestry

sliding window:
    old but physically retained keys may be outside the legal window

The backend may encode these cases through lengths, cache positions, block tables, or a custom mask. The model-level audit remains the predicate legal(query_position,key_position).

Packing is a joint attention, position, and loss contract

Padding a batch to its longest sequence can waste most token slots when length distributions are broad.

Packing removes that waste by concatenating independent examples into fewer physical rows:

But concatenation alone silently changes the training problem. A correct packed representation needs at least:

token IDs:
    the physical packed row

segment or cumulative-length metadata:
    which tokens belong to one logical example

position IDs:
    whether positions reset per example or follow another documented policy

attention visibility:
    block-diagonal causal edges, no cross-example reads

loss boundaries:
    labels must not predict the first token of the next packed document from
    the final token of the previous document unless that transition is intended

For example lengths [3,4,5,4], cumulative boundaries are:

cu_seqlens = [0,3,7,12,16]

A variable-length attention kernel can consume those boundaries and run the four causal examples without constructing a dense 16 x 16 block mask. The dense block-diagonal matrix and cu_seqlens are semantically equivalent only if query/key ordering and position policy agree.

The LongAlign figure retained by A11 shows that packing also has a scheduling dimension: filling token capacity does not guarantee balanced per-rank work or equal example weighting.

Three concerns should be measured separately:

padding efficiency:
    fraction of physical token slots carrying real tokens

compute balance:
    whether workers receive comparable sequence work

optimization weighting:
    whether long examples dominate because loss is averaged per token rather
    than per logical example

The source packing survey compares combinations of sorting, padding, fixed length packing, position IDs, and document isolation:

The figure is a source-reported design comparison, not a universal ranking. The correct choice depends on whether the workload permits truncation, whether the kernel accepts variable lengths, how distributed batches are balanced, and how the loss weights examples. The non-negotiable correctness rule is simpler:

tokens from independent examples must not exchange attention or labels unless
the dataset explicitly defines that relation.

Packing conformance test

A minimal test compares packed and unpacked execution with dropout disabled:

def assert_packed_equivalence(model, examples, pack):
    separate = [model(x, causal=True) for x in examples]

    packed_ids, segment_ids, positions = pack(examples)
    packed_out = model(
        packed_ids,
        segment_ids=segment_ids,
        position_ids=positions,
    )

    cursor = 0
    for expected, example in zip(separate, examples):
        length = example.shape[-1]
        actual = packed_out[:, cursor:cursor + length]
        torch.testing.assert_close(actual, expected, rtol=1e-4, atol=1e-5)
        cursor += length

This catches cross-example leakage, position-reset mistakes, and wrong cumulative boundaries. A separate label test must confirm that loss transitions also stop at document boundaries.

Loop 5 contract: normalize one finite legal row

Loop 4 froze the support relation. For one fixed [batch, head, query] row, let

be the legal key indices and let s_j denote the finite scaled score plus any finite position bias for j in J. Loop 5 owns the map

It does not choose J, move a key into or out of the support, define a dropout realization, or choose an IO-aware kernel schedule. Those boundaries matter because four objects that are often called “the softmax row” are not interchangeable:

The strict reference domain is therefore:

support is non-empty;
every legal score is finite;
masked entries are absent from the reduction;
normalization is over the key axis;
probabilities are inspected before attention dropout.

NaN, legal +inf, legal -inf, and empty support are explicit policy boundaries later in this section. They are not silently repaired by changing the formula.

One log-partition function generates the entire row geometry

Define the log-partition function on the legal support:

Differentiate it with respect to one legal score:

Thus softmax is not an unrelated activation placed after the score function:

This one identity will generate the translation invariance, probability ratios, Jacobian, entropy relation, and temperature derivatives below. It also fixes the difference between a probability and its normalizer:

A(s):       one scalar log-normalizer for the row
p_i:        one derivative of that scalar with respect to score i
sum_i p_i:  one, because all legal terms share the same denominator

Adding a common constant c gives

so its gradient is unchanged:

The logits are consequently identifiable only up to a common row offset. Two implementations can use different representatives of the same row and still produce exactly the same real-arithmetic probabilities.

Softmax is entropy-regularized competition

Softmax also solves a constrained optimization problem:

where

Introduce a Lagrange multiplier for sum_j p_j = 1. For an interior optimum,

Therefore p_j is proportional to exp(s_j), and normalization recovers softmax. The score term rewards concentrating on large logits; Shannon entropy rewards spreading mass. Temperature changes the strength of that competition. Removing or replacing softmax is therefore a model change, not merely deleting one expensive function call.

The softmax Jacobian and gradient coupling

For probability vector p = softmax(s), the derivative is:

In matrix form:

The off-diagonal terms are not zero. Raising one logit changes all probabilities because the row must continue to sum to one. Given upstream gradient g_p, the efficient vector-Jacobian product is:

This form appears later in the full backward trace. It also explains why inspecting each attention edge independently is incomplete: legal edges in the same row compete through normalization.

The same matrix is the Hessian of the log-partition function:

For any vector z over the legal keys,

So the Jacobian is symmetric positive semidefinite. Its constant direction is null:

This is the differential form of row-offset invariance. If all n legal probabilities are positive, the Jacobian has rank n-1: softmax maps an n-coordinate logit row onto the interior of an (n-1)-dimensional probability simplex. No gradient can identify or update a common score offset through this operation alone.

The covariance reading is also an implementation test. A computed Jacobian should be symmetric, have row and column sums near zero, and have no materially negative eigenvalue beyond the declared numerical tolerance. Those checks are stronger than testing only sum(p) = 1.

Saturation from a large logit gap

For two logits [0, delta]:

As delta grows, p_2 approaches one and the derivative approaches zero. Scaling by sqrt(d_k) is therefore connected directly to optimization: it reduces dimension-driven logit gaps before softmax. It does not guarantee high entropy after training; the model can still learn sharp distributions.

Two boundary cases sharpen that statement:

one legal key:
    p = [1] and J = [0] for every finite score;
    this is structural certainty, not a numerical failure

tied largest scores:
    lowering temperature does not choose one tied key;
    the limiting mass is shared uniformly across the maximal set

For example, softmax(alpha * [3,3,1]) approaches [0.5,0.5,0] as alpha -> infinity, not a one-hot vector. An implementation oracle must include ties and singleton support instead of testing only a unique maximum.

Entropy is descriptive, not an objective by default

The entropy of one attention row is:

Uniform attention over n legal keys has entropy log(n); a one-hot limit has entropy zero. Temperature controls this distribution:

Increasing tau flattens the row; decreasing it sharpens the row. But low entropy is not automatically “better focus,” and high entropy is not automatically “confusion.” A head may need a broad average, a special-token sink, a local copy, or a sparse retrieval depending on layer and task. Unless the training loss contains an explicit entropy term, entropy is a diagnostic of the learned routing distribution rather than a direct target.

The log-partition identity gives a more precise temperature path. Let alpha = 1 / tau >= 0 and

Differentiate one probability:

Large-score keys gain mass and below-mean keys lose mass. The expected score obeys

Using

we obtain

For nonnegative inverse temperature, sharpening cannot increase entropy in exact arithmetic. Equality holds when the legal scores are all equal or at alpha = 0. The endpoints are:

and, as alpha -> infinity, uniform mass over the set of maximal scores. This derivation is row-local. It does not establish that changing temperature improves a trained model, preserves calibration, or generalizes across context lengths.

Length-dependent temperature is a model change

A10 discusses scaling logits as context length changes, schematically using a factor related to log(n)/log(m) for inference length n and training length m. The motivation follows from the gap equation: sharper logits can offset a larger legal support.

Write a general multiplier c(n):

Increasing c(n) lowers the effective temperature and amplifies every score difference. This may preserve concentration in some distributions, but it is not an algebraically exact context extension. It changes attention weights, can over-sharpen already decisive rows, interacts with position geometry, and may alter behavior on heads that intentionally average broadly.

The claim should therefore be scoped:

source-local motivation:
    longer legal support can increase entropy or dilute fixed score gaps

rewrite interpretation:
    length-dependent logit scaling is one possible compensation

not guaranteed:
    one scaling law preserves quality or entropy for every head and input

M3.5 owns the full context-extension comparison, because score scaling must be evaluated beside RoPE frequencies, interpolation, fine-tuning, retrieval tests, and serving constraints.

Better diagnostics than one mean entropy number

For one attention row, useful diagnostics include:

entropy H(p):
    overall uncertainty, depends on support size

normalized entropy H(p)/log(n_legal):
    concentration relative to the maximum for that legal support

effective support exp(H(p)):
    number of equally weighted keys with the same entropy

top-k mass:
    how much probability lies in the largest k entries

maximum probability and logit gap:
    sharpest edge and separation from competitors

mass on designated evidence spans:
    task-conditioned retrieval diagnostic, when labels exist

These measurements should be stratified by layer, head, query type, and context length. Averaging all rows can hide a mixture of one-hot copy heads and broad aggregation heads.

Attention softmax is not vocabulary softmax

A10 also surveys log-softmax, hierarchical softmax, and adaptive softmax. Those methods primarily address output distributions over a large vocabulary. They are not direct replacements for attention normalization over key positions.

vocabulary softmax axis:
    candidate token IDs, often tens or hundreds of thousands

attention softmax axis:
    legal key/value rows for one query

Log-softmax is valuable when a downstream loss consumes log probabilities, but attention needs probabilities or an equivalent normalized value accumulator for P@V. Hierarchical/adaptive vocabulary methods exploit token-frequency and class structure; key positions in one attention row do not follow that fixed vocabulary tree.

This is a deliberate non-import from the handwritten source: the broader softmax background remains useful in A10, while M3.1 retains only methods whose axis and contract match attention.

Alternative row normalizers change the routing geometry

The design space is broader than ordinary softmax, but each alternative gives up at least one part of the contract just derived:

Sparsemax is defined as a simplex projection:

Unlike finite-logit softmax, the projection can land on the simplex boundary and assign exact zero mass. Martins and Astudillo derived its Jacobian and applied it to neural attention. Peters, Niculae, and Martins later introduced alpha-entmax sequence models, where softmax and sparsemax are members of a larger family and alpha > 1 can produce sparse alignments.

These alternatives are not drop-in numerical fixes for an unstable softmax:

switching normalizer changes forward probabilities;
switching normalizer changes gradients and learned optima;
exact sparsity does not automatically produce kernel speedup;
checkpoint weights trained with one map need not preserve behavior under another.

Loop 5 records this comparison so that “tiny floating-point probability” is not confused with “model-defined zero support.” M3.1 Loop 9 owns the full alternative-interaction comparison, including kernel/linear attention and the M4 runtime boundary.

Exceptional rows need policy before arithmetic

Subtract-max softmax has a finite-input contract. Applying it blindly to extended or undefined values produces ambiguous or invalid operations:

A convenient extended-real convention could assign all mass to one +inf, or split mass uniformly across several +inf entries. That convention is not the ordinary real-valued softmax formula, and different frameworks need not infer it. The strict oracle used here rejects non-finite legal logits and allows -inf only as a physical encoding of an already-excluded entry.

Do not repair an undefined row after the fact:

# Too late: the reduction has already lost the semantic distinction.
probabilities = torch.nan_to_num(torch.softmax(scores, dim=-1))

This can turn empty support, an upstream NaN, and a legal +inf into the same zero-filled symptom. Validate support and finiteness before normalization so the failure retains its cause.

Naive, safe, and online normalization are the same row contract

A10 retained a two-pass naive softmax algorithm: first accumulate the denominator, then divide each exponential.

The direct algorithm is mathematically clear but numerically unsafe if logits are large. For a finite row, define its native denominator

Subtracting the final maximum m gives

The common e^m cancels from every probability:

Safe softmax therefore changes coordinates, not the ideal row distribution. Online softmax changes when those same coordinates are discovered. Suppose a prefix has state (m_old, ell_old) and an incoming tile has maximum m_tile. Choose one common reference:

Every old and incoming exponential must be expressed under that reference:

By induction, after the final tile:

Native, safe, scalar-online, and block-online normalization thus share one denominator in exact arithmetic. They are not necessarily bitwise identical in finite precision because reduction and merge order change rounding.

The smallest auditable online normalizer keeps an explicit validity branch so the empty-state identity never evaluates -inf - (-inf):

def online_normalizer(finite_scores):
    valid = False
    running_max = -float("inf")
    running_sum = 0.0

    for score in finite_scores:
        if not math.isfinite(score):
            raise ValueError("legal score must be finite")
        if not valid:
            running_max = score
            running_sum = 1.0
            valid = True
            continue

        new_max = max(running_max, score)
        running_sum = (
            running_sum * math.exp(running_max - new_max)
            + math.exp(score - new_max)
        )
        running_max = new_max

    if not valid:
        raise ValueError("softmax support is empty")
    return running_max, running_sum

For the adversarial row [1000, 999, 998], direct exponentiation overflows in ordinary floating-point arithmetic. Subtract-max and online normalization both reduce it to the finite gaps [0,-1,-2] and produce approximately:

[0.66524096, 0.24472847, 0.09003057]

The A10 image is sourced from Milakov and Gimelshein’s online-normalizer paper. It captures the naive two-scan denominator, but not the safe maximum pass or the online recurrence. The existing M4.3 visual below supplies that missing algorithm comparison:

M3.1 owns the distribution identity, strict finite-row domain, and parity oracle. M4.3 owns scan/access counts, merge trees, online value numerators, FlashAttention scheduling, HBM traffic, and backward reconstruction. Carrying the same recurrence here is a semantic handoff, not a second kernel chapter.

Loop 5 executable gate

The publication oracle exercises properties that a shape-only example misses:

translation:
    softmax(s + c) == softmax(s)

partition gradient:
    finite_difference(grad(logsumexp)) == softmax(s)

curvature:
    J == diag(p) - p p^T
    J is symmetric positive semidefinite
    J @ ones == 0

temperature:
    entropy is non-increasing with nonnegative inverse temperature
    tied maxima share limiting mass

support:
    singleton -> p=[1], J=[0]
    empty support -> explicit error

finite precision:
    native overflow is observable
    subtract-max avoids positive overflow
    low precision may underflow tiny legal probabilities to zero

exceptional input:
    legal NaN, +inf, and -inf are rejected before reduction

The oracle also reconciles scalar-online and block-online states against the same stable reference. It does not benchmark a GPU kernel; that experiment belongs to M4.3.

Loop 6 contract: one realized stochastic local operation

Loop 5 ended with a deterministic pre-dropout probability tensor. Loop 6 adds one random object, uses it to mix values, and then differentiates that exact realization. For one batch/head/query row, write:

The consumed routing weights and head output are:

The distinction among these objects is part of the training contract:

This loop owns the dense semantic equations, their tensor shapes, one fixed-RNG gradient trace, and the state required to reproduce that trace. It does not own how a tiled kernel visits query/key blocks, where those blocks live, or which gradient tile has exclusive hardware ownership. Those schedules remain in M4.3.

Historically, general dropout randomly removes units or connections during training to reduce co-adaptation. Srivastava et al. provide that general contract. The Annotated Transformer makes the attention-specific placement explicit in code: normalize scores, drop entries of p_attn, then multiply by value. That placement is a model choice, not an algebraic consequence of softmax.

Its element-wise variance is:

The row sum is a random variable:

Assuming independent Bernoulli draws across legal edges:

This variance depends on concentration. A diffuse row has small sum_j p_ij^2; a nearly one-hot row inherits almost the full rho/(1-rho) variance of one Bernoulli multiplier. If a row has n_i legal keys, its probability of losing every legal route is:

That event is legal under ordinary element-wise attention dropout. It produces a zero head-output row for this attention call; it is not the same condition as an all-masked softmax row, which Loop 4 rejects before normalization.

The output is also an unbiased random estimator:

Its covariance under independent edge draws is:

The covariance is value-dependent. Attention dropout therefore injects noise in the payload directions selected by the routing distribution; it is not just noise in a scalar probability diagnostic.

The Loop 6 oracle checks these equations with 200,000 fixed-seed draws for:

P = [0.1, 0.2, 0.3, 0.4]
rho = 0.2

analytic E[row sum]       = 1.000000
empirical E[row sum]      = 1.000564
analytic Var(row sum)     = 0.075000
empirical Var(row sum)    = 0.075137

analytic E[output]        = [0.800000, 0.975000]
empirical E[output]       = [0.798995, 0.975084]
Pr(all four routes drop)  = 0.001600

These are deterministic fixture results, not a benchmark or a claim that independence is the only useful dropout structure.

One realization is not a convex combination

Before dropout, P_i V lies in the convex hull of the visible values. After dropout, every coefficient remains nonnegative, but the coefficient sum is R_i, not necessarily one. A realized output is therefore a nonnegative, rescaled combination, not generally a convex combination.

This changes a useful translation identity. Add one constant vector c to every visible value row. Before dropout:

For one post-dropout realization:

Only in expectation does the shift return to c. In the fixed oracle trace, the two realized row sums are 1.004632 and 0.990504; adding c=[2,-1.5] consequently adds [2.009265,-1.506949] and [1.981008,-1.485756], respectively.

Alternative dropout placements are different estimators

“Apply dropout to attention” does not identify one operation. The dropped object, placement relative to softmax, sharing pattern, schedule, and fallback policy all change the training distribution.

DropAttention studies element-, column-, and contiguous-region structures over attention weights. DropKey moves stochastic deletion before attention normalization for vision Transformers. Scheduled DropHead drops whole heads and changes the rate across training. Their reported quality results belong to their tasks and settings; the reusable point here is that they randomize different semantic objects.

Renormalization is especially easy to mislabel as an implementation detail. Enumerating every keep mask for P=[0.1,0.2,0.7] and rho=0.5 gives:

Both renormalized variants bias the expected routing weights in this unequal row. Preserving row sum one and preserving the original expectation are not the same contract.

RNG state is part of the training input

For gradient derivation, M is held fixed. Backward differentiates the sampled program that ran in forward; it does not differentiate the Bernoulli sampling decision and must not draw a fresh mask. Reproducible execution therefore needs more than a nominal dropout probability:

global/model seed
device-specific generator identity
generator seed and counter/offset before the operation
logical batch, head, query, and key coordinates
draw-to-coordinate mapping
training/evaluation mode and dropout probability

A saved dense mask is sufficient but can be quadratic. A fused implementation may instead save compact generator state and replay each logical draw. Saving a seed alone is insufficient if a changed tile order, vector width, or counter advance maps that seed to different (batch,head,query,key) coordinates.

The visual is reused from M4.3 because the semantic objects are identical. M3.1 owns why replay is required. M4.3 owns how tiled forward/backward regenerate those objects without materializing the global probability or mask matrices.

PyTorch’s functional.dropout takes an explicit training flag. Its functional scaled_dot_product_attention has a sharper contract: it applies dropout whenever dropout_p > 0, so a module wrapper must pass 0.0 explicitly during evaluation.

Precision, dtype, and backend are implementation contracts

The mathematical score path does not specify its accumulation dtype. In a mixed-precision model, Q/K/V may be float16 or bfloat16, while an eager implementation computes softmax in float32 and casts the probabilities back:

attn_weights = torch.softmax(scores, dim=-1, dtype=torch.float32)
attn_weights = attn_weights.to(query.dtype)

Transformers v5.13.1 uses this pattern in its eager LLaMA attention path. The point is not that every backend must copy those two lines. Fused SDPA or FlashAttention kernels can choose their own accumulation and normalization strategy without materializing scores or p_attn as full tensors.

The local audit is:

Q/K/V dtype:
    checkpoint and kernel contract

position bias and additive mask:
    compatible device and floating dtype

softmax accumulation:
    eager or fused backend policy

returned attention/output dtype:
    caller and residual-stream contract

An unintended promotion from a float32 bias can increase memory traffic; an unsafe downcast can lose the intended mask magnitude; and a CPU mask added to CUDA scores fails before the attention equation is evaluated. Numerically close eager and fused outputs also need not be bitwise identical because accumulation and reduction orders can differ.

Fixed two-query training trace

The expectation equations do not reveal what one backward pass consumes. The Loop 6 fixture therefore freezes a rectangular call with two query rows, three key/value rows, value width two, dropout probability rho=0.25, and one realized keep mask:

P =
  [[0.753474, 0.246526, 0.000000],
   [0.257122, 0.191056, 0.551822]]

M =
  [[1, 0, 0],
   [0, 1, 1]]

Z = M / 0.75 =
  [[1.333333, 0.000000, 0.000000],
   [0.000000, 1.333333, 1.333333]]

Pdrop = P * Z =
  [[1.004632, 0.000000, 0.000000],
   [0.000000, 0.254741, 0.735763]]

row sums(Pdrop) = [1.004632, 0.990504]

With:

V = [[ 1.0, -1.0],
     [ 2.0,  0.5],
     [-0.5,  3.0]]

the value matmul produces:

O = Pdrop @ V =
  [[1.004632, -1.004632],
   [0.141601,  2.334659]]

This trace distinguishes three zeros that must not be conflated. P[0,2] is zero because the edge is illegal. Pdrop[0,1] is zero because a legal edge was dropped in this realization. Pdrop[1,0] is another dropped legal edge, even though its pre-dropout probability is nonzero. Only the first zero is a model support statement.

This distinction is useful when interpreting attention as retrieval. The head does not copy an existing value unless its routing is effectively one-hot. It normally interpolates among learned value directions.

Attention masks define a directed communication graph

For one layer, create directed edge j -> i when query row i may read key row j. The visibility mask is then an adjacency relation:

full encoder attention:
    complete directed graph among real tokens

causal attention:
    edges from current/past keys into each query

local window:
    bounded-neighborhood graph

packed attention:
    disconnected block-diagonal components

cross-attention:
    bipartite source-to-target read graph

After multiple layers, information can travel along graph paths even when one layer has no direct edge between two positions. This is the representational cost hidden by sparse edge counts: fewer edges can require more layers or special global routes to connect distant tokens.

A11 also retained a theoretical source on masked attention and token-rank collapse:

The figure contains Chinese exposition and summarizes one paper’s assumptions and bound. It should be read narrowly: under its stated conditions, repeated masked mixing on a sufficiently connected graph can drive token representations toward a lower-rank/common subspace, with graph radius and attention concentration affecting the bound. It is not proof that every trained Transformer inevitably becomes rank one.

Real blocks include residual connections, normalization, learned value maps, MLPs, finite depth, and data-dependent attention. A11’s useful contribution is the design question it raises:

mask topology affects more than FLOPs;
it changes information paths and repeated mixing dynamics.

This creates a three-way tradeoff for local/sparse masks:

fewer edges reduce pairwise work;
longer graph paths can delay global communication;
different connectivity/mixing can change smoothing and rank dynamics.

Those properties require model-level experiments and assumptions. M3.1 keeps the graph interpretation because it follows directly from visibility, while avoiding a universal convergence claim.

Backward path: every role receives a different signal

Backward differentiates the exact sampled forward program. Hold Z fixed and write:

The value matmul sends gradients down two branches. Its value gradient is weighted by the consumed stochastic matrix:

The gradient of the consumed weights is:

The fixed dropout multiplier then gates the direct probability gradient:

Dropout does not create a differentiable path into the sampled Bernoulli decision. Conventional attention dropout treats Z as realized training input. If a method learns a drop rate or a discrete gate with another estimator, that is a different optimization contract.

Softmax couples every legal score in one query row. Define the row contraction:

Substituting the preceding branches gives a second form that is useful in fused backward algorithms:

The softmax gradient is then:

For S=alpha QK^T, with alpha=1/sqrt(d_k), the score matmul returns:

The scale appears once in both branches. Omitting it, or applying it again inside G_S, can preserve all shapes while changing every Q/K gradient.

Fixed-mask backward trace

For the forward fixture above, freeze:

G_O = [[ 0.7, -0.4],
       [-0.2,  0.9]]

The two equivalent row contractions and the resulting score gradient are:

D from sum(P * G_P) = [1.105096, 2.072873]
D from sum(G_O * O) = [1.105096, 2.072873]

G_S =
  [[ 0.272434, -0.272434,  0.000000],
   [-0.532981, -0.383297,  0.916279]]

The maximum central-finite-difference error over Q, K, and V is 2.84e-10 in float64 with epsilon 1e-6. Reconstructing P from saved row log-sum-exp differs by at most 2.78e-17, and replaying the same Z changes the manual gradients by at most 1.11e-16. Drawing the fixture’s different fresh mask instead produces a maximum gradient mismatch of 0.771809. That is not stochastic tolerance; it is differentiation of a different program.

Dropped, masked, and all-dropped routes have different gradients

The fixed trace exposes a subtle consequence of softmax coupling:

legal edge (query 0, key 1) is dropped:
    Z[0,1] = 0
    direct G_P[0,1] = 0
    score G_S[0,1] = -0.272434   # generally nonzero

illegal edge (query 0, key 2) is masked:
    P[0,2] = 0
    score G_S[0,2] = 0

The dropped edge receives no direct gradient through its own consumed route, but its score still affects the softmax denominator that allocated pre-dropout mass to kept edges. A masked edge is absent from that denominator and has zero ideal score gradient.

If every legal edge in one row drops, then Pdrop_i=0 and O_i=0. For that row, the contributions to G_V, G_P, D_i, G_S, G_Q, and G_K are all zero. This well-defined stochastic event is different from asking softmax to normalize an empty legal support. The oracle checks both distinctions and also verifies:

sum_j G_S[i,j] = 0 for each legal finite row
masked G_S entries = 0
all-dropped row attention gradients = 0

This exposes the training semantics that the forward mnemonic hides:

V learns what payload helps the downstream loss.
Q learns what each output row should request.
K learns how each memory row should advertise itself.

An additive mask with true -inf gives blocked entries zero probability; their score gradients are correspondingly zero under the ideal softmax semantics. This does not mean a padded token can never receive gradients elsewhere in the network. Residual paths, embeddings, losses, and query-side computation have their own masking contracts.

Projection gradients reconnect the local attention call to the model

M3.1 receives projected tensors, but training eventually updates the projection weights owned by M3.2. If:

then:

Input-state gradients accumulate all roles in self-attention because the same X feeds Q, K, and V:

In cross-attention, target states receive the Q branch while source memory receives the K and V branches. This is the backward counterpart of the forward stream contract:

self-attention:
    one state stream receives Q + K + V gradient contributions

cross-attention:
    target stream receives Q contribution
    source-memory stream receives K + V contributions

The split matters when freezing an encoder, reusing source memory, adding adapters to selected projections, or debugging a detached memory tensor.

Requesting attention weights can change the execution path

An optimized kernel can produce O without writing the full P matrix to HBM. Asking for all attention weights may therefore:

force an eager or different backend;
materialize B*H*Q*K elements;
increase backward saved state;
change latency enough to invalidate a benchmark;
still return a tensor that is unsuitable as a causal explanation.

Debug runs should state whether weights are pre-dropout or post-dropout and whether they came from the same backend as normal execution. Production profiling should normally disable weight return unless the feature itself is a requirement.

PyTorch MultiheadAttention documents need_weights=False as the path that permits its optimized SDPA implementation. That is an API performance contract, not a mathematical statement that P does not exist. Conversely, a returned tensor named attn_weights does not by itself prove whether it is P, Pdrop, head-averaged P, or a tensor recomputed by another backend. The wrapper contract must say which object it returns.

An attention map is also only one part of the head function. Jain and Wallace showed counterexamples to treating learned attention weights as explanations; Wiegreffe and Pinter argued that stronger, task- and definition-specific tests can still make attention analytically useful. The bounded conclusion here is not that attention weights are meaningless. It is that a routing matrix alone omits V, W_O, residual state, downstream nonlinearities, and counterfactual behavior, so it must not be labeled a causal explanation without an explicit method and validation contract.

Training state saved by the local operation

Backward needs enough state to recover Q, K, V, the pre-dropout probability, and the exact sampled dropout multiplier. It does not require every one of those objects to be stored as a dense tensor. The saved-state choice is a time-memory-reproducibility contract:

One dense reconstruction route saves the row log-partition L_i and recomputes:

on legal entries. The Loop 6 oracle reconstructs the fixture probabilities with maximum error 2.78e-17. It then replays the fixed coordinate-wise dropout multiplier and reproduces the analytic gradients within 1.11e-16. Those are semantic conformance checks, not expected low-precision GPU tolerances.

The visual is reused at a different level of abstraction. M3.1 derives the dense operation that must be preserved. M4.3 owns the FlashAttention tile schedule, HBM/SRAM traffic, recomputation order, and gradient-write ownership.

This creates a three-way distinction:

forward mathematical state:
    Q, K, V -> exact output O

debug-visible state:
    optional full scores or probabilities

autograd saved state:
    backend-specific tensors/statistics needed for backward

Conflating them leads to claims such as “attention requires storing the full matrix” or “FlashAttention has no attention probabilities.” The mathematical probabilities exist; whether they are materialized, returned, or recomputed is an execution choice.

Loop 6 failure lab

Loop 6 executable gate

The evidence package turns this section into executable claims:

cd docs/numinens-evidence/m3-1-attention-score-path
python3 loop6_attention_oracle.py
python3 loop6_contract_audit.py
python3 -m unittest -v test_loop6_contract

Publication requires all of the following:

analytic and Monte Carlo dropout moments agree within declared tolerances;
fixed-mask manual Q/K/V gradients pass central finite differences;
the two D_i contractions agree;
masked, dropped, and all-dropped cases remain distinct;
same-mask replay agrees and fresh-mask replay demonstrably fails;
alternative renormalization is not mislabeled as unbiased inverted dropout;
source and visual provenance resolve locally;
Loops 1-5 section hashes remain frozen.

Loop 7 contract: reconcile source, equation, and program

Loop 7 freezes three descriptions of one operation:

source description:
    target rows query an encoder/source memory;
    source row j owns both key j and value j;
    source validity determines which memory rows are legal

equation description:
    S = alpha Q K^T
    S_bias = S + B
    S_mask = where(visible, S_bias, -inf)
    P = softmax_K(S_mask)
    O = (P * Z) V

program description:
    explicit tensors, views, broadcast adapters, work dtype,
    saved trace objects, assertions, and return semantics

The three must agree on row identity, not only dimensions. The fixture runs in evaluation mode, so its dropout multiplier is Z=1. Loop 6 already owns the stochastic training derivation and backward replay. Loop 7 records where that multiplier would enter the complete forward program without reopening its statistics.

Use the following firewall when reading the executable fixture:

The trace therefore starts before the M3.1 API only to prove lineage. Its projection matrices use equation orientation [Dmodel,Dout], so row vectors compute X @ W. A PyTorch nn.Linear checkpoint stores its weight in [Dout,Din]; M3.2 owns that orientation conversion.

The Harvard source and the safe reference have different jobs

A10 and A11 reconstruct the compact function from the Annotated Transformer:

d_k = query.size(-1)
scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)
if mask is not None:
    scores = scores.masked_fill(mask == 0, -1e9)
p_attn = scores.softmax(dim=-1)
if dropout is not None:
    p_attn = dropout(p_attn)
return torch.matmul(p_attn, value), p_attn

That code is an excellent equation spine. It is not a complete execution manifest. A defensive oracle makes assumptions explicit:

PyTorch’s torch.matmul broadcasts its leading batch dimensions. That behavior is useful, but it can hide a missing head or batch axis. The safe oracle requires Q/K/V leading axes to match exactly and permits broadcasting only for declared metadata such as visibility and finite bias. PyTorch’s broadcasting rules align trailing dimensions; they do not know that one axis means head and another means query.

The broad A11 reconstruction is useful because it shows the complete source path in one surface:

Source-local A11 figure. Read it critically: input/position construction is upstream, projections and multi-head merge belong to M3.2, and the multiplication symbol drawn between the score matrix and mask is schematic. Authoritative visibility is not score multiplication; it is support exclusion before softmax, as derived in Loop 4.

Fixed fixture: two target rows read three source rows

The fixture is a one-batch, one-head cross-attention call:

B=1, H=1, Q=2, K=3, Dmodel=3, Dk=2, Dv=2
target token IDs: [31, 47]
source token IDs: [11, 22, PAD]
source row IDs:   [source:0, source:1, source:2]

Token IDs are provenance labels, not arithmetic inputs to M3.1. Upstream embedding/block computation supplies target states Xq and source memory Xm:

The caller’s deterministic projection matrices are:

Stage 1: project and expose the head axis

The M3.2 caller computes:

After adding batch and head axes:

Q: [1,1,2,2]
K: [1,1,3,2]
V: [1,1,3,2]

Key row j and value row j still name the same source row. That paired-row invariant is more important than the coincidence Dk=Dv=2 in this fixture. Permuting K without applying the same source-row permutation to V leaves every matrix multiplication shape-valid and changes the retrieved content.

Stage 2: transpose the key view and compute compatibility

K.transpose(-2,-1) is a logical [1,1,2,3] view. The batched product is:

With alpha=1/sqrt(2)=0.70710678:

The output row axis comes from Q and the column axis comes from K. This is why the result is target-by-source [2,3], not a square matrix inferred from tensor rank.

Stage 3: add finite preference, then enforce source visibility

Use a query-specific finite bias:

The padded source row deliberately receives +7. This turns operation order into an executable test: no finite preference may revive an illegal edge.

The semantic visibility object begins as a source-row vector:

source_visible: [True, True, False]      shape [B,K] = [1,3]
visible:        [True, True, False]      shape [B,1,1,K] = [1,1,1,3]
broadcast view: two identical query rows shape [B,H,Q,K] = [1,1,2,3]

After bias:

After authoritative support exclusion:

The large third-column bias remains observable in biased_scores for audit, but it cannot enter the denominator.

Stage 4: expose the stable-softmax state

The legal row maxima are:

Subtracting them and exponentiating legal entries gives:

with denominators:

Therefore:

The fixture uses float64 so the published values and executable oracle can agree tightly. A production eager call may use float32 for the softmax work state and cast probabilities back. The torch.softmax dtype parameter makes that policy explicit; fused kernels may preserve the same mathematical row through a different internal schedule.

Stage 5: consume probabilities and mix source values

Evaluation mode fixes:

The per-head output is:

For the first target row:

0.3650 * value(source:0) + 0.6350 * value(source:1)
= 0.3650 * [1, 2]       + 0.6350 * [2, -1]
= [1.6350, 0.0951]

value(source:2) contributes exactly zero. The reason is semantic exclusion, not a small score and not attention dropout.

Every intermediate has both values and execution metadata

The complete local ledger is:

Full matrices belong in a tiny oracle trace, not in production telemetry. A runtime manifest normally records shape, dtype, device, stride, contiguity, materialization policy, and small checksums/statistics. Dumping attention matrices for real batches changes memory lifetime, may expose sensitive token relationships, and can force a different backend.

A traceable safe eager reference

The following function is intentionally more verbose than the code spine. It is a semantic oracle and debugging instrument, not the recommended production kernel:

def eager_attention_trace(
    q, k, v, *, visible, bias=None,
    dropout_multiplier=None, softmax_dtype=torch.float32,
):
    # Local contract: already projected/head-shaped tensors.
    assert q.ndim == k.ndim == v.ndim == 4
    assert q.shape[:2] == k.shape[:2] == v.shape[:2]
    assert q.shape[-1] == k.shape[-1]
    assert k.shape[-2] == v.shape[-2]
    assert q.device == k.device == v.device

    B, H, Q, Dk = q.shape
    K, Dv = k.shape[-2], v.shape[-1]
    expected_scores = (B, H, Q, K)
    trace = {"q": q, "k": k, "v": v}

    k_t = k.transpose(-2, -1)
    dot = torch.matmul(q, k_t)
    alpha = Dk ** -0.5
    scaled = dot * alpha

    if bias is None:
        bias_effective = torch.zeros_like(scaled)
    else:
        bias = bias.to(device=scaled.device, dtype=scaled.dtype)
        bias_effective = torch.broadcast_to(bias, expected_scores)
    biased = scaled + bias_effective

    visible = visible.to(device=scaled.device, dtype=torch.bool)
    visible_effective = torch.broadcast_to(visible, expected_scores)
    if not visible_effective.any(dim=-1).all():
        raise ValueError("every consumed query row needs at least one legal key")
    masked = biased.masked_fill(~visible_effective, -torch.inf)

    work = masked.to(softmax_dtype)
    row_max = work.max(dim=-1, keepdim=True).values
    shifted = work - row_max
    exp_shifted = torch.where(
        visible_effective, torch.exp(shifted), torch.zeros_like(shifted)
    )
    denominator = exp_shifted.sum(dim=-1, keepdim=True)
    pre_dropout_work = exp_shifted / denominator
    pre_dropout = pre_dropout_work.to(v.dtype)

    if dropout_multiplier is None:
        dropout_multiplier = torch.ones_like(pre_dropout)
    else:
        assert dropout_multiplier.shape == expected_scores
        dropout_multiplier = dropout_multiplier.to(
            device=pre_dropout.device, dtype=pre_dropout.dtype
        )
    consumed = pre_dropout * dropout_multiplier
    output = torch.matmul(consumed, v)

    trace.update({
        "k_t": k_t,
        "dot": dot,
        "alpha": alpha,
        "scaled": scaled,
        "bias_effective": bias_effective,
        "biased_scores": biased,
        "visible_effective": visible_effective,
        "masked_scores": masked,
        "row_max": row_max,
        "shifted": shifted,
        "exp_shifted": exp_shifted,
        "denominator": denominator,
        "pre_dropout": pre_dropout,
        "dropout_multiplier": dropout_multiplier,
        "consumed": consumed,
        "output": output,
    })
    assert output.shape == (B, H, Q, Dv)
    return output, trace

Two choices are deliberate.

First, Q/K/V do not broadcast. torch.matmul could broadcast leading axes, but a safe reference rejects that convenience because physical head mapping is an M3.2 decision. Visibility and finite bias may broadcast because their manifests explicitly declare that behavior.

Second, torch.broadcast_to returns an expanded view. A broadcast head/query axis can have stride zero: several logical coordinates name the same storage. The trace never mutates bias_effective or visible_effective in place. A later implementation that writes through an expanded view is not equivalent to reading broadcast metadata.

Source identity survives every legal transformation

An end-to-end debugger should carry row labels beside tensor values:

query row labels:
    [target:0, target:1]

key/value row labels:
    [source:0, source:1, source:2]

score cell [i,j]:
    compatibility(target:i, source:j)

probability cell [i,j]:
    normalized legal routing weight from target:i to source:j

output row [i]:
    mixture of source value rows for target:i

Transpose does not reorder source identities; it changes how K participates in the matrix product. Broadcasting the source mask does not create new source rows; it applies one source-validity decision to several heads/query rows. Softmax does not move identities; it normalizes each fixed query row over the key axis. Value mixing removes the key axis by contracting each probability column with the correspondingly indexed value row.

This yields a useful conservation rule:

before P @ V:
    key axis j and value row j must denote the same source object

after P @ V:
    key axis is contracted away;
    query row identity and value feature identity remain

The rule catches a data-pipeline bug that pure tensor algebra cannot: if K is sorted, sharded, cached, or gathered under one index map and V under another, the calculation is numerically valid and semantically corrupt.

Self-attention, cross-attention, and cached self-attention are caller variants

The safe function does not branch on an attention_type string. The caller creates different source objects and coordinates:

The full numeric fixture is cross-attention because Q=2 and K=3 make axis ownership visible. A rectangular cached-self call can have the same score shape while representing a different graph. Shape does not determine whether the source is encoder memory or history from the same stream.

Reconciliation assertions

The fixture checks values at stage boundaries rather than trusting only the final output:

output, t = eager_attention_trace(...)

assert t["k_t"].shape == (1, 1, 2, 3)
assert t["dot"].shape == (1, 1, 2, 3)
assert t["visible_effective"].shape == t["dot"].shape
assert torch.isneginf(t["masked_scores"][..., 2]).all()
assert (t["pre_dropout"][..., 2] == 0).all()
torch.testing.assert_close(
    t["pre_dropout"].sum(dim=-1),
    torch.ones((1, 1, 2), dtype=t["pre_dropout"].dtype),
)
torch.testing.assert_close(output, t["consumed"] @ t["v"])

The evidence oracle additionally compares:

vectorized safe trace
    == scalar row-by-row reference
    == Harvard-style translated-mask reference

output row i
    == sum_j probability[i,j] * value[j]

masked source column
    == zero probability and zero value contribution

The Harvard comparison is restricted to its shared contract: one head-shaped Q/K/V call, no empty legal row, the same finite bias inserted before its mask step, dropout disabled, and 1=legal/0=blocked mask adaptation. Agreement does not prove that -1e9 is a universal mask policy or that implicit dtype/backend choices are safe.

Loop 7 failure lab

Loop 7 executable gate

The deterministic evidence package runs the full fixture and freezes the source/visual contract:

cd docs/numinens-evidence/m3-1-attention-score-path
python3 loop7_source_to_tensor_oracle.py
python3 loop7_contract_audit.py
python3 -m unittest -v test_loop7_contract

Publication requires:

every projected row retains target/source provenance;
every intermediate shape and numeric value matches the fixture;
vectorized, scalar, and Harvard-style references agree;
the illegal high-bias source row remains excluded;
K/V row permutations fail the semantic oracle despite valid shapes;
only declared metadata broadcasts;
the M3.1/M3.2 ownership firewall remains explicit;
Loops 1-6 section hashes remain frozen.

Loop 7 ended with a defensive eager function whose intermediate values can be inspected. Production models rarely call that function directly. They project and position Q/K/V, adapt a semantic visibility relation to a framework API, ask a dispatcher for an eligible implementation, and return only the objects the caller requested.

That indirection is useful, but it creates a new class of bugs. Two runtime paths may implement the same equation while disagreeing about boolean polarity, causal alignment, dropout mode, GQA head mapping, returned weights, or whether a requested backend actually ran.

Owner question. How does the exact eager operation from Loop 7 cross a framework and model-runtime boundary without allowing API defaults or backend dispatch to redefine its support, scale, stochastic state, or output meaning?

Loop 8 contract: preserve semantics across four runtime layers

The word SDPA is overloaded. It can name the mathematical operation, the PyTorch functional API, or whichever kernel that API selects. Loop 8 keeps four objects separate:

The contract is:

same semantic manifest
    -> backend-specific adapter
    -> framework call
    -> local output O with declared tolerance

never:
framework default
    -> inferred model semantics

This section is version-scoped to the PyTorch 2.13 functional SDPA and MultiheadAttention documentation, plus the Hugging Face Transformers attention-interface and LLaMA implementation snapshots linked below. These are external implementation contracts, not timeless properties of every framework version. The manifest must record actual package versions and source revisions used by a run.

One equation, one functional call, several hidden decisions

PyTorch’s functional entry point has the shape:

F.scaled_dot_product_attention(
    query,
    key,
    value,
    attn_mask=None,
    dropout_p=0.0,
    is_causal=False,
    scale=None,
    enable_gqa=False,
)

The version-scoped documentation gives the logical reference program:

where scale=None means:

and E=query.size(-1). Passing scale=alpha supplies the multiplier itself; it is not a denominator and it is not multiplied by the default a second time.

The documented tensor axes are more general than the four-axis fixture:

Q: [N, ..., Hq, L,  E]
K: [N, ..., H,  S,  E]
V: [N, ..., H,  S, Ev]
O: [N, ..., Hq, L, Ev]

N,... are batch-like axes, Hq is the query-head count, H is the key/value head count, L is query length, and S is key/value length. The score object is logically [N,...,Hq,L,S]. The API may broadcast an attention mask to that shape, but Loop 2’s strict contract still rejects accidental broadcasting of Q, K, or V batch axes. Framework permissiveness is not evidence that the model intended replication.

The returned tensor alone does not reveal:

which mask encoding was consumed;
whether causal alignment was upper-left or cache-aligned;
whether K/V heads were repeated logically;
whether dropout was active;
which implementation executed;
whether a quadratic probability tensor was materialized;
which numerical reduction order was used.

Those are runtime-manifest fields, not facts to reverse-engineer from O.

The visibility adapter is a polarity firewall

M3.1’s canonical relation remains:

For a dense additive encoding, lower it exactly once:

The adapter matrix is deliberately explicit:

PyTorch explicitly documents the boolean polarity difference between functional SDPA and MultiheadAttention. This is not a naming inconvenience. The following two tensors encode the same support:

semantic / SDPA bool visible:      MHA bool blocked:
[[1, 0, 0],                       [[0, 1, 1],
 [1, 1, 0],                        [0, 0, 1],
 [1, 1, 1]]                        [0, 0, 0]]

A float 1/0 matrix is neither representation. Under additive semantics, 0 keeps a score and 1 rewards it. It masks nothing.

def lower_visibility_for_sdpa(visible, *, q, k):
    score_shape = (*q.shape[:-2], q.size(-2), k.size(-2))
    visible = torch.broadcast_to(visible, score_shape)
    if not visible.any(dim=-1).all():
        raise ValueError("all-masked query row")
    return visible  # SDPA bool polarity: True participates


def lower_visibility_for_mha(visible):
    return ~visible  # MHA bool polarity: True blocks

The real adapter must also validate dtype, device, batch/head expansion, and the all-masked-row policy frozen in Loop 4. ~mask is not a complete adapter.

Causal is a coordinate relation, not a triangular adjective

For a square prefill with query and key rows beginning at the same absolute position, causal visibility is:

For cached decoding, query row i can begin at absolute offset q_offset:

If L=2, S=5, and the two query rows are the final two rows of the key history, then q_offset=S-L=3. The desired lower-right relation is:

             key j ->  0 1 2 3 4
query abs 3 / row 0:  1 1 1 1 0
query abs 4 / row 1:  1 1 1 1 1

An upper-left triangle instead means:

             key j ->  0 1 2 3 4
query row 0:           1 0 0 0 0
query row 1:           1 1 0 0 0

Both are lower-triangular-looking relations. Only one uses the caller’s cache coordinates. PyTorch 2.13 documents is_causal=True as upper-left alignment for a non-square matrix. Its causal_lower_right(L,S) bias expresses the second relation. Therefore:

square prefill:
    is_causal=True may encode the complete support

rectangular cached call:
    do not infer cache alignment from is_causal=True
    provide a validated lower-right/cache-position relation or backend adapter

custom padding/packing/window constraints:
    compile the full semantic relation once
    do not also ask another layer to add an assumed causal triangle

The safest runtime object carries coordinates, not only a boolean named is_causal:

@dataclass(frozen=True)
class AttentionCoordinates:
    q_positions: Tensor  # [B, L]
    k_positions: Tensor  # [B, S]
    q_segments: Tensor | None
    k_segments: Tensor | None


visible = (
    coordinates.k_positions[:, None, None, :]
    <= coordinates.q_positions[:, None, :, None]
)

Packing, windows, prefix regions, and padding compose with this relation before backend lowering. M3.9 owns the packed-example metadata; M3.10 owns the request-local logical cache-position/frontier transition. M5.1 owns persistent publication/lifecycle, and M5.3 owns physical cache mutation. M3.1 owns checking that the concrete relation arriving at the score boundary matches those coordinates.

GQA changes head mapping, not the local probability axis

Functional SDPA can receive more query heads than key/value heads:

Q: [B, Hq, L, E]
K: [B, Hkv, S, E]
V: [B, Hkv, S, Ev]

For equal-size groups, require:

With Hq=4 and Hkv=2:

query head:  0 1 2 3
KV head:     0 0 1 1

The documented enable_gqa=True reference repeats K and V over the head axis before the same score operation. That establishes semantic equivalence: