M3.4 — Rotary Position Embedding

0x00 Overview And Background

M3.3 put coordinates into the state rows:

absolute position:
    x_i = content_i + p_i

M3.3 also put pairwise geometry into the attention logit matrix:

relative bias:
    scores_ij = q_i k_j^T / sqrt(d_head) + b_ij

RoPE uses a third implementation boundary:

RoPE:
    q_i, k_i are projected first
    q_i, k_i are rotated by their positions
    scores are computed from the rotated Q/K vectors

In code-object terms:

x
    -> q, k, v projections
        -> rotate q and k
            -> q_rot @ k_rot.T
                -> mask
                    -> softmax
                        -> weighted sum of v

RoPE is not:

x = x + position_vector

and it is not:

scores = scores + rope_bias

It is:

q_rot, k_rot = apply_rope(q, k, position_ids_or_freqs)
scores = q_rot @ k_rot.transpose(-2, -1) / math.sqrt(d_head)

The overview figure from the handwritten RoPE article is the right mental model: Q and K are transformed by position-dependent rotations before attention scoring. V is not rotated.

The RoFormer paper frames the same idea at the method level: absolute positions are encoded with rotation matrices while the self-attention interaction gains explicit relative-position dependency.

The owner question is:

Why does an absolute rotation of each Q/K vector make their dot product depend on relative displacement, and what exact tensor, precision, pairing, and cache contracts preserve that identity in an implementation?

M3.4 owns the derivation and the vanilla operator. M3.5 owns modifications whose purpose is context extension. M4 owns fused kernel execution, and M5 owns multi-request cache allocation. Keeping those boundaries explicit prevents a RoPE config survey from replacing the actual rotation proof.

---

Background knowledge: sinusoidal coordinates can act as rotations

Sinusoidal position encoding established the useful idea of assigning different frequency bands to pairs of model dimensions. RoPE keeps a frequency ladder but changes where it acts. Each two-dimensional Q/K pair is treated as a plane and rotated by an angle determined by position.

absolute coordinate:
    position m selects rotation R_m

pairwise interaction:
    R_m^T R_n reduces to a rotation determined by n-m

implementation:
    complex multiplication or an equivalent real rotate-half operation

This is why RoPE is often described as a bridge between absolute and relative position: absolute indices choose the rotations, while the dot product exposes their relative offset. The implementation still depends on dimension pairing, frequency base, layout, and position continuity.

Historical pressure: the positional object kept moving

RoPE is easier to understand when it is placed in the sequence of design problems that produced it. The original Transformer made parallel token rows distinguishable by adding sinusoidal coordinates to their input states. That solved the missing-address problem, but position entered before Q/K/V projection and mixed with content throughout the residual stream.

Relative-position methods then moved some geometry into the pairwise interaction. Shaw-style relative representations, for example, attach a displacement-dependent object to a query-key pair. M3.3 owns that taxonomy and its score/value tensors. The design question that led to RoPE is narrower:

Can absolute token addresses transform projected Q and K so that their inner product exposes relative displacement without storing a learned table for every possible pair?

The 2021 RoFormer paper answered with position-indexed rotations. LLaMA later made the operator familiar in decoder-only language models, while GPT-J, GPT-NeoX, Transformers, and FlashAttention made clear that the mathematical operator is inseparable from a pairing and layout convention. A runtime that caches already-rotated K turns those conventions into persistent state.

This chronology distinguishes two names that are often collapsed:

RoFormer:
    the model and paper that introduced and evaluated the method

RoPE:
    the reusable position-dependent Q/K rotation operator

The article studies the second object. It uses RoFormer as the defining primary source, A17 as the handwritten explanation inventory, and pinned production implementations as evidence for layouts and runtime contracts. Model-quality or speed claims remain source-reported rather than properties of every RoPE model.

The ownership firewall

The same position index appears in several neighboring articles, but each owns a different object:

Article	Object it owns	What M3.4 consumes or defers
M3.1	projected Q/K and the score path	M3.4 starts after projection and ends before masking/softmax
M3.2	query-head and KV-head layout	M3.4 must rotate each existing head layout without redefining MHA/GQA/MQA
M3.3	position addresses and alternative positional objects	M3.4 consumes the integer or logical coordinate chosen there
M3.4	vanilla RoPE derivation, pair layout, frequency state, Q/K rotation, and rotated-cache identity	this article’s owner surface
M3.5	transformations intended to extend context	PI, NTK scaling, YaRN, LongRoPE, and extension evaluation stay there
M3.10	complete single-request prefill/decode transition	M3.4 supplies only the rotary coordinate and cache-format invariant
M4/M5	fused execution and serving-scale cache management	kernel timing, allocation, eviction, and prefix scheduling are deferred
X1.1-X1.3	modality-specific unit geometry, fusion, and mixed-origin runtime state	M3.4 may compare changed rotary objects but not teach the multimodal pipelines

This boundary matters because a vanilla RoPE derivation cannot establish long-context quality. The formula can generate an angle for any integer, but M3.5 must ask whether those angles are in distribution and whether a particular scaling/training policy preserves quality.

Claim classes used in this article

The evidence surface is deliberately split:

source-local:
    stated or illustrated in A17

primary-paper verified:
    supported by RoFormer or another named paper

official-code verified:
    tied to a pinned implementation revision

derivation:
    follows from assumptions written next to the equation

engineering interpretation:
    a runtime or debugging consequence of those contracts

uncertain / source-reported:
    empirical or general claims not implied by the local algebra

This is especially important for two recurring statements. Orthogonal rotation does preserve each Q/K pair norm. A universal monotonic decay of every attention score with distance does not follow from that fact; the learned content coefficients and frequency mixture matter. The spectral section will audit that distinction rather than inherit it as folklore.

0x01 The local object: Q/K before attention scoring

M3.4 starts after Q/K/V projection. The input to this article is not raw text, token IDs, embedding rows, or additive positional encodings. The local objects are Q and K inside an attention layer:

q = project_q(x)  # [B,n,h,d_head] or [B,h,n,d_head]
k = project_k(x)  # [B,n,kv_heads,d_head] or [B,kv_heads,n,d_head]
v = project_v(x)  # same sequence axis as k

q_rot, k_rot = apply_rope(q, k, position_ids_or_freqs)

scores = torch.matmul(q_rot, k_rot.transpose(-2, -1)) / math.sqrt(d_head)
scores = scores.masked_fill(attention_mask == 0, -float("inf"))
attn = torch.softmax(scores, dim=-1)
out = torch.matmul(attn, v)

The important invariant is:

q_rot.shape == q.shape
k_rot.shape == k.shape

RoPE changes the coordinate frame of each Q/K vector. It does not change batch size, sequence length, number of heads, or head width.

The layout matters. Many explanations use:

[B,n,h,d_head]

because LLaMA-style code often projects to [batch, seq_len, heads, head_dim] before transposing for attention. Other libraries use:

[B,h,n,d_head]

because score computation prefers the head axis before the sequence axes. The same RoPE formula can work in either layout, but the cos, sin, or freqs_cis broadcast axes change. Hugging Face’s LLaMA implementation exposes this as unsqueeze_dim: one value makes [B,n,d] position tables broadcast over [B,h,n,d]; another fits [B,n,h,d].

MQA and GQA add one more source of bugs:

q heads:        h
k/v heads:      kv_heads

K can have fewer heads than Q. RoPE still rotates Q and K over their per-head d_head coordinates before scoring or before K is repeated for grouped-query attention. The FlashAttention rotary implementation treats these packed QKV and MQA/GQA layouts as part of the rotary interface, not as a harmless wrapper detail.

---

0x01A The immutable rotary call manifest

The function signature apply_rope(q, k, position_ids) hides most of the operator. Two calls can receive the same Q/K shapes and still implement different models. A reproducible call therefore needs an immutable manifest, not a loose list of defaults.

checkpoint:
  model_revision: "content-addressed model revision"
  config_sha256: "hash of normalized model config"
  implementation_revision: "pinned runtime source revision"

tensor:
  q_layout: "B,Hq,N,D"
  k_layout: "B,Hkv,N,D"
  q_heads: 8
  kv_heads: 2
  head_dim: 16
  rotary_dim: 8
  pair_layout: "half_split"   # or interleaved

frequency:
  rope_type: "default"
  rope_theta: 10000.0
  exponent_denominator: "rotary_dim"
  attention_factor: 1.0

position:
  semantic_origin: 0
  position_ids_shape: "B,N"
  packed_policy: "continuous"

numeric:
  trig_compute_dtype: "float32"
  qk_input_dtype: "bfloat16"
  qk_output_dtype: "bfloat16"
  table_representation: "half_width_cos_sin"

cache:
  key_storage: "rotated"
  cache_slot_semantics: "logical_token_position"

The exact field names vary by framework. The semantic fields cannot. In the pinned Transformers source, LlamaRotaryEmbedding derives inv_freq and an attention factor from config, computes cosine/sine from position_ids, and passes those tensors into each attention layer. In the pinned Meta LLaMA source, precompute_freqs_cis creates a complex table, the model slices that table using start_pos, and apply_rotary_emb rotates Q/K before the K cache write. FlashAttention exposes still more of the call identity directly: interleaved, seqlen_offsets, rotary_dim, packed-sequence offsets, and Q-head versus K-head counts.

Those are not three contradictory definitions of RoPE. They are three interfaces that must instantiate the same checkpoint-specific operator.

Source-to-tensor derivation

Let

B: batch rows
N: query/key positions in this call
Hq: query heads
Hkv: key/value heads
D: head width
R: rotary_dim, with R even and R <= D

The normalized manifest derives the following objects:

Q:             [B,Hq,N,D]
K:             [B,Hkv,N,D]
inv_freq:      [R/2]
position_ids:  [B,N]
phase:         [B,N,R/2]
cos_half/sin_half:
               [B,N,R/2]
cos/sin after pairing expansion:
               broadcastable to Q[..., :R] and K[..., :R]
Q_rot:         [B,Hq,N,D]
K_rot:         [B,Hkv,N,D]

For default RoPE, the frequency and phase construction is

$$\omega_i = \theta^{-2i/R}, \qquad i=0,\ldots,R/2-1,$$ $$\Phi_{b,n,i} = p_{b,n}\omega_i.$$

Here theta is rope_theta, and p is the semantic position selected by the position contract. phase is an outer product over position and frequency, not over attention heads. Heads share the basis unless the architecture says otherwise; Q and K may have different head counts while sharing the same per-head pair coordinates.

Why the exponent denominator belongs in the manifest

A familiar implementation is

inv_freq = 1.0 / (
    rope_theta
    ** (torch.arange(0, rotary_dim, 2, dtype=torch.float32) / rotary_dim)
)

Replacing rotary_dim in the denominator with head_dim is harmless only when the two are equal. Under partial rotary, the replacement changes every frequency except the first while preserving [R/2]. The table broadcasts, Q/K retain their shape, and no type checker detects the model change.

The same applies to the pair layout. half_split and interleaved both rotate R/2 planes, but they assign scalar channels to different planes. They become equivalent only if Q/K projection rows are permuted consistently with the checkpoint conversion. Toggling a runtime flag alone is not a conversion.

Cache identity is a product of the call manifest

If K is rotated before cache write, the bytes already encode:

pair layout
rotary_dim
frequency basis and scaling
semantic position origin and IDs
Q/K channel ordering inherited from the checkpoint
numeric path used to produce the stored values

A cache manager can hash the normalized semantic subset into a rotary compatibility digest:

rope_cache_key = sha256(canonical_json({
    "model_revision": model_revision,
    "config_sha256": config_sha256,
    "pair_layout": pair_layout,
    "rotary_dim": rotary_dim,
    "rope_type": rope_type,
    "rope_theta": rope_theta,
    "attention_factor": attention_factor,
    "semantic_origin": semantic_origin,
    "packed_policy": packed_policy,
    "key_storage": key_storage,
}))

That digest is an engineering admission guard, not part of the RoFormer formula. Its purpose is to make hidden compatibility assumptions explicit. A digest match is necessary for reuse under this contract; it is not sufficient if tokenization, attention visibility, model weights, or other cache-producing state differs. M5 owns that larger prefix-cache key.

The executable manifest oracle in the article evidence rejects odd or oversized rotary_dim, wrong position-row lengths, unknown pair layouts, and invalid cache formats. It also proves that changing only base, pairing, or position origin changes the cache digest even when all tensor shapes remain identical.

---

0x02 Target equation: absolute rotations produce relative offsets

For one query at position $m$ and one key at position $n$, RoPE applies separate rotations:

$$q_m' = R_m q_m$$ $$k_n' = R_n k_n$$

Then attention scores use:

$${q_m'}^\top k_n'$$

Expand the dot product:

$${q_m'}^\top k_n' = (R_m q_m)^\top (R_n k_n)$$ $$= q_m^\top R_m^\top R_n k_n$$

For rotation matrices:

$$R_m^\top R_n = R_{n-m}$$

So:

$${q_m'}^\top k_n' = q_m^\top R_{n-m} k_n$$

This is the core RoPE identity. The absolute positions $m$ and $n$ select two rotations separately, but the dot product contains a relative rotation between them.

The sign can appear as $n-m$ or $m-n$ depending on whether the derivation uses row vectors, column vectors, transposes, or a different rotation convention. The implementation does not get to be casual about this. A sign flip keeps the same tensor shape while changing the attention geometry.

Rotation matrices are orthogonal. They preserve vector norm, but they change the angular relation used by the Q/K dot product. That is the local mechanical reason RoPE belongs on Q and K.

Source: RoFormer: Enhanced Transformer with Rotary Position Embedding, Figure 1. The upper panel is one 2D pair; the lower panel repeats the operation across dimension pairs. It is a geometric implementation sketch, not a tensor-layout specification: [B,n,h,d_head] versus [B,h,n,d_head], pairing order, and partial rotary_dim remain checkpoint contracts.

---

0x02A Deriving the rotation from the desired attention property

The source A17 article begins from a functional requirement rather than from rotate_half. Let content at positions m and n be x_m and x_n. We want query and key maps such that

$$\langle f_q(x_m,m), f_k(x_n,n)\rangle = g(x_m,x_n,m-n).$$

The score may depend on both content vectors, but its positional part should depend on relative displacement rather than two unrelated absolute IDs. A simple family is

$$f_q(x_m,m)=R_mW_Qx_m, \qquad f_k(x_n,n)=R_nW_Kx_n,$$

where R_m is an orthogonal representation of translation. The dot product is

$$(R_mW_Qx_m)^T(R_nW_Kx_n) =x_m^TW_Q^TR_m^TR_nW_Kx_n.$$

To depend only on n-m, the rotations must satisfy

$$R_m^TR_n=R_{n-m}.$$

The ordinary 2D rotation group has exactly this property because

$$R(\alpha)^TR(\beta) =R(-\alpha)R(\beta) =R(\beta-\alpha).$$

Choosing alpha=m*theta and beta=n*theta therefore gives the relative-angle identity. Repeating the construction with several theta_i values creates the block-diagonal multi-frequency operator used by RoPE.

What is assumed before the functional equation is solved

The desired property alone does not uniquely imply RoPE. We first choose a factorized family in which content is projected linearly and position acts by a shared operator afterward:

$$f_q(x,m)=\rho(m)W_Qx, \qquad f_k(y,n)=\rho(n)W_Ky.$$

This ansatz excludes other valid relative-position mechanisms: additive score biases, learned relative vectors, content-position cross terms, and nonlinear position-conditioned projections. M3.3 owns those alternatives. Under the factorization above, the query-key interaction becomes

$$x^TW_Q^T\rho(m)^T\rho(n)W_Ky.$$

Now the positional middle factor must be a function of n-m if a common shift of the pair is to cancel for arbitrary projected content. A sufficient contract is that rho be an orthogonal representation of integer translation.

From the integer translation group to one generator

Positions form the additive group of integers. A representation must satisfy

$$\rho(0)=I,$$ $$\rho(a+b)=\rho(a)\rho(b),$$

and therefore

$$\rho(-m)=\rho(m)^{-1}.$$

Every integer representation is determined by the image of one step. Let

$$U=\rho(1).$$

Repeated composition gives

$$\rho(m)=U^m$$

for positive positions, while negative positions use inverse powers. If U is orthogonal, then

$$\rho(m)^{-1}=\rho(m)^T=\rho(-m).$$

The score factor now reduces without an extra assumption:

$$\rho(m)^T\rho(n) =\rho(-m)\rho(n) =\rho(n-m).$$

This is the algebraic source of the relative-offset identity. Orthogonality also gives norm preservation, but norm preservation is not what creates relativity; the group composition law does that. Orthogonality makes the inverse equal the transpose used by the Euclidean dot product.

Why two-dimensional rotation blocks appear

A real orthogonal matrix can be put, by an orthogonal change of basis, into independent blocks: real eigenvalues +1 or -1, and two-dimensional rotation planes associated with complex-conjugate eigenvalues. RoPE selects the 2D rotation blocks. For pair i, the one-step generator is

$$U_i=R(\theta_i) = \begin{bmatrix} \cos\theta_i & -\sin\theta_i \\ \sin\theta_i & \cos\theta_i \end{bmatrix}.$$

Then

$$U_i^m=R(m\theta_i).$$

Putting R/2 such planes side by side yields

$$\rho(m) = \bigoplus_{i=0}^{R/2-1}R(m\theta_i).$$

The direct sum explains three implementation facts at once:

rotary_dim must be even:
    every active scalar belongs to one 2D plane

different pairs can use different frequencies:
    block composition remains independent

pair order is checkpoint state:
    it chooses which Q/K scalar channels occupy each invariant plane

The +1 block would be position independent. A -1 block alternates sign on integer steps. Standard RoPE instead uses a ladder of nonzero rotation angles; that is a design choice, not a theorem that other orthogonal blocks are impossible.

Shared bases are required by this factorization

Suppose Q uses angle ladder theta_i^q and K uses theta_i^k. One pair then contains

$$R(-m\theta_i^q)R(n\theta_i^k) = R(n\theta_i^k-m\theta_i^q).$$

Unless the two frequencies agree, this cannot generally be written as a function of n-m alone. The same problem appears if Q and K use different position origins or scaling rules. It is therefore insufficient for two frequency tensors merely to have shape [R/2]; they must instantiate the same representation.

Sign conventions are representations too

This article treats Q/K pair vectors as columns and defines

$$q_m'=R(m\theta)q_m, \qquad k_n'=R(n\theta)k_n.$$

The score therefore contains R((n-m)theta). Code that stores row vectors, defines clockwise-positive rotation, or conjugates the opposite complex operand may write (m-n)theta. Those formulas can be equivalent when every convention changes together. Mixing only one sign, transpose, or conjugation produces a different operator with unchanged shapes.

The executable group oracle verifies identity, composition, inverse/transpose, common-shift invariance, and block-direct-sum reduction. It also includes a counterexample showing that mismatched Q/K frequencies fail the common-shift test. That counterexample is important: the oracle checks the theorem’s assumptions instead of merely reproducing its expected answer.

This derivation explains why several tempting alternatives are not equivalent:

add one absolute scalar to Q and K:
    not norm preserving; cross terms do not reduce to an offset representation

rotate only Q:
    score depends on m but has no R_m^T R_n cancellation

rotate Q and K with different frequency bases:
    R_m^T R_n is no longer one shared relative representation

rotate V:
    changes the payload after attention weights are selected; unnecessary for
    the relative score identity

Translation of a pair leaves its RoPE score unchanged

If both positions are shifted by s, then

$$(R_{m+s}q)^T(R_{n+s}k) =q^TR_{m+s}^TR_{n+s}k =q^TR_{n-m}k.$$

The common shift cancels. The content vectors can still differ; the statement is only that the coordinate contribution is translation invariant.

Expanding one complex pair into real score terms

Represent a query pair as q0 + i*q1, a key pair as k0 + i*k1, and let delta=(m-n)*theta. The real part of the complex product expands to

$$\operatorname{Re}\left[(q_0+iq_1)(k_0-ik_1)e^{i\delta}\right]$$ $$= (q_0k_0+q_1k_1)\cos\delta +(q_0k_1-q_1k_0)\sin\delta.$$

The first coefficient is the ordinary pair dot product. The second is an oriented cross term. RoPE therefore does more than multiply an ordinary score by cos(delta); both cosine and sine components are content dependent.

The skew generator behind rotate_half

Define the quarter-turn matrix

$$J= \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}.$$

It satisfies

$$J^T=-J, \qquad J^2=-I.$$

The matrix exponential reduces to

$$e^{\phi J} = \cos\phi\,I+\sin\phi\,J =R(\phi).$$

This identity is the mathematical content of the real implementation

y = x * cos + rotate_half(x) * sin

provided rotate_half(x) applies J to every intended 2D pair. For an interleaved pair (x0,x1), that action is (-x1,x0). In a half-split layout, the same planes are stored as two channel halves, so the slicing looks different even though each logical pair still receives J.

Complex multiplication is a view of the same real map

Map the real pair to

$$z=x_0+i x_1.$$

Then

$$z e^{i\phi} =(x_0\cos\phi-x_1\sin\phi) +i(x_0\sin\phi+x_1\cos\phi).$$

Reading real and imaginary parts gives exactly R(phi)x. The equivalence has three preconditions:

the last real axis is paired as the complex view expects
the sign of phi matches the real rotation convention
the query-key product conjugates the operand implied by the derivation

view_as_complex enforces a physical adjacent-pair view. A checkpoint whose logical planes are half-split must be permuted or handled by a half-split real operator before that view is valid.

The score is one bounded sinusoid with learned phase

For one pair, define

$$C=q_0k_0+q_1k_1,$$ $$S=q_0k_1-q_1k_0.$$

The relative score is

$$s(\delta)=C\cos\delta+S\sin\delta.$$

Using the two-dimensional Lagrange identity,

$$C^2+S^2 =(q_0^2+q_1^2)(k_0^2+k_1^2) =\|q\|_2^2\|k\|_2^2.$$

Therefore there is a content-dependent phase psi such that

$$s(\delta) =\|q\|_2\|k\|_2\cos(\delta-\psi),$$

where

$$\psi=\operatorname{atan2}(S,C).$$

This form makes two limits precise. The pair score is bounded by ||q|| ||k||, consistent with orthogonality and Cauchy-Schwarz. It is not a fixed cosine of distance: learned Q/K choose both amplitude and phase for every content pair. Summing many frequency planes produces a learned trigonometric mixture, not a universal monotone decay curve.

Forward and backward of one rotary plane

Let

$$y=R(\phi)x.$$

The input Jacobian is

$$\frac{\partial y}{\partial x}=R(\phi).$$

For upstream gradient g, reverse mode gives

$$\bar{x}=R(\phi)^Tg=R(-\phi)g.$$

The angle derivative follows from dR/dphi = J R = R J:

$$\frac{\partial y}{\partial\phi}=Jy,$$ $$\bar{\phi}=g^TJy.$$

If phi=p*omega, then

$$\bar{\omega}=p\,\bar{\phi}.$$

Standard fixed RoPE does not optimize omega, but this derivative explains why learned-frequency variants accumulate position-scaled gradients. It also shows why frequency computation deserves a higher-precision boundary even when Q/K outputs are returned to a low-precision model dtype.

The pair oracle compares matrix, real rotate_half, and Python complex paths; checks C^2+S^2; and compares analytic input/angle gradients with central finite differences. The finite-difference test is an independent numerical check, not another transcription of the same backward formula.

This matters when interpreting “long-distance decay.” The basis may exhibit an aggregate decay tendency under assumptions about learned coefficients, but an individual Q/K pair is not forced to decrease monotonically with distance.

0x03 Two-dimensional rotation sanity check

Use one two-dimensional pair before thinking about a full head.

Let:

$$q = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$$

and:

$$k = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$$

Without position:

$$q^\top k = 1$$

Rotate the query by $90^\circ$ and the key by $30^\circ$:

$$R(90^\circ)q = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$$ $$R(30^\circ)k = \begin{bmatrix} \cos 30^\circ \\ \sin 30^\circ \end{bmatrix}$$

The rotated dot product is:

$$\begin{bmatrix} 0 & 1 \end{bmatrix} \begin{bmatrix} \cos 30^\circ \\ \sin 30^\circ \end{bmatrix} = \sin 30^\circ = 0.5$$

Using the relative form:

$$q^\top R(30^\circ - 90^\circ)k = q^\top R(-60^\circ)k$$

This example is not a full proof. It is a debugging check. It tells you what to print when an implementation gives correct-shaped tensors but a wrong sign convention, wrong pair ordering, or wrong axis broadcast.

---

Norm, scale, and backward behavior

For an orthogonal rotation,

$$R_m^TR_m=I.$$

Therefore

$$\|R_mq\|_2=\|q\|_2.$$

Vanilla RoPE changes direction but not Q/K norm. It does not by itself replace the 1/sqrt(d_head) attention scaling. It also does not introduce trainable parameters when the frequency basis is fixed.

Backward propagation applies the transpose rotation:

$$\frac{\partial \mathcal L}{\partial q} =R_m^T\frac{\partial \mathcal L}{\partial q'}.$$

Since R_m is orthogonal, gradient norm through the pure rotation is preserved:

$$\left\|\frac{\partial \mathcal L}{\partial q}\right\|_2 = \left\|\frac{\partial \mathcal L}{\partial q'}\right\|_2.$$

This local result does not mean a whole attention layer preserves gradients; softmax, masking, projections, and residual paths still shape the Jacobian. It does explain why the vanilla rotary operator is numerically mild compared with an unconstrained learned positional projection.

0x04 Complex notation as implementation shorthand

Many RoPE derivations use complex numbers. That does not mean the Transformer becomes a complex-valued model. It means a real 2D pair:

$$(a,b)$$

can be represented as:

$$a + ib$$

Multiplication by:

$$e^{i\phi} = \cos \phi + i\sin \phi$$

rotates the pair by angle $\phi$:

$$(a + ib)e^{i\phi} = (a\cos\phi - b\sin\phi) + i(a\sin\phi + b\cos\phi)$$

The real coordinates are exactly:

$$\begin{bmatrix} a\cos\phi - b\sin\phi \\ a\sin\phi + b\cos\phi \end{bmatrix}$$

The handwritten RoPE article uses this complex-number view as the bridge from formulas to implementation.

The final tensors returned to the attention kernel are still real-valued. Complex notation is only a compact way to rotate pairs of real dimensions.

---

0x05 From 2D pairs to per-head rotary dimensions

An attention head is not a single 2D vector. It might have:

d_head = 64

RoPE groups the rotary part of that head into pairs:

(0,1), (2,3), (4,5), ..., (62,63)

Each pair receives a different frequency. If the whole head is rotated:

rotary_dim = d_head

If only part of the head is rotated:

rotary_dim < d_head

Partial-rotary implementations rotate the first rotary_dim dimensions and preserve the tail:

q[..., :rotary_dim]      -> rotated
q[..., rotary_dim:]      -> unchanged

The tail is not an implementation accident. It is part of the checkpoint interface. If a checkpoint was trained with partial rotary and inference rotates the whole head, the shapes still line up but the learned Q/K geometry changes.

The rotation matrix for position $m$ is block diagonal:

$$R_m = \operatorname{diag} \left( R(m\theta_0), R(m\theta_1), \ldots, R(m\theta_{\text{rotary\_dim}/2-1}) \right)$$

Then:

$$q_m' = R_m q_m$$ $$k_n' = R_n k_n$$

The same 2D rotation sanity check is repeated across all dimension pairs inside each head.

The block structure gives a useful determinant and inverse check:

$$\det R_m = 1, \qquad R_m^{-1}=R_m^T=R_{-m}.$$

An implementation test can rotate by m and then by -m; the result should recover the original rotary channels within dtype tolerance. This catches sign and pair-order mistakes without requiring a trained model.

q_roundtrip = apply_rotation(
    apply_rotation(q, position_ids, inv_freq),
    -position_ids,
    inv_freq,
)
torch.testing.assert_close(q_roundtrip, q, rtol=2e-3, atol=2e-3)

The tolerance must match the compute path. A float32 reference can use much tighter bounds than repeated bf16 rotations.

---

0x06 Frequency ladder, rope_theta, and wavelengths

For dimension pair $i$, RoPE uses a frequency like:

$$\theta_i = \text{base}^{-2i/d}$$

where $d$ is usually d_head or rotary_dim, depending on the implementation.

The angle at position $m$ is:

$$\phi(m,i) = m\theta_i$$

The approximate wavelength is:

$$\lambda_i \approx \frac{2\pi}{\theta_i}$$

In code:

dim = 64
base = 10000.0
inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float() / dim))
wavelength = 2 * math.pi / inv_freq

The structure should be monotonic:

early pairs:
    larger inv_freq, shorter wavelength, faster rotation

later pairs:
    smaller inv_freq, longer wavelength, slower rotation

This is the local meaning of rope_theta, often called base. It is not a cosmetic config field. It controls the frequency allocation each attention head sees.

M3.4 stops at that local fact. Changing the base, scaling position_ids, interpolating positions, or using mixed frequency scaling is M3.5 territory. If a model was trained with one rope_theta and served with another, this article can identify the geometry mismatch. M3.5 decides whether a changed geometry extends context reliably.

Frequency, wavelength, and phase-resolution ledger

For pair i,

$$\theta_i=b^{-2i/r}, \qquad \lambda_i=\frac{2\pi}{\theta_i}=2\pi b^{2i/r},$$

where r=rotary_dim. The phase difference for token displacement delta is

$$\Delta\phi_i=\delta\theta_i.$$

This gives three useful regimes:

delta << lambda_i:
    small phase difference; pair changes smoothly over that displacement

delta near lambda_i:
    pair traverses a substantial fraction of a cycle

delta >> lambda_i:
    many wraps; phase alone aliases multiple distances

No pair is a globally unique distance code. Relative displacement is represented jointly across the frequency ladder and through learned Q/K coefficients. This is why changing only the lowest-frequency pairs can alter long-range behavior without identically changing local behavior.

def rope_frequency_ledger(rotary_dim, base, max_distance):
    pair = torch.arange(0, rotary_dim, 2, dtype=torch.float64)
    inv_freq = base ** (-pair / rotary_dim)
    wavelength = 2 * math.pi / inv_freq
    cycles_at_max = max_distance / wavelength
    return torch.stack((pair / 2, inv_freq, wavelength, cycles_at_max), dim=1)

Inspecting this table is more informative than logging only rope_theta. Checkpoint compatibility is determined by the resulting frequency vector, rotary pairing, and position map together.

Base sensitivity is pair dependent

To avoid overloading theta, write the configured base as beta and the radian-per-token frequency as omega_i:

$$\omega_i=\beta^{-2i/R}.$$

Adjacent pairs have a constant geometric ratio:

$$\frac{\omega_{i+1}}{\omega_i}=\beta^{-2/R},$$

while wavelengths have the reciprocal ratio. The slowest pair has index R/2-1, hence

$$\lambda_{\max} =2\pi\beta^{(R-2)/R}.$$

It approaches 2*pi*beta as R grows, but is not exactly that value for finite rotary width.

The logarithmic sensitivity to the base is

$$\frac{\partial\log\omega_i}{\partial\log\beta} =-\frac{2i}{R}.$$

The fastest pair i=0 is unchanged when the base changes. Slower pairs move progressively more. Therefore “increase rope_theta” is not one uniform stretch of every rotary plane. It changes the spectral shape while preserving the first frequency.

For R=128, beta=10,000, and a 4,096-position span:

The cycle count is descriptive, not a quality threshold. A pair can be useful after many wraps if learned coefficients exploit its local phase, and a slow pair can still be out of distribution if training never exposed the relevant relative offsets.

Exact recurrence, dense phases, and machine aliasing are different

One pair repeats at an integer displacement T only if

$$T\omega_i=2\pi k_i, \qquad k_i\in\mathbb Z.$$

The full vector repeats only if the same nonzero integer T satisfies every pair. Standard RoPE’s first frequency is 1 radian per token, and no nonzero integer T equals 2*pi*k in exact real arithmetic because 2*pi is irrational. Thus “the first pair wraps every 6.28 tokens” describes a continuous wavelength, not an exact integer-token collision.

Approximate recurrence is still unavoidable over sufficiently long ranges. Irrational rotations are dense on the circle, and a finite set of phases can return arbitrarily close to an earlier configuration. Machine arithmetic adds a third phenomenon: the position or phase can round before sine/cosine is evaluated.

exact mathematical recurrence:
    all phases equal modulo 2*pi

approximate Diophantine recurrence:
    all phases are close modulo 2*pi

machine aliasing:
    distinct integer positions become equal or nearly equal after casting,
    multiplication, or trigonometric argument reduction

IEEE float32 cannot represent every consecutive integer above 2^24 exactly; bfloat16 loses consecutive-integer resolution above 2^8. Production code usually keeps position_ids integral and performs phase construction in float32, but the exact cast and autocast boundary must be audited. This is a numerical representation limit, not evidence that a model can usefully attend to contexts near those lengths.

What the attenuation papers do and do not establish

The RoFormer paper derives an upper-bound tendency under a particular analysis and reports empirical improvements for its evaluated models. That supports RoPE as a structured relative basis; it does not imply that every learned query-key score decreases with distance.

The later Round and Round We Go! analysis makes the limitation constructive: for suitable content vectors, a RoPE head can maximize attention at a chosen relative displacement. For independent isotropic zero-mean Gaussian Q and K, the expected unnormalized activation is zero for every displacement. Its Gemma 7B and Llama 3.1 observations further suggest that low and high frequencies can take different learned roles, but those model-specific measurements are source-reported rather than a law of all checkpoints.

The Base of RoPE Bounds Context Length paper derives a relationship between base and context under its stated analytical assumptions and supports it with experiments. That is relevant evidence for why base matters. It does not turn a larger base into a sufficient long-context recipe. Training distribution, scaling method, attention architecture, evaluation, and runtime consistency remain part of M3.5.

The resulting claim ladder is:

derived here:
    pair frequencies, wavelengths, base sensitivity, score oscillation

primary-paper verified:
    named analytical results and source-reported model observations

not implied:
    universal monotonic decay
    universal frequency specialization
    guaranteed context extension from a larger base

Relative-distance tendency is learned, not guaranteed

RoFormer reports a useful relative-distance attenuation property, and the A17 article visualizes a decreasing upper-bound tendency. The exact score for one query/key pair remains content dependent and oscillatory across frequencies.

The safe claim is:

RoPE supplies a structured multi-frequency function of relative displacement.
The model learns how content coefficients use that basis.

It is unsafe to claim that vanilla RoPE always imposes a monotonic locality penalty like ALiBi.

Periodicity, exact collisions, and approximate aliasing

Each pair is periodic:

$$R_{m+\Delta}^{(i)}=R_m^{(i)} \quad\Longleftrightarrow\quad \Delta\theta_i=2\pi k_i, \qquad k_i\in\mathbb Z.$$

The full rotary vector repeats exactly only if one nonzero integer displacement $\Delta$ satisfies that equation for every pair. This requires all frequency ratios to be commensurate with one common period. The geometric frequency ladder does not generally provide a small exact integer period, but finite precision and approximate phase alignment can still create near-collisions.

For two positions $m,n$, a useful positional-kernel diagnostic is

$$C(\Delta) =\frac{2}{r}\sum_{i=0}^{r/2-1}\cos(\Delta\theta_i), \qquad \Delta=m-n.$$

It is the average dot product of equal unit content across rotary pairs. $C(\Delta)$ oscillates; it is not injective and not strictly decreasing. Similar values at different $\Delta$ illustrate positional aliasing in this simplified kernel, though learned Q/K coefficients need not weight all pairs equally.

def positional_kernel(delta, inv_freq):
    # delta: [D], inv_freq: [r/2]
    phase = delta[:, None] * inv_freq[None, :]
    return phase.cos().mean(dim=-1)

The base-number-system metaphor and its limit

A17 relates RoPE to a base-$\beta$ representation. Since

$$\theta_i=b^{-2i/r} =\beta^{-i}, \qquad \beta=b^{2/r},$$

the coordinate vector can be written

$$[\cos(m/\beta^0),\sin(m/\beta^0),\ldots, \cos(m/\beta^{r/2-1}),\sin(m/\beta^{r/2-1})].$$

Slow pairs resemble high-order digits because they change over longer distances; fast pairs resemble low-order digits because they wrap quickly. This is an intuition about hierarchical scales, not literal digit extraction: RoPE does not apply floor/modulo to produce discrete digits, and learned projections can mix pair information continuously.

Direction is present in the oriented sine term

The pure cosine kernel above is even:

$$C(\Delta)=C(-\Delta).$$

But the full content score derived in 0x02A includes

$$(q_2k_1-q_1k_2)\sin(\Delta\theta),$$

which changes sign under $\Delta\mapsto-\Delta$. Therefore RoPE does not force every learned score to ignore left-versus-right direction. Symmetry appears only for special content coefficients or averaged positional kernels that discard the oriented term.

This corrects an overly broad source-series statement that RoPE cannot learn directional differences.

Frequency-domain interpretation

RoPE turns each query/key pair into coefficients on a fixed trigonometric basis over displacement:

$$s(\Delta) =\sum_i a_i\cos(\Delta\theta_i) +b_i\sin(\Delta\theta_i),$$

where $a_i,b_i$ depend on learned content projections. Fast pairs can express rapidly varying local patterns; slow pairs can vary smoothly over larger distances.

This does not prove a clean semantic/local specialization of every pair. It gives the function basis available to learning. Causal data, initialization, optimization, and layer role determine how frequencies are used.

Long-distance attenuation analyses and HoPE boundary

A17 includes source-reported analyses of U-shaped and activating components at long distance. These results refine the simple attenuation story: learned combinations can oscillate back upward or behave out of distribution.

HoPE is one proposed alternative that changes which positional frequency components remain position-dependent. It is not vanilla RoPE and cannot be enabled by changing one inference scalar. Its training and extrapolation claims require the paper’s architecture and evaluation setup; M3.5 compares context-extension interventions without treating this source-reported proposal as a default.

---

0x07 LLaMA-style precompute_freqs_cis and complex multiplication

The handwritten RoPE source walks through a LLaMA-style implementation that precomputes complex rotations:

def precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):
    freqs = 1.0 / (
        theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim)
    )
    t = torch.arange(end, device=freqs.device, dtype=torch.float32)
    freqs = torch.outer(t, freqs)
    freqs_cis = torch.polar(torch.ones_like(freqs), freqs)
    return freqs_cis

The shape path is the point:

Object	Shape
dim / rotary_dim	scalar, even
torch.arange(0, dim, 2)	[dim/2]
freqs	[dim/2]
t	[end]
torch.outer(t, freqs)	[end, dim/2]
freqs_cis	[end, dim/2] complex

Each complex entry represents:

$$e^{im\theta_i}$$

where $m$ is the position and $i$ is the pair index.

Then Q and K are converted into complex pairs:

xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))

For a LLaMA-style layout:

xq:
    [B, n, h, d_head]

xq_:
    [B, n, h, d_head/2] complex

The rotation table is reshaped so it broadcasts over batch and head:

freqs_cis:
    [n, d_head/2]

broadcast form:
    [1, n, 1, d_head/2]

Then:

xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(3)
xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(3)

returns real tensors:

xq_out:
    [B, n, h, d_head]

xk_out:
    [B, n, h, d_head]

The central implementation figure from A17 ties these steps together.

There are several implementation contracts hidden in this short code:

view_as_complex:
    the last dimension must be paired correctly

float cast:
    the complex path often casts Q/K to float, then returns to the original dtype

freqs_cis slice:
    freqs_cis[start_pos:start_pos+seqlen] must match real token positions

device:
    the rotation table must move with the hidden states

These are not decorative details. If freqs_cis is sliced from position 0 during decode, the new token is rotated as if it were the first token.

---

0x08 Real-valued rotate_half and pairing conventions

Some implementations avoid complex tensors and use cos, sin, and rotate_half:

def rotate_half(x):
    x1 = x[..., : x.shape[-1] // 2]
    x2 = x[..., x.shape[-1] // 2 :]
    return torch.cat((-x2, x1), dim=-1)

def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
    cos = cos.unsqueeze(unsqueeze_dim)
    sin = sin.unsqueeze(unsqueeze_dim)

    q_embed = (q * cos) + (rotate_half(q) * sin)
    k_embed = (k * cos) + (rotate_half(k) * sin)

    return q_embed, k_embed

For a single pair:

[x1, x2] * cos + [-x2, x1] * sin
    =
[x1*cos - x2*sin, x2*cos + x1*sin]

That is a 2D rotation.

The dangerous word is “pair.” Different implementations pair dimensions differently.

GPT-J / interleaved style:
    (0,1), (2,3), (4,5), ...

GPT-NeoX / half-split style:
    (0,d/2), (1,d/2+1), ...

FlashAttention exposes this as an interleaved option: True means even/odd GPT-J-style pairs; False means first-half/second-half GPT-NeoX-style pairs. It also exposes rotary_dim, seqlen_offsets, and packed QKV shapes because those are part of the rotary contract.

The pairing figure from A17 is worth keeping because it shows why this is not a stylistic formatting choice.

Both conventions can be mathematically valid before training. They are not interchangeable after training unless a checkpoint conversion also permutes the Q/K dimensions consistently. A shape test will not catch the bug.

Deriving the layout permutation

For rotary width R=8, write four logical planes as

interleaved:
    x_i = [a0,b0,a1,b1,a2,b2,a3,b3]

half-split:
    x_h = [a0,a1,a2,a3,b0,b1,b2,b3]

There is a permutation matrix P such that

$$x_h=Px_i.$$

P is orthogonal because it only reorders coordinates:

$$P^{-1}=P^T.$$

Let O_i(m) be the block-diagonal rotary operator in adjacent interleaved storage. The same logical rotations in half-split storage are

$$O_h(m)=P O_i(m)P^T.$$

Therefore

$$O_h(m)x_h =P O_i(m)P^TPx_i =P O_i(m)x_i.$$

The outputs are the same logical vector in different physical orders. The dot product is preserved when Q and K are both converted because

$$(Pq)^T(Pk)=q^Tk.$$

Checkpoint conversion must permute projection rows

If an interleaved checkpoint projects

$$q_i=W_Q^i x,$$

then the half-split checkpoint must store

$$W_Q^h=P W_Q^i$$

and likewise for K. A projection bias, if present, must be permuted by P too. For a fused QKV matrix, only the rotary channel rows inside the Q and K blocks are permuted; V is not rotary and must not be changed for this conversion.

Changing only interleaved=True/False leaves W_Qx and W_Kx in their old channel order but applies the new logical plane grouping. That is not the conjugation above. It creates valid tensors and different logits.

For partial rotary, extend the permutation with an identity tail:

$$\widetilde P =P_R\oplus I_{D-R}.$$

Only the first R projection rows move. The unrotated tail remains byte-for-byte in place. A conversion that permutes the full head accidentally changes learned non-rotary channels.

Tensor-axis layout is a separate permutation

The feature pairing above acts on the last D axis. Changing

[B,H,N,D] <-> [B,N,H,D]

is a transpose of tensor axes, not a pairing conversion. It is exactly equivalent only when the position table inserts its singleton head axis in the corresponding location:

[B,H,N,D]: cos/sin [B,1,N,R]
[B,N,H,D]: cos/sin [B,N,1,R]

A broadcast can succeed against the wrong axis when dimensions happen to be equal. The conformance test should use distinct H and N, not a square toy shape, so an accidental swap cannot hide.

GQA repetition commutes with a shared feature rotation

Let Repeat_g copy each KV head g=Hq/Hkv times along the head axis, and let O apply the same rotary operator to the feature axis of every head. Then

$$\operatorname{Repeat}_g(OK) =O\operatorname{Repeat}_g(K).$$

So rotate-before-repeat and repeat-before-rotate are mathematically equivalent under exact arithmetic when every repeated head uses the same positions, frequencies, feature layout, and dtype. Rotate-before-repeat is normally cheaper because it processes Hkv heads rather than Hq heads.

This qualified equivalence is more precise than saying the order is always a checkpoint incompatibility. The order becomes observably different if a bug broadcasts Q-head tables onto K, applies head-dependent rotary state, changes rounding/fusion, or slices the packed QKV head partition incorrectly.

Packed QKV partition contract

Two common packed layouts are

[B,N,3,H,D]

[B,N,Hq + 2*Hkv,D]

The first assumes equal Q/K/V head counts. The second can represent GQA/MQA but requires exact split points:

Q: [0:Hq]
K: [Hq:Hq+Hkv]
V: [Hq+Hkv:Hq+2*Hkv]

RoPE rotates the Q and K partitions, each over the first R feature channels. It does not rotate the V partition and does not infer split points from total width alone. The packed-layout manifest must travel with the kernel call.

The layout oracle constructs P, proves O_h=P O_i P^T, converts toy Q/K projection outputs, and checks score equality. It also proves that a flag-only change differs, that partial tails are untouched, and that shared GQA repetition commutes with rotation.

Layout also changes broadcast:

q/k layout [B,h,n,d]:
    cos/sin [B,n,d] need an inserted head axis

q/k layout [B,n,h,d]:
    cos/sin [B,n,d] need an inserted head axis at another position

That is why a parameter like unsqueeze_dim is not just an implementation convenience. It decides which axis receives the position table.

---

0x08A One complete real-valued tensor trace

The source implementation diagrams are easiest to audit when reduced to an explicit batch/head trace.

Assume:

B = 2
N = 4
Hq = 8
Hkv = 2
Dh = 64
rotary_dim = 32

Then:

q:              [2,8,4,64]
k:              [2,2,4,64]
position_ids:   [2,4]
inv_freq:       [16]
angles:         [2,4,16]
cos_half/sin:   [2,4,16]
cos/sin expanded according to pairing convention
q_rotary:       [2,8,4,32]
k_rotary:       [2,2,4,32]
q_tail:         [2,8,4,32]
k_tail:         [2,2,4,32]
q_out:          [2,8,4,64]
k_out:          [2,2,4,64]

A layout-explicit implementation is:

def apply_partial_rope(q, k, position_ids, inv_freq, rotary_dim):
    # q: [B,Hq,N,Dh], k: [B,Hkv,N,Dh]
    assert rotary_dim % 2 == 0
    assert rotary_dim <= q.shape[-1] == k.shape[-1]

    angles = position_ids.to(torch.float32)[..., None] * inv_freq[None, None, :]
    # [B,N,R/2]
    cos_half = angles.cos()
    sin_half = angles.sin()

    # This duplication matches a half-split rotate_half convention.
    cos = torch.cat((cos_half, cos_half), dim=-1)[:, None, :, :]
    sin = torch.cat((sin_half, sin_half), dim=-1)[:, None, :, :]
    # [B,1,N,R]

    q_rot, q_tail = q[..., :rotary_dim], q[..., rotary_dim:]
    k_rot, k_tail = k[..., :rotary_dim], k[..., rotary_dim:]

    q_rot = q_rot * cos + rotate_half_split(q_rot) * sin
    k_rot = k_rot * cos + rotate_half_split(k_rot) * sin

    return torch.cat((q_rot, q_tail), dim=-1), torch.cat((k_rot, k_tail), dim=-1)

For an interleaved convention, do not reuse this duplication and rotate_half_split; construct paired even/odd channels consistently. The formula name does not identify the layout.

Reference conformance tests

def explicit_pair_rotation(x, angles):
    # x: [...,R], angles: broadcastable [...,R/2]
    pair = x.reshape(*x.shape[:-1], -1, 2).float()
    a, b = pair.unbind(-1)
    c, s = angles.cos(), angles.sin()
    out = torch.stack((a * c - b * s, a * s + b * c), dim=-1)
    return out.flatten(-2).to(x.dtype)

q_complex = rope_complex_reference(q, position_ids, inv_freq)
q_real = explicit_pair_rotation(q, angles)
torch.testing.assert_close(q_real, q_complex, rtol=2e-3, atol=2e-3)

# Position zero must be identity.
q0 = explicit_pair_rotation(q[..., :1, :], torch.zeros_like(angles[..., :1, :]))
torch.testing.assert_close(q0, q[..., :1, :])

# A common shift must preserve pairwise scores when content is held fixed.
score_a = dot(rotate(q, m), rotate(k, n))
score_b = dot(rotate(q, m + shift), rotate(k, n + shift))
torch.testing.assert_close(score_a, score_b, rtol=2e-3, atol=2e-3)

The common-shift test is stronger than checking output shape. It exercises the relative-offset identity directly.

Precision and autocast boundary

Large position IDs multiplied by small frequencies can lose phase accuracy if angles are formed in low precision. A robust path commonly computes position_ids * inv_freq and trigonometric functions in float32, then casts the resulting cos/sin or rotated Q/K to the model dtype.

with torch.autocast(device_type=q.device.type, enabled=False):
    angles = position_ids.float()[..., None] * inv_freq.float()[None, None, :]
    cos = angles.cos()
    sin = angles.sin()

q_rot = apply_rotation(q.float(), cos, sin).to(q.dtype)
k_rot = apply_rotation(k.float(), cos, sin).to(k.dtype)

Exact casting policy varies by kernel. The invariant is that a long-context path should not silently quantize position IDs or angle construction more aggressively than the checkpoint/runtime was validated for.

Large-index numerical limits

Float32 represents every integer exactly only through $2^{24}$. Beyond that, adjacent integer position IDs can map to the same float before multiplication. BF16 and FP16 lose integer resolution much earlier. Most present context windows are smaller, but offsets from prefix reuse, persistent sessions, or synthetic tests can expose the boundary.

Even below integer-collision limits, trigonometric argument reduction for large angles can differ across devices and kernels. Compare tables by their resulting complex phase rather than raw angle because

$$e^{i\phi}=e^{i(\phi+2\pi k)}.$$

Useful numerical tests include:

adjacent-position phase difference at maximum supported position
on-the-fly versus precomputed cos/sin
CPU float64 reference versus device float32 kernel
prefill table slice versus decode offset path
common-shift score invariance at large offsets

Precompute, cache, or generate on demand

For maximum position $L$ and rotary width $r$, a full cosine+sine table stores roughly

$$2L(r/2)=Lr$$

real elements if half-width tables are retained, or $2Lr$ if both are expanded to the full rotary layout. Complex tables store $Lr/2$ complex elements, which is the same real-scalar count as half-width cosine+sine.

Design choices:

full precompute:
    simple indexing, fixed maximum length, resident table memory

lazy extension:
    grow tables when a request exceeds current capacity,
    must synchronize concurrent users

on-the-fly:
    lower resident state, repeated outer-product/trig work

kernel-generated:
    fuse angle construction/rotation,
    backend becomes numerical contract

When context scaling is dynamic, a table keyed only by length is insufficient. Its cache key must include base/scaling parameters, rotary width, dtype/device, and any sequence-dependent factor.

0x08B Pinned framework and kernel traces

An implementation name is not evidence unless it identifies a revision and a source path. The evidence workspace freezes seven code snapshots. Three form the main trace because they expose different parts of the operator.

Meta LLaMA: complex table, explicit offset, rotated cache write

At pinned revision 689c7f2, precompute_freqs_cis constructs

frequency exponents from head_dim
    -> outer(position, inv_freq)
        -> torch.polar(...): complex64 [L,D/2]

reshape_for_broadcast asserts that the selected table length and complex pair width match the Q/K complex view. apply_rotary_emb casts Q/K to float, reshapes adjacent scalar pairs with view_as_complex, multiplies by freqs_cis, converts back to real, flattens the pair axis, and returns the original Q/K dtype.

The model call then makes the runtime ordering explicit:

slice freqs_cis[start_pos:start_pos+seqlen]
project Q/K/V
rotate Q and K
write rotated K and raw V to cache
read prefix through current slot
repeat KV heads for GQA
score

This source supports the article’s rotated-K cache claim. It does not establish that every engine uses the same cache format.

Transformers LLaMA: config-derived real tables shared across layers

At pinned revision d09f53a, LlamaRotaryEmbedding derives inv_freq and an attention_scaling value from normalized config. The inverse frequency buffers are registered with persistent=False: they are module state used at runtime but reconstructed from config rather than guaranteed checkpoint tensors.

Its forward path disables surrounding autocast, expands inv_freq and position_ids in float32, computes their matrix product, duplicates the phase for the half-split layout, evaluates cosine/sine, and casts the result to the input dtype. The model creates position_embeddings=(cos,sin) once and passes that pair to each decoder layer. Each attention layer:

projects and reshapes Q/K/V to [B,H,N,D]
applies half-split rotate_half to Q/K
updates the cache with rotated K and unrotated V
dispatches the selected attention interface

The decorator on apply_rotary_pos_emb also allows a registered external kernel. Therefore the Python body is the semantic fallback, while backend selection may change execution. A substituted kernel must preserve the same pairing, broadcast, dtype, and alias behavior.

FlashAttention: partial width, packed offsets, in-place mutation, backward

At pinned revision df61ab6, table width defines

$$R=2\,\text{cos.shape[-1]}.$$

The reference function asserts R <= D, rotates only x[..., :R], and concatenates the unchanged tail. The interface then exposes contracts hidden by a simple framework helper:

interleaved:
    GPT-J even/odd versus GPT-NeoX half-split pairing

seqlen_offsets:
    one integer or one offset per batch row

cu_seqlens + max_seqlen:
    packed variable-length ownership

num_heads_q:
    split point for packed GQA/MQA Q + K + V heads

inplace:
    output aliases and mutates the input storage

The custom autograd function saves cosine/sine plus offset/packing metadata, not the full Q/K input. Backward applies the same rotary kernel to the upstream gradient with conjugate=True, which is the inverse rotation derived earlier:

$$\bar{x}=R(-\phi)\bar{y}.$$

It returns no gradient for cosine, sine, layout flags, or offsets. That is correct for fixed tables. A learned-frequency architecture cannot silently reuse this backward contract because it needs gradients through phase construction.

GPT-NeoX and GPT-J establish the naming convention

The pinned GPT-NeoX helper uses a half-split rotate_half, slices cached cosine/sine tables by an integer offset, and returns rotated Q/K. The pinned mesh-transformer-jax GPT-J path rotates adjacent even/odd pairs and supports a partial pe_rotary_dims prefix with an unchanged pass-through tail.

These snapshots justify the terms “GPT-NeoX/half-split” and “GPT-J/interleaved” used in this article. They are historical implementation lineages, not a claim that every model carrying either family name uses the same current code.

Saved state and alias ledger

In-place rotary is safe only while no consumer needs the unrotated storage and autograd’s version/alias rules are respected. A fused QKV buffer is particularly sensitive: rotating the wrong head partition mutates V or another request’s view before any shape error appears.

The pinned-source audit checks the exact commits, snapshot hashes, required function names, and claim-bearing source ranges. Future library releases may improve or reorganize these paths; they cannot retroactively alter the evidence used by this article.

0x09 RoPE placement in attention, masks, and V boundary

The complete local path is:

q = project_q(x)
k = project_k(x)
v = project_v(x)

q, k = apply_rope(q, k, position_ids)

scores = torch.matmul(q, k.transpose(-2, -1)) / math.sqrt(d_head)
scores = scores.masked_fill(causal_mask == 0, -float("inf"))
attn = torch.softmax(scores, dim=-1)
out = torch.matmul(attn, v)

Only Q and K are rotated.

Q:
    what am I looking for?

K:
    what do I offer to be matched?

V:
    what payload is mixed after attention weights are chosen?

RoPE changes the geometry used to match Q against K. V is the payload averaged by attn @ v, so rotating V is normally the wrong object.

RoPE also does not replace masks. A causal or padding mask is still a visibility rule:

RoPE:
    changes the dot product between visible query/key rows

mask:
    removes illegal key rows before softmax

A future token rotated to a useful-looking angle is still future. It must remain masked.

This matters for multimodal prefixes and interleaved rows:

[IMG_0, IMG_1, ..., IMG_195, <bos>, question, answer]

If text starts at RoPE position 196 in training, resetting it to 0 at inference changes the coordinate system. If the model was trained to reset text after media, continuing through media rows can be wrong instead. M3.4 does not decide the multimodal architecture; it gives the debugging question:

which rows received which RoPE positions during training?

---

0x09A Training and derived-state lifecycle

Vanilla RoPE has no learned position table, but it is still learned through. Q/K projection weights adapt to the fixed rotary basis during training. A checkpoint therefore encodes assumptions about:

rotary_dim
pairing convention
frequency base and exponent denominator
which layers use RoPE
which Q/K heads are rotated
special-token and packed-sequence position IDs
dtype/precision of the operator

Changing those fields after training changes the function even though RoPE has no trainable parameter of its own.

Parameter-free is not state-free

The phrase “RoPE has no parameters” is useful only if parameter means a tensor updated by the optimizer. A reproducible training or inference run must distinguish at least six kinds of state:

The pinned Transformers LLaMA implementation illustrates the distinction. Its inv_freq buffers are registered with persistent=False: they are derived from configuration and reconstructed rather than guaranteed checkpoint tensors. That does not make rope_theta, rotary_dim, or the scaling policy optional metadata. Those fields are the recipe from which the buffer obtains meaning.

Derived tables need an invalidation key

Suppose a runtime stores a table

$$T=(\cos \Phi,\sin \Phi), \qquad \Phi_{m,i}=p_m\theta_i.$$

Its shape does not identify its contents. A safe table key includes every field that can alter a phase value or its numerical representation:

table_key = hash(
    rotary_dim,
    base,
    scaling_type,
    scaling_parameters,
    position_origin_or_explicit_ids,
    maximum_required_position,
    trig_compute_dtype,
    returned_table_dtype,
    device,
    training_or_inference_table_policy,
)

Changing only device or returned dtype normally means regenerate or copy and revalidate derived tables. Changing base, scaling, rotary width, pairing, or position policy means the semantic operator changed; existing rotated cache rows and checkpoint assumptions cannot be admitted under the old identity. Dynamic scaling can make the table depend on current sequence length, so max_position may be part of the semantics rather than only an allocation bound.

One runtime decision procedure is:

def admit_rotary_state(saved, requested):
    if saved.checkpoint_semantics != requested.checkpoint_semantics:
        return "migrate checkpoint or reject"
    if saved.position_policy != requested.position_policy:
        return "reject packed/cache state"
    if saved.learned_frequency_schema != requested.learned_frequency_schema:
        return "reject checkpoint"
    if saved.table_device_dtype_length != requested.table_device_dtype_length:
        return "regenerate derived tables"
    if requested.learned_frequencies and not saved.optimizer_state_available:
        return "inference only; exact training resume unavailable"
    return "reuse"

A reproducible state-transition test should exercise those cases while keeping reuse, regenerate, migrate, and reject separate: rebuilding a cosine table is not equivalent to converting trained Q/K channels.

Numerical state and recomputation

The phase product and trigonometric functions are commonly evaluated in FP32 even when Q/K use BF16 or FP16:

with autocast_disabled_for_rotary():
    phase = position_ids.float()[..., None] * inv_freq.float()[None, ...]
    cos = phase.cos()
    sin = phase.sin()

q_rot = apply_rope(q.float(), cos, sin).to(q.dtype)
k_rot = apply_rope(k.float(), cos, sin).to(k.dtype)

That is a numerical contract, not a claim that every backend executes each operation as an unfused FP32 kernel. A fused implementation may use a different schedule while matching an agreed tolerance. Release tests therefore need both operator-level reference outputs and end-to-end logits, especially at large positions where phase rounding becomes visible.

Gradient checkpointing can either save cos/sin or reconstruct them during the recomputed forward. Reconstruction is valid only when the same position IDs, frequency configuration, compute precision, dynamic-scaling inputs, and table policy are replayed. A recomputation that silently sees a longer sequence or a different sampled packed-document origin is a different forward pass.

Paper and runtime variants

Multi-axis and hybrid rotary designs

Images, video, and multimodal streams may assign different coordinate axes to different rotary channel groups. For axis $a$ and its pair set $I_a$:

$$R(\mathbf p) =\operatorname{blockdiag} \left( \{R(p_a\theta_i):i\in I_a\}_{a=1}^{A} \right).$$

Examples of axes include temporal index, image row, and image column. Two visual tokens can share time but differ in height/width, while text tokens may advance all configured axes together or use one dedicated coordinate convention.

Tensor trace:

position_ids: [A,B,N]
inv_freq:     [R/2]
axis sections sum to R/2 pairs
angles_a:     [B,N,|I_a|]
concatenated: [B,N,R/2]
cos/sin:      broadcast to Q/K head axes

Flattening a 2D image to one 1D token index and applying text RoPE is a different geometry from axial/multimodal RoPE, even if both produce [B,N,d] tensors. Cache reuse must preserve the full coordinate tuple or the exact derived phase table.

Hybrid stacks may use RoPE in only some layers, only a fraction of each head, or combine rotary and non-positional attention heads. This changes both representation and cache compatibility:

per-layer rotary config
    -> which K rows are position-transformed
    -> which cache format each layer stores

One global rope_theta field cannot fully describe such a model.

Learned frequencies

If frequencies become trainable,

$$\theta_i=\exp(\omega_i)$$

is a convenient positive parameterization. The phase derivative is

$$\frac{\partial}{\partial\omega_i} \cos(m e^{\omega_i}) =-m e^{\omega_i}\sin(m e^{\omega_i}),$$

so gradient magnitude can grow with position and oscillate rapidly. Learned frequencies add optimizer/checkpoint state and can overfit the training window. They are a trained architecture variant, not a runtime context-extension knob.

Two real implementations can produce identical output only if one also permutes channels between their pairing conventions. For adjacent pair vector

[a0,b0,a1,b1,...]

and half-split vector

[a0,a1,...,b0,b1,...]

the rotation mathematics is the same after a permutation Pi:

$$R_{\text{half}} = \Pi R_{\text{interleaved}}\Pi^T.$$

The checkpoint’s Q/K output channel order decides which representation is correct. A loader cannot flip interleaved without also transforming the relevant projection weights.

Packed training needs logical, not merely physical, positions

Suppose two documents are packed into one physical row:

[doc_A tokens][doc_B tokens]

Possible position contracts include:

continuous:
    doc_B starts after doc_A physical length

reset:
    doc_B starts from zero and block-diagonal attention prevents cross-document reads

randomized/offset:
    each logical document receives a sampled origin

All three can use the same [B,N] tensor shape. Training and inference must agree, and prefix-cache identity must include any position origin that affects rotated K.

For the reset policy, position reset and attention isolation are two separate requirements:

document segment IDs
    -> construct block-diagonal visibility

document-local offsets
    -> construct position_ids

Resetting positions without blocking cross-document attention allows one document to read another through a coordinate system that now contains duplicate positions. Blocking attention without resetting positions implements the continuous policy, not the reset policy. If origins are randomized during training, the sampled origins are forward inputs and must be replayed during activation recomputation.

Gradient and optimizer state

Fixed RoPE adds no optimizer state. Learned Q/K projections still receive gradients through the rotation. If inv_freq is made learnable in a research variant, it becomes a different architecture with global, position-dependent gradients:

$$\frac{\partial \phi(m,i)}{\partial \theta_i}=m.$$

Long positions scale those gradients. Such a parameter needs its own optimizer, precision, regularization, and checkpoint treatment; it should not be described as ordinary fixed RoPE.

Checkpoint conversion and release admission

A loader should not infer rotary semantics from tensor shapes. A release manifest should make the following transition explicit:

For an interleaved/half-split conversion, the same permutation applied to Q/K projection rows must also be applied to their gradient accumulators, momentum, variance estimates, and any FP32 master weights. Converting only the model checkpoint can preserve inference while corrupting a resumed optimizer step.

A robust migration finishes with three independent checks:

1. operator oracle:
   old-layout RoPE output == converted-layout output after inverse view mapping

2. model oracle:
   golden prompt logits agree within the declared dtype tolerance

3. cache oracle:
   prefill plus decode agrees with one-shot scoring under the new cache format

The release should record the semantic manifest and invalidate derived tables rather than serializing them as if they were authoritative model weights. If a runtime does serialize tables for startup speed, it should store and verify the manifest digest that generated them.

0x09B Alternative methods change different mathematical objects

“RoPE variant” is too broad to be an implementation instruction. Some names describe an equivalent representation of the same map; others change the map, the trained parameterization, the coordinate domain, or the attention score itself. The first debugging question is therefore:

which mathematical object changed?

Equivalent representations versus different operators

Complex multiplication, an interleaved real kernel, and a half-split real kernel can represent the same block-diagonal orthogonal map after the exact channel permutation derived earlier. They need no retraining if checkpoint rows, biases, optimizer tensors, and cache format are converted consistently.

Partial RoPE instead changes the operator to

$$O_R(p)=\rho_R(p)\oplus I_{d_{head}-R}.$$

It remains norm preserving, but it gives only the first $R$ checkpoint channels a position-dependent basis. Changing $R$ without training or a validated model migration is not a representation conversion.

xPos adds reciprocal scale to rotation

A Length-Extrapolatable Transformer generalizes each complex/real pair with a positive per-pair scale $\zeta_i$. Using one compact convention,

$$f_q(q_m,m)=\zeta_i^mR(m\theta_i)q_m, \qquad f_k(k_n,n)=\zeta_i^{-n}R(n\theta_i)k_n.$$

Their pairwise product is

$$f_q(q_m,m)^Tf_k(k_n,n) =\zeta_i^{m-n}q_m^TR((n-m)\theta_i)k_n,$$

so it still depends on relative displacement under this reciprocal construction. Unlike vanilla RoPE, each Q or K map is not individually norm preserving when $\zeta_i\ne1$. A rotated-and-scaled K cache embeds both the phase and reciprocal scale contract. The paper also evaluates blockwise causal attention; that mask change is a separate intervention and its source-reported extrapolation results must not be attributed to the scale term alone. M3.5 owns the full extension recipe and evaluation contract.

Identity subspaces: p-RoPE and High-frequency RoPE

Round and Round We Go reports that, in the audited Gemma 7B and Llama 3.1 8B models, high-frequency components support some positional heads while low-frequency bands carry substantial Q/K norm and appear to serve semantic matching. Those are model-specific mechanistic observations, not a universal allocation law.

Its p-RoPE intervention removes position dependence from a selected lowest- frequency fraction. At the operator level,

$$O_{p\text{-RoPE}}(p) =\operatorname{blockdiag} \left( \{R(p\theta_i):i\in I_{pos}\}, I_{semantic} \right).$$

The identity blocks provide channels whose dot products do not rotate with distance; the retained blocks still encode relative position. The paper’s performance findings are source-reported training experiments, not permission to zero frequencies in an arbitrary deployed checkpoint.

HoPE also retains selected high-frequency rotary signals while replacing particular components with position-independent ones, motivated by its analysis of local decay and global U-shaped attention. It is a distinct proposed architecture and evidence contract despite a related operator-level pattern. p-RoPE and HoPE should therefore be compared by the exact selected subspace, training recipe, and evaluation setup, not merged into one runtime option.

Multi-axis RoPE changes the coordinate domain

Text RoPE uses one scalar position $p$. Qwen2-VL’s M-RoPE, for example, splits rotary channels across temporal, height, and width coordinates. Its operator is of the form already introduced above,

$$O(\mathbf p) =\bigoplus_a\bigoplus_{i\in I_a}R(p_a\theta_i),$$

with $\mathbf p=(p_t,p_h,p_w)$. For text, the cited design gives the components identical position IDs; for image/video tokens, the axes differ. A scalar position cannot reconstruct that state. X1.1 owns non-text unit and geometry metadata, X1.2 owns fusion placement, and X1.3 owns mixed-origin runtime/cache policy; M3.4 owns only the fact that the coordinate tuple and channel partition enter the rotary operator and cache key.

Non-rotary alternatives change the score path

NoPE, additive relative bias, and ALiBi are not degenerate tensor layouts of RoPE:

$$s_{m,n}^{NoPE}=q_m^Tk_n,$$ $$s_{m,n}^{bias}=q_m^Tk_n+b(m-n),$$ $$s_{m,n}^{ALiBi}=q_m^Tk_n-a_h d_{m,n},$$

with causal distance

$$d_{m,n}=m-n, \qquad m\ge n.$$

ALiBi applies a head-specific distance penalty to logits rather than rotating Q/K. NoPE omits an explicit positional term, although a causal Transformer can still infer some positional structure through visibility and learned computation. M3.3 owns those score-space methods and their comparative evidence. Replacing RoPE with any of them changes trained attention geometry and usually requires training or adaptation.

Contract matrix

The decision rule is strict: an equivalent layout may be converted; a changed operator requires an architecture migration; a changed training objective or mask requires its paper contract; and a changed coordinate domain requires the upstream modality policy. Similar output shapes do not collapse those cases.

0x0A Prefill/decode/cache-position invariant

RoPE is applied before K is used for scoring. In autoregressive inference, that means cached keys carry a position-dependent rotation.

During prefill:

position_ids:
    [0,1,2,...,prompt_len-1]

q/k:
    [B,h,prompt_len,d_head]

cache write:
    write rotated K for those positions

During decode:

position_ids:
    [prompt_len] or cache_position

q/k:
    [B,h,1,d_head]

score:
    [B,h,1,past_len+1]

The invariant is:

the current query and every cached key it scores against must live in the same RoPE coordinate system

The compact table is the easiest way to debug this.

prompt:
    <bos> I ate an apple

prefill:
    token:          <bos>   I   ate   an   apple
    position_id:       0    1     2    3       4
    cache slot:        0    1     2    3       4

stored keys:
    K_0 rotated with position 0
    K_1 rotated with position 1
    K_2 rotated with position 2
    K_3 rotated with position 3
    K_4 rotated with position 4

next decode token:
    token:          <eos>
    position_id:       5
    cache slot:        5

The wrong decode path is:

K_new rotated with position 0

The right decode path is:

K_new rotated with position 5

Serving code often tracks several objects:

input_ids
attention_mask
position_ids
cache_position
cache write slice
cache read slice

M3.10 owns the full single-request loop and M5 owns cache lifecycle at serving scale. M3.4 owns the RoPE invariant: those objects must select the same coordinate for the key being written and for the query that will score against it.

Left padding and right padding complicate this. In a batch:

row 1 visible length: 8
row 2 visible length: 3

the next generated token might need different position_ids per row. A shared [0..n-1] table can be shape-compatible and semantically wrong.

Semantic position is not physical cache address

For one new token, keep these objects separate:

logical token identity:
    which token in which request/document is being processed?

semantic RoPE position p:
    which coordinate selects R(p)?

physical cache slot s:
    where will the K/V row be stored?

cache-visible length/read set:
    which previously committed slots may attention read?

manifest digest:
    which pairing/base/scaling/precision contract produced the row?

In a contiguous toy cache, p == s is common. It is not a mathematical requirement. Left padding, packed documents, sliding windows, paged allocation, prefix reuse, and branch-local staging can make the integers differ. The rotation uses $p$; the allocator writes slot $s$.

For left-padded rows, one common local-position construction is the cumulative count of visible tokens minus one, with padded cells masked or clamped only to keep indexing valid:

position_ids = attention_mask.long().cumsum(-1) - 1
position_ids = position_ids.masked_fill(attention_mask == 0, 0)

The zero placed in a padded cell does not make it a real position-zero token; the attention mask must still exclude that cell. During decode, each row’s next semantic position follows its own visible logical length even if the batch uses one rectangular physical cache tensor.

One cache-write transaction

A rotated-key cache can expose a row only after the coordinate and representation used to create it are known:

def stage_rotated_kv(hidden, token, request, allocator, rope):
    slot = allocator.reserve(request)          # physical address s, not position p
    position = request.next_position(token)    # semantic coordinate p
    q, k, v = project_qkv(hidden)
    cos, sin = rope.tables(position)
    q_rot = rope.rotate(q, cos, sin)
    k_rot = rope.rotate(k, cos, sin)

    staged = {
        "slot": slot,
        "position": position,
        "k": k_rot,
        "v": v,
        "rope_digest": rope.semantic_digest,
        "cache_format": "rotated_k",
        "valid": False,
    }
    return q_rot, staged

def commit(staged, cache):
    cache.write(staged["slot"], staged["k"], staged["v"])
    cache.metadata[staged["slot"]] = (
        staged["position"], staged["rope_digest"], staged["cache_format"]
    )
    cache.publish_valid(staged["slot"])

publish_valid is the semantic commit point. A fused kernel may combine reserve, rotate, write, and publish more tightly, but it must preserve the same result: no reader observes a slot as valid with a rejected token’s key, a stale digest, or a position selected for another row.

Why one-shot, chunked prefill, and decode can agree

Let the semantic position of token $j$ be $p_j$. A one-shot forward stores

$$\widetilde{k}_j=R(p_j)k_j$$

and the score from token $t$ to token $j$ is

$$s_{t,j} =\left(R(p_t)q_t\right)^T\widetilde{k}_j =q_t^T R(p_j-p_t)k_j.$$

Now partition the same sequence into arbitrary prefill chunks. If every chunk uses the same $p_j$, same Q/K tensors, same rotary manifest, and same numerical contract, each stored $\widetilde{k}_j$ is identical to the one-shot row. The physical write order or slot address does not enter the equation. Therefore the later score is identical up to the declared floating-point tolerance.

Resetting the local position to zero at every chunk instead stores $R(j-c_{\text{start}})k_j$ instead of $R(j)k_j$. The shapes and cache lengths still pass, but cross-chunk scores now use the wrong relative displacement. The A conformance test can compare one-shot, chunked, and decode paths and inject exactly this reset failure.

Rollback, truncation, and branch ownership

Tentative generation may create candidate rows that never become part of the accepted prefix. M3.4 does not own the speculative-decoding algorithm, but it does own the rotary cache boundary:

accepted candidate row:
    publish the staged row produced from that candidate's actual token and p

rejected candidate row:
    keep invalid or remove from the cache-visible read set

replacement/correction token:
    compute a new Q/K/V row for the replacement token;
    never relabel the rejected candidate's rotated K as the replacement

Truncating a request must retract both the visible slot and its semantic metadata. Reusing the same physical page later is safe only after its old valid bit and request ownership are cleared. M3.10 owns the complete single-request state transition, M7 owns draft acceptance, and M5 owns allocation; the contract here is that cache publication preserves token, position, and rotary identity.

Rotated versus unrotated K cache designs

The common decoder path rotates K before cache write:

q_rot, k_rot = apply_rope(q, k_new, position_ids)
cache.write(k_rot, v_new)
scores = q_rot @ cache.k.transpose(-2, -1)

This avoids rerotating historical K every decode step. It also binds the cache to the exact RoPE scheme, base, scaling, and position origin used at write time.

An alternative research/runtime design can store unrotated K plus position metadata and rotate during read or inside a kernel:

cache.write(k_unrotated, v_new, key_positions)
k_rot = rotate_for_read(cache.k, cache.key_positions, rope_config)

That cache is more flexible under some transformations but performs more work and needs position metadata to remain exact. The two cache formats are not interchangeable.

Their read equations make the difference explicit. A rotated cache stores

$$C_j^{\text{rot}}=R(p_j)k_j, \qquad s_{t,j}=\left(R(p_t)q_t\right)^T C_j^{\text{rot}}.$$

An unrotated cache stores both

$$C_j^{\text{raw}}=k_j, \qquad p_j,$$

and reconstructs $R(p_j)C_j^{\text{raw}}$ during the read. Omitting $p_j$ or using a different rotary manifest changes the score. Merely tagging both tensors as cache_k is insufficient.

Cache format	Write cost	Read/decode cost	Compatibility identity
rotated K	rotate new rows once	use stored K directly	RoPE scheme embedded in K bytes
unrotated K + positions	store raw projection	rotate historical/read tiles	raw K layout plus exact key positions and RoPE config

M5 owns allocation and reuse policy. M3.4 supplies the semantic key that a cache manager must include before declaring two prefixes compatible.

A minimal RoPE-dependent prefix admission record is:

rope_cache_identity = hash(
    rotary semantic digest,
    cache format: rotated_k | raw_k_plus_positions,
    logical position origin/policy,
    numerical compatibility policy,
)

The complete M5 cache key must also cover model weights, tokenization, visibility, adapter state, and any other input that changes K/V bytes. Matching this local RoPE identity is necessary, not sufficient, for prefix reuse.

Local rotation-work and frequency-state ledger

Let r=rotary_dim, with even r. A complex frequency table stores n*r/2 complex values. A real implementation commonly stores cosine and sine tables broadcastable over n*r rotated scalar positions. Representation and dtype determine the bytes:

complex path: [n,r/2] complex
real path:    cos [n,r] plus sin [n,r]

For n=4096 and r=128, a complex64 freqs_cis table contains 262,144 complex values and occupies 2 MiB. Two bf16 tables cos and sin, each [4096,128], also occupy 2 MiB together. Float32 cosine/sine doubles that. Here n is the stored or active position range being counted. A precomputed table sized for L_max uses L_max in the storage formula even when one decode call selects only a one-position slice. Some kernels store half-width cos/sin: [L_max,r/2] and apply the pairing convention in the rotary kernel; the full-width [L_max,r] example above matches implementations that duplicate frequencies across the paired scalar dimensions. Audit the actual table shape before assigning bytes.

In the real path:

q_rot = q * cos + rotate_half(q) * sin
k_rot = k * cos + rotate_half(k) * sin

each rotated scalar performs approximately two multiplications and one addition; rotate_half is primarily a sign/permutation operation. With batch B, query heads H_q, KV heads H_kv, and sequence length n, the rotated scalar slots are:

$$N_{\text{rot slots}} = Bn(H_q+H_{kv})r.$$

For B=1, n=4096, H_q=32, H_kv=8, and r=128:

rotated scalar slots: 20,971,520
approx multiplications: 41,943,040
approx additions:       20,971,520

These are local arithmetic counts, not a kernel timing model. Fused layouts, complex instructions, table recomputation, and memory access change execution.

The persistent-state invariant is more important than the small table: typical decode paths rotate K before writing it to cache. The cache therefore stores positioned K in the checkpoint’s pairing/base/scaling convention. Reusing that K with a query built from a different rope_theta, scaling rule, or position offset is a same-shape coordinate-system failure. Placement must keep frequency state and cache-producing kernels configuration-consistent across prefill and decode.

---

0x0B Shape ledger

---

0x0C Failure lab

Q/K/V application bugs

Rotating V. The model runs, but the value payload is transformed even though RoPE should normally affect Q/K scoring geometry.

Applying RoPE to x before Q/K projection. That is not the same object. It moves position into the input-state path instead of rotating projected Q/K.

Applying RoPE after score construction. Too late. The Q/K dot product has already been computed.

Applying RoPE to Q but not K. Same shapes, incompatible coordinate systems.

Applying RoPE to K but not Q. Same failure in reverse.

Dimension pairing and rotary_dim bugs

Wrong dimension pairing. GPT-J/interleaved and GPT-NeoX/half-split can both produce valid tensors, but a checkpoint expects the convention used during training or conversion.

Odd rotary_dim. Dimension pairs cannot be formed cleanly.

Applying RoPE to full d_model instead of per-head d_head. Wrong object. RoPE rotates inside each attention head.

Applying rotary to the wrong number of dimensions. Partial rotary implementations rotate only rotary_dim; rotating the tail or failing to preserve it changes checkpoint behavior.

Model conversion changes dimension order. Same shape, wrong pairing.

Complex implementation bugs

view_as_complex on the wrong paired axis. The complex numbers no longer represent the intended dimension pairs.

Forgot dtype handling for the complex path. Some implementations cast Q/K to float for complex multiplication and return to the original dtype afterward.

freqs_cis broadcasts over heads instead of sequence. The rank looks plausible, but positions attach to the wrong axis.

freqs_cis device or dtype mismatch. This can fail at runtime or produce precision/device drift.

Flattening the wrong dimension after view_as_real. The final shape can be forced back while pair order is corrupted.

rotate_half implementation bugs

rotate_half sign reversed. Shape unchanged; rotation direction flips.

cos / sin unsqueezed on the wrong axis. Rotations broadcast across batch, head, or sequence incorrectly.

Using a half-split implementation with an interleaved checkpoint. Correct shape, wrong logits.

Treating interleaved as performance-only. It is a geometry convention.

Frequency and base bugs

Wrong rope_theta. Same shapes, different frequency basis.

Wrong exponent denominator. dim, d_head, and rotary_dim must be consistent in frequency construction.

Changing scaling config between prefill and decode. Cached keys and new queries use different coordinate systems.

Assuming a larger base automatically fixes long context. That is M3.5 territory. M3.4 only explains the local frequency effect.

Position ID and padding bugs

position_ids reset after BOS, chat template, or image prefix. The content token is right; its coordinate is not.

Left-padding batch uses visible-column indices instead of semantic positions. Batched and single-prompt generations diverge.

Right padding plus last-token selection contaminates positions. M3.10 owns last-valid gather; M3.4 flags the coordinate symptom.

Different rows need different position_ids but receive one shared [0..n-1] range. Shape works; geometry differs.

Prefill/decode/cache bugs

freqs_cis sliced from position 0 during decode. The new token is rotated as if it were the first token.

cache_position reset after prefill. K is written and rotated under the wrong position.

Cached K rotated under one scaling rule and new Q under another. Dot products mix coordinate systems.

Cache slot translated to the wrong position ID. The cache tensor is valid but geometric meaning is inconsistent. The slot and the position may differ, as 0x0A allows; the bug is a broken slot-to-position translation, such as treating the physical slot as the RoPE position.

Prompt K recomputed with a different position convention from cached K. Prefill/decode mismatch.

Mask and bias confusion

Treating RoPE as scores += bias. Wrong object. RoPE transforms Q/K before scoring.

Relying on RoPE instead of a causal mask. Future keys remain visible.

Applying mask before RoPE. Mask applies to score visibility; RoPE applies to Q/K coordinates. Confusing those stages usually hides a deeper implementation error.

MQA/GQA and KV-head layout bugs

Treating rotation-after-repeat as inherently different from rotate-before-repeat. For shared feature-wise RoPE, repetition and rotation commute exactly. Rotate-before-repeat is normally cheaper. A real mismatch comes from different tables, pairing, head-dependent state, packed-head slicing, broadcast axes, or numerical/fusion policy—not from the order alone.

Broadcasting Q-head cos/sin onto KV-head dimension incorrectly. Shape may broadcast while head semantics break.

Packed QKV layout misread. Packed layouts can be [B,n,3,h,d] or [B,n,num_heads_q + 2*num_heads_k,d]. Slicing Q/K incorrectly rotates the wrong tensors.

Checkpoint compatibility bugs

Checkpoint trained with a different pairing convention. Valid tensors, poor logits.

Checkpoint trained with a different rope_theta. Valid tensors, wrong frequency basis.

Checkpoint trained with partial rotary but inference rotates the full head.

Checkpoint conversion permutes Q/K dimensions but not RoPE convention.

Checkpoint conversion permutes weights but not optimizer state. Inference may match while the first resumed optimizer step updates the wrong Q/K channels.

A non-persistent inv_freq buffer is treated as optional configuration. The buffer can be regenerated only if its base, width, scaling, and position policy are preserved.

Long-context scaling confusion

Applying PI, NTK-aware scaling, YaRN, LongRoPE, or xPos inside a vanilla RoPE article. Those are context-extension methods. M3.5 owns them.

Mixing a scaled-RoPE cache with an unscaled-RoPE query. Valid shapes, inconsistent geometry.

Failure-to-guard matrix

No single end-to-end perplexity check localizes these failures. The release gate should include pair algebra, tensor layout, position construction, cache transaction, and golden-logit checks because each guards a different ownership boundary.

---

0x0D Handoff to M3.5 context extension

Vanilla RoPE defines how positions become Q/K rotations:

position_id, dimension pair
    -> angle
        -> q_rot, k_rot
            -> dot product with relative-offset structure

That does not prove a model will behave well beyond the positions it saw during training. RoPE can compute rotations for very large positions, but the resulting frequency patterns may be outside the training distribution.

M3.5 opens that problem:

what happens when positions exceed the training regime?
what happens when position IDs are scaled?
what happens when rope_theta changes?
what happens when cache memory and long-context runtime become the bottleneck?

M3.4 stops at the local invariant:

Q and K must be rotated with the same coordinate convention the checkpoint learned.

---

0xFF Reference

• RoFormer: Enhanced Transformer with Rotary Position Embedding
• LLaMA: Open and Efficient Foundation Language Models
• Hugging Face Transformers LLaMA source, pinned d09f53a
• Hugging Face RoPE utilities, pinned d09f53a
• FlashAttention rotary implementation, pinned df61ab6
• Meta LLaMA model source, pinned 689c7f2
• GPT-NeoX positional embeddings, pinned ea7aefd
• mesh-transformer-jax partial rotary source, pinned f8315e3
• xPos: A Length-Extrapolatable Transformer
• Round and Round We Go! What Makes Rotary Positional Encodings Useful?
• HoPE: A Novel Positional Encoding Without Long-Term Decay
• Qwen2-VL: Multimodal Rotary Position Embedding
• The Impact of Positional Encoding on Length Generalization in Transformers
• ALiBi: Train Short, Test Long
• Extending Context Window of Large Language Models via Positional Interpolation
• Scaling Laws of RoPE-based Extrapolation
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• Base of RoPE Bounds Context Length
• Transformer Series (8) — Position Encoding
• Transformer Series (9) — Location Encoding Classification
• Transformer Series (11) — Mask
• Transformer Series (17) — RoPE
• Transformer Series (23) — Length Extrapolation
• Transformer System M3.1 — Scaled Dot-Product Attention and Masks
• Transformer System M3.2 — Multi-Head and Cross-Attention
• Transformer System M3.10 — Single-Request Inference Execution
• Transformer System M3.3 — Positional Geometry and Attention Bias
• Transformer System M3.5 — Context Extension
• Transformer System M4.2 — GPU Execution Model for Transformer Kernels
• Transformer System M5.1 — Autoregressive Serving State
• Transformer System M5.3 — KV Memory Layout and Exact Reuse
• Transformer System X1.1 — Unitization and Geometry Metadata
• Transformer System X1.2 — Encoders, Projectors, and Fusion
• Transformer System X1.3 — Masks, Training, Runtime, and State