M6.3 - LoRA Foundations: Low-Rank Updates, Gradients, and Target Modules

0x00 Overview: From Full Fine-Tuning To A Structured Update

Suppose one pretrained model must support a legal-document workflow, a customer-support assistant, and a code-review tool. All three applications can reuse the base model’s language and reasoning capabilities, but they do not need exactly the same behavior. Each has its own data, output contract, vocabulary emphasis, and failure policy.

The direct solution is to copy the pretrained checkpoint and fully fine-tune one copy per task. That gives every task the freedom to change any selected weight. It also turns adaptation into a model-sized lifecycle:

one base model
    -> full task-A training state -> full task-A checkpoint
    -> full task-B training state -> full task-B checkpoint
    -> full task-C training state -> full task-C checkpoint

The cost is not only the number of floating-point operations in one optimizer step. Each task may own model-sized gradients, optimizer moments, distributed training state, checkpoints, evaluation artifacts, and deployment copies. When a base revision changes, every derived model also needs an explicit compatibility and migration decision.

Parameter-efficient fine-tuning changes the ownership question:

Can one frozen base support many downstream behaviors by storing and optimizing only a small task-specific object?

0x00A Adaptation Changes A Function, Not Merely A File

Pretraining produces a general-purpose parameter set $\theta_0$ from a broad data distribution. A downstream task asks for a related function under a new data distribution. Fine-tuning learns a parameter change:

$$\theta' = \theta_0 + \Delta\theta$$

This notation hides several physically different objects:

base state:
    theta_0 and the exact model architecture it belongs to

optimization state:
    trainable parameters, gradients, moments, activations, and communication

task artifact:
    the values and metadata required to reconstruct the adapted function

serving state:
    loaded base weights, active adaptation, caches, and runtime buffers

Full fine-tuning permits $\Delta\theta$ to be dense over the selected parameters. That is the least restrictive adaptation hypothesis, but its trainable and task-specific state scales with the selected base surface. PEFT is not one algorithm that simply makes this tensor smaller. It is a family of ways to restrict where or how adaptation enters the model.

0x00B PEFT Methods Change Different Contracts

Four broad PEFT families are useful because they alter different execution paths:

Family	Training intervention	Forward/inference consequence	Typical artifact
additive	insert learned modules or sequence-side states	extra module, prompt, or prefix path remains active	added module or learned prompt state
selective	unfreeze a chosen subset of existing parameters	original graph, but selected base tensors differ	sparse subset or derived checkpoint
reparameterized	optimize a compact parameterization of a weight change	live branch or algebraically transformed weight	factors plus reconstruction metadata
hybrid	combine several interventions	more than one execution/state contract	composite artifact

Source-local visual from A22 - LoRA, based on the PEFT survey cited there. Gray components are frozen and pink components are learned. The diagram contrasts sequence-side prompt state, low-rank attention updates, and bottleneck adapters in the feed-forward path. These methods can have similar trainable-parameter counts while changing different tensors and runtime paths.

A bottleneck adapter inserts a new nonlinear module. Prompt or prefix tuning changes sequence-side state. Selective tuning changes chosen base tensors. LoRA is reparameterized PEFT: it leaves the pretrained matrix frozen and expresses its task update through a constrained factorization. Calling all of these methods “adapters” without naming the changed state is too imprecise for implementation or deployment.

0x00C Why Low Rank Is A Hypothesis About The Update

For one pretrained linear weight $W_0\in\mathbb{R}^{d_{out}\times d_{in}}$, full fine-tuning can learn an unrestricted dense update $\Delta W$ with $d_{out}d_{in}$ degrees of freedom. LoRA instead searches inside the family:

$$\Delta W = sBA, \qquad B\in\mathbb{R}^{d_{out}\times r}, \qquad A\in\mathbb{R}^{r\times d_{in}}$$

with $r\ll\min(d_{in},d_{out})$. The inner width limits the update to at most $r$ independent directions. This is a structural claim about the task update, not a claim that the pretrained weight is low rank or that task knowledge occupies a few individual neurons.

The intuition is that pretraining has already built a rich representation. Some downstream tasks may need to steer or recombine that representation along far fewer directions than would be needed to learn it from scratch. The LoRA paper reports empirical evidence consistent with low-rank adaptation updates in its studied settings. It does not prove that every layer, task, model, or domain shift has the same useful rank.

An analogy is a patch against a large codebase: many task variants can share one upstream tree and store only their change. The analogy breaks in two places. A textual patch is often sparse in coordinates, whereas $BA$ is generally a dense matrix update. And version control does not optimize the patch through gradient descent. LoRA constrains the update by rank, not by the number of nonzero entries.

0x00D Training, Checkpointing, And Serving Are Different Phases

A complete LoRA lifecycle is:

construct:
    load an exact base revision
    discover target modules
    inject A/B factors and freeze the intended base parameters

train:
    run the base path and adapter path
    store gradients and optimizer state only for trainable adaptation state

save:
    write adapter tensors plus configuration and base-model identity

load:
    recreate the same target-module transformation
    validate keys, shapes, orientation, scaling, and base revision

serve:
    keep a live adapter branch for switching
    or derive a merged weight when the representation and operating mode permit it

This separation prevents three common overclaims:

1. fewer trainable parameters do not mean the base weights vanish from training or inference memory;
2. a small checkpoint does not mean the adapter can be loaded onto any shape-compatible base;
3. an algebraically mergeable update does not mean every quantized, sharded, multi-adapter runtime can merge and unmerge it without additional constraints.

0x00E What This Article Must Establish

LoRA applies the structured update to each selected matrix:

$$W' = W_0 + sBA$$

The base matrix remains present and participates in every forward pass; only the low-rank branch is trainable in vanilla LoRA. That compact equation creates several questions that require separate derivations:

geometry:
    what does rank r permit the update to express?

execution:
    in what order do X, A, and B multiply?

optimization:
    which gradients exist when one factor starts at zero?

state:
    what remains frozen, trainable, checkpointed, or mergeable?

placement:
    which Transformer matrices receive an adapter?

artifact:
    how does a library inject, save, reload, activate, and merge it?

The Transformer Series A22 article establishes fine-tuning, PEFT, matrix rank, and SVD before presenting LoRA. M6.3 preserves that learning path, then extends it with explicit tensor axes, forward/backward derivations, optimizer ownership, target-module consequences, framework injection, checkpoint keys, and failure ledgers.

0x00F The Problem-Response Lineage

LoRA is easier to understand when it is placed in the sequence of problems that adaptation methods tried to solve. The sequence is not a winner’s history in which every new method replaces the previous one. Each method moves task-specific state to a different location and therefore changes a different training, artifact, or runtime contract.

Response to task adaptation	Task-specific state	Problem it addresses	Contract it leaves behind
full fine-tuning	selected base weights and model-sized optimizer state	maximize freedom to alter the pretrained function	one derived weight state per task, with full compatibility and release obligations
bottleneck adapters	added nonlinear modules inside blocks	reuse a frozen backbone while learning compact task modules	an extra inference path and adapter architecture must remain present
intrinsic-dimension reparameterization	a small coordinate vector projected into a larger update space	test whether useful fine-tuning directions occupy fewer degrees of freedom	a measurement protocol and projection map, not a low-rank matrix theorem
prefix tuning	learned layer-wise attention prefix state	steer generation while freezing model weights	additional K/V-like sequence state changes attention length and cache identity
prompt tuning	learned input embeddings	condition a frozen model with a minimal sequence-side artifact	extra input positions remain part of the request
LoRA	low-rank factors for selected weight deltas	make weight-space adaptation trainable and compact without inserting a nonlinear bottleneck	base identity, target modules, factor orientation, scaling, and merge state must be preserved

The chronology matters because several phrases that sound similar describe different claims:

motivation:
    useful adaptation may occupy a much smaller search space than all base parameters

operator:
    one selected dense update is parameterized as s * B * A

artifact:
    factors, configuration, target names, and base revision reconstruct the adaptation

runtime:
    the update is executed as a live branch or represented in a compatible merged weight

The intrinsic-dimension study supplies evidence about a low-dimensional optimization space. It does not prove that each useful matrix update has low matrix rank. Prefix-Tuning, Prompt Tuning, and the Houlsby bottleneck adapter show that a frozen base can be adapted by changing sequence-side or module-side state. LoRA v2 makes a different move: it freezes the pretrained matrix and learns a low-rank weight delta for selected modules.

The paper establishes the operator and reports its experiments. The pinned Microsoft loralib repository is separate implementation evidence for injected modules and merge behavior. The pinned PEFT v0.19.1 LoRA reference source is later framework evidence for target discovery, named adapters, configuration, and state handling. M6.3 will compare those evidence surfaces rather than treating a current library interface as if it were part of the 2021 paper.

0x01 Prerequisites: Fine-Tuning, PEFT, Rank, And Factorization

0x01A Pretraining, Fine-Tuning, And Adaptation State

These stages optimize different objects under different constraints:

Stage	Starting point	Typical trainable object	Output artifact
pretraining	random or partially initialized model	most model parameters	general base checkpoint
full fine-tuning	pretrained checkpoint	most or all selected base parameters	another full checkpoint
PEFT	pretrained checkpoint	small added or selected parameter set	base reference plus adaptation state

For one linear layer $W_0\in\mathbb{R}^{d_{out}\times d_{in}}$, full fine-tuning makes the full matrix trainable. Even if only the final weight delta is saved, training commonly carries weight-shaped state:

base or trainable weight: [d_out,d_in]
gradient:                 [d_out,d_in]
optimizer moments:        one or more [d_out,d_in] tensors

PEFT does not make forward activations or backpropagation through the model disappear. It primarily reduces which parameter tensors require gradients, optimizer state, and task-specific storage.

0x01B PEFT Is A Family, Not A Synonym For LoRA

Different PEFT methods modify different parts of the model contract:

Family	What becomes trainable	Representative idea
additive	new modules or learned states	adapters, prompt/prefix tuning
selective	a subset of existing parameters	biases or selected layers
reparameterized	a compact parameterization of a weight change	LoRA
hybrid	more than one of the above	prompt plus adapter, LoRA plus other modules

LoRA is reparameterized PEFT. At inference it can remain as a live side branch or, under compatible conditions, be algebraically merged into the base matrix. Prompt tuning changes sequence-side state; bottleneck adapters add new nonlinear modules; selective tuning changes chosen existing parameters. Their parameter counts may all be small, but their execution and deployment contracts are not interchangeable.

0x01C Matrix Rank Measures Independent Directions

The rank of a matrix is the dimension of its row space, equivalently its column space. Consider:

$$M= \begin{bmatrix} 1 & 2 & 3\\ 2 & 4 & 6 \end{bmatrix}$$

The second row is twice the first, so there is only one independent row direction and $\operatorname{rank}(M)=1$.

Now consider:

$$N= \begin{bmatrix} 1 & 2 & 3\\ 0 & 2 & 3\\ 1 & -2 & -3 \end{bmatrix}$$

The first two rows are independent, while the third satisfies:

$$N_{3,:}=N_{1,:}-2N_{2,:}$$

so $\operatorname{rank}(N)=2$, not 3. Rank counts independent directions, not nonzero entries or the number of visibly different rows.

For a product:

$$B\in\mathbb{R}^{d_{out}\times r}, \qquad A\in\mathbb{R}^{r\times d_{in}}$$

linear algebra gives:

$$\operatorname{rank}(BA)\le r$$

The inner width $r$ is therefore both a parameter budget and an expressivity constraint. It is not merely a compression ratio. Increasing $r$ enlarges the family of updates that can be represented, but also increases trainable state and adapter compute.

0x01D Low-Rank Factorization Is A Two-Stage Map

For one input row $x\in\mathbb{R}^{d_{in}}$:

$$u=Ax\in\mathbb{R}^{r}$$ $$\Delta y=Bu\in\mathbb{R}^{d_{out}}$$

The first factor extracts $r$ adapter coordinates from the input. The second maps those coordinates into output-space directions. The update can communicate through only $r$ intermediate channels:

x:         [d_in]
    A
adapter u: [r]
    B
delta y:   [d_out]

This geometric reading will later make the forward order, gradient shapes, and target-module choice easier to diagnose.

0x01E SVD Supplies Vocabulary, Not The Vanilla LoRA Algorithm

Singular value decomposition writes a matrix as:

$$M=U\Sigma V^\top$$

and a truncated SVD keeps the strongest $r$ singular directions. By the Eckart-Young-Mirsky theorem, that truncation is a best rank-$r$ approximation under the spectral and Frobenius norms. This explains why low-rank structure can capture a large fraction of some matrices’ variation.

Vanilla LoRA does not compute the SVD of an unknown future full-fine-tuning update and then store it. It directly optimizes $A$ and $B$ through task loss. The empirical claim in the LoRA paper is narrower: useful adaptation updates often exhibit low intrinsic rank or rank-deficient structure, making a low-rank constraint effective in the studied settings. That is motivation, not a theorem that every task or target module has the same optimal rank.

0x01F Target Modules Define What Behavior May Change

A Transformer contains many linear weights:

attention:
    W_Q, W_K, W_V, W_O

MLP:
    gate/up/down projections

other possible targets:
    embeddings, output head, modality projectors

Applying rank 8 to q_proj is a different adaptation hypothesis from applying rank 8 to every attention and MLP projection. The rank is local to a factor pair; the total budget is the sum across target modules. Target choice also affects runtime and cache compatibility: adapting K/V projections can alter cached key/value state, while adapting only later output-side projections has different consequences.

Source-local visual: A22 - LoRA, citing Low-Rank Adaptation for Foundation Models: A Comprehensive Review v2. The Chinese annotations state that low-rank factors perturb the attention projections and thereby change token dynamics. The important implementation reading is narrower: adapting W_Q, W_K, or W_V inserts three distinct low-rank branches, and K/V-targeted branches change the state that can be cached.

This picture should not be read as an instruction to target all three projections. The original LoRA paper’s language-model experiments concentrated on attention projections and found target choice to be empirical. A reproducible adapter must therefore record the exact module set, not only its rank.

0x01G Reading Map For The LoRA Sequence

M6.3:
    vanilla LoRA geometry, forward/backward, initialization,
    target modules, optimizer state, and checkpoint contract

M6.4:
    variants that change base precision, rank policy, scaling,
    optimization, initialization, or weight parameterization

M6.5:
    live multi-adapter kernels, residency, batching,
    hotswap, scheduling, and request/KV correctness

0x01H Claim Status And Scope

Claim	Status	Evidence	Safe interpretation
LoRA freezes pretrained weights and learns low-rank update factors.	Source-local and externally verified	A22; LoRA paper	The trainable update is smaller; the base model still exists.
A product of rank-$r$ factors has rank at most $r$.	Mathematical fact	Linear algebra	LoRA restricts the update subspace, not necessarily the rank of the frozen weight.
Standard zero-$B$, random-$A$ initialization begins as a no-op.	Source-local and externally verified	A22; LoRA paper	The model function is preserved at step zero, but the two factors do not receive identical first-step gradients.
Low-rank adaptation is supported by empirical rank-deficiency observations.	Externally verified, but not universal	LoRA paper	The evidence motivates the constraint; it does not prove every task has the same useful rank.
Target-module choice changes the adaptation hypothesis.	Source-local and implementation fact	A22; PEFT implementations	Rank alone does not define the adapter budget or what behavior can change.
Merged LoRA can remove the live branch from inference.	Mathematical fact and externally verified	LoRA paper; PEFT docs	Mergeability depends on the adapter form and deployment representation; runtime serving is owned by M6.5.

Benchmark gains and memory ratios in papers are source-reported observations under particular models, tasks, optimizers, and hardware. They are not universal LoRA constants.

0x01I Owner And Evidence Contract

The address m6-3-lora-qlora-adapter-serving keeps its historical wording. The route is historical metadata, not the current ownership boundary. The article title and body own vanilla LoRA only.

Relation	This article’s contract
owns	frozen-base delta vocabulary; vanilla LoRA factors, forward/backward paths, initialization, scaling, dropout, target injection, adapter-only checkpoint, and algebraic merge
consumes	ordinary linear and block identity from M3; training-state accounting from M4; ordinary K/V lifecycle vocabulary from M5.1
defers to M6.4	QLoRA, AdaLoRA, rsLoRA, LoRA+, LoRA-FA, PiSSA, LoftQ, DoRA, and other mechanisms that change the vanilla contract
defers to M6.5	multi-adapter residency, segmented kernels, batching, hotswap, admission, eviction, request identity, and serving rollback
defers to M6.7	intrinsic-dimension experiments, subspace reconstruction, principal angles, factor geometry, kernel-regime arguments, and theorem scope
defers to M6.9	learned routing, shared factors, adapter composition, and continual adaptation
defers to M8	generic quantized-number formats, calibration, kernels, and deployment representation

Evidence is also typed:

A22:
    source-local teaching lineage, terminology, and figure context

primary papers:
    claims made by the original method or analysis under a pinned revision

official code:
    behavior visible in a pinned implementation revision

Transformer System derivation:
    tensor, gradient, state, and failure consequences reconstructed here

Loop 1 freezes those sources and all 53 A22 figure dispositions. It does not execute a model, reproduce a benchmark, prove a production serving path, or allow later framework behavior to rewrite the historical paper contract.

0x01J Freeze Identity Before Algebra

The equation W' = W + sBA is not yet an executable contract. It does not say whether the stored base weight uses [d_out,d_in] or [d_in,d_out], which adapter owns the factors, where dropout acts, which dtype performs each multiplication, whether two module paths refer to one object, or whether a merged tensor is the canonical base artifact. Those omissions can produce shape-valid but semantically different implementations.

Loop 2 therefore freezes a small linear-layer teaching fixture before later loops derive or execute it:

State class	Frozen identity	Teaching value	Why it is not interchangeable
model	base model and revision	base-demo@revision-17	shape compatibility does not prove semantic base compatibility
target	canonical module object	module:q_proj:0	a path string is a lookup result, not the module object itself
path	canonical and alias paths	two paths resolve to module:q_proj:0	injecting both paths independently would duplicate one logical target
base parameter	parameter and storage	parameter:W0, storage:W0	a parameter identity and its backing storage identity answer different alias questions
adapter	adapter name and artifact	legal, adapter:legal:v1	rank, scale, factors, and activation state are adapter-local
factors	stored axes	A:[2,6], B:[4,2]	swapping names or orientations can remain matrix-compatible under another convention
scaling	alpha, rank, rule, value	8, 2, alpha_over_r, 4.0	the effective scalar cannot reconstruct the original rank policy by itself
dropout	site and mode	branch input only; p=0.25 in training	dropping the base input changes a different function
dtype	storage, branch compute, accumulation, output	bf16, bf16, fp32, bf16	one label such as “mixed precision” does not define cast or reduction behavior
merge	canonical and derived state	live unmerged base plus distinct derived merged storage	output equivalence does not make base and merged artifacts the same object

Stored Axes And Runtime Axes

The fixture follows the PyTorch linear storage convention used by this article:

stored:
    W0:   [d_out, d_in] = [4, 6]
    bias: [d_out]       = [4]
    A:    [r, d_in]     = [2, 6]
    B:    [d_out, r]    = [4, 2]

runtime for T = 3 rows:
    X:         [T, d_in]  = [3, 6]
    X_branch:  [T, d_in]  = [3, 6]
    U:         [T, r]     = [3, 2]
    V:         [T, d_out] = [3, 4]
    Y_base:    [T, d_out] = [3, 4]
    Y:         [T, d_out] = [3, 4]

Stored orientation and runtime multiplication are related but not identical vocabularies. A:[r,d_in] is stored once; the row-major runtime operation consumes A.T:[d_in,r]. A manifest that records only the runtime matmul shape cannot reconstruct the checkpoint orientation.

Module Aliases And Storage Ties Are Different

The fixture intentionally includes two alias cases:

module-object alias:
    two Python paths -> one module object -> one W0 parameter -> one storage

storage tie:
    embedding module and output-head module are different objects
    but their weight parameters refer to one tied storage group

An injector must canonicalize the first case by module-object identity, create one adapter slot, and rebind every alias path to the same wrapped object. It must not infer that all shared storage means one module object. Conversely, wrapping or replacing one member of a tied-weight group must not silently break an intended storage tie. Loop 6 will execute those graph-rewrite cases; Loop 2 only freezes the identities that the rewrite must preserve.

Scaling, Dropout, And Dtype Are State

The fixture records rank=2, alpha=8, and scaling_rule=alpha_over_r, which yields an effective scale of 4.0. Storing only 4.0 loses the rule needed to compare a changed rank or reload a method with another scaling convention.

Dropout belongs to the adapter branch input:

base path:       X -> W0
training branch: dropout(X, mask:forward-unique) -> A -> B -> scale
evaluation:      identity(X) -> A -> B -> scale

The base branch never consumes the adapter dropout mask. The mask is an ephemeral per-forward object, not checkpoint state. Evaluation must replace the stochastic operator with identity before live-branch/merged equivalence is tested.

The dtype contract is similarly directional:

base/factor storage:     bfloat16
branch input compute:    bfloat16
matmul accumulation:     float32 in this teaching contract
branch add/output:       restored to bfloat16
merge construction:     float32 work, then explicit target representation

This is not a claim that every LoRA kernel uses fp32 accumulation. It is a declared fixture policy that prevents later ledgers from treating storage bytes, compute dtype, accumulator dtype, and exported dtype as one number.

Live, Disabled, And Merged Are Representation States

The baseline state is LIVE_UNMERGED: W0 is frozen, adapter legal is active, and the branch executes separately. DISABLED_BASE_ONLY keeps the same base artifact but omits the branch. MERGED_DERIVED constructs a distinct runtime representation from the base digest, adapter digest, scale rule, and target dtype.

The derived merged storage must not be mislabeled as the immutable base artifact. An in-place implementation may temporarily mutate a runtime tensor and later unmerge it, as the pinned loralib and PEFT implementations demonstrate, but reproducible artifact identity still requires the original base revision and an explicit merge record. Numerical equality after merge/unmerge is a later executable claim, not something established by naming a state merged.

The Loop 2 oracle validates this manifest and rejects one-field substitutions for axes, factor ownership, scale, dropout location, dtype role, alias identity, checkpoint metadata, runtime shapes, trainable state, and merge provenance. It derives 20 trainable factor scalars and 40 bf16 adapter bytes for the fixture: 2 * 6 + 4 * 2 = 20, then 20 * 2 bytes = 40 bytes. This is a source-to-state reconciliation, not model execution or a memory benchmark.

0x02 Start From The Dense Fine-Tuning Update

Consider one frozen linear transformation:

$$y = Wx$$

with:

x: [d_in]
W: [d_out, d_in]
y: [d_out]

Full fine-tuning learns a new dense matrix:

$$W' = W_0 + \Delta W$$

where both $W_0$ and $\Delta W$ have shape:

$$[d_{out}, d_{in}]$$

If every entry of $\Delta W$ is independently trainable, the adaptation surface has:

$$P_{\text{dense update}} = d_{out}d_{in}$$

parameters.

For a square 4096 by 4096 projection:

P_dense_update = 4096 * 4096
               = 16,777,216 parameters

at bf16:
M_dense_update = 16,777,216 * 2 bytes
               = 32 MiB

That 32 MiB is only one weight-shaped copy. Training also needs some combination of gradients, optimizer moments, master weights, activation storage, and distributed-training metadata. The exact multiplier depends on optimizer and precision policy, but the central scaling problem is already visible: dense adaptation state grows like $d_{out}d_{in}$.

0x02A Factor The Update, Not The Frozen Weight

LoRA replaces the arbitrary dense update by:

$$\Delta W = BA$$

with:

$$A \in \mathbb{R}^{r \times d_{in}}$$

and:

$$B \in \mathbb{R}^{d_{out} \times r}$$

where:

$$r \ll \min(d_{in}, d_{out})$$

The parameter count becomes:

$$P_{\text{LoRA}} = rd_{in} + d_{out}r = r(d_{in}+d_{out})$$

For the same 4096 by 4096 projection with $r=8$:

A: [8, 4096]      -> 32,768 parameters
B: [4096, 8]      -> 32,768 parameters

P_LoRA             -> 65,536 parameters
bf16 adapter bytes -> 128 KiB

The ratio for this matrix is:

$$\frac{P_{\text{LoRA}}}{P_{\text{dense update}}} = \frac{r(d_{in}+d_{out})}{d_{out}d_{in}}$$

For a square $d \times d$ matrix:

$$\frac{P_{\text{LoRA}}}{P_{\text{dense update}}} = \frac{2r}{d}$$

With $d=4096$ and $r=8$:

$$\frac{2r}{d} = \frac{16}{4096} \approx 0.00390625$$

or about 0.39 percent of the dense update parameter count.

This is the parameter-efficiency argument. It does not say the base matrix is compressed, and it does not say the task update is exactly representable at rank 8.

Source: LoRA v2, Figure 1. The original diagram freezes (W), initializes one low-rank factor to zero, and trains only the two low-rank factors. Matrix names and multiplication orientation vary across implementations, so the surrounding tensor equations remain the authoritative convention for this article.

0x02B What Rank Constrains

For compatible matrices:

$$\operatorname{rank}(BA) \le \min(\operatorname{rank}(B), \operatorname{rank}(A)) \le r$$

Therefore LoRA restricts $\Delta W$ to rank at most $r$.

There are two useful geometric readings.

First, $A$ maps the input row into $r$ adapter coordinates:

$$u = Ax$$

Second, $B$ maps those coordinates into output-space directions:

$$\Delta y = Bu$$

So:

A:
    decides which combinations of input channels matter

B:
    decides which output directions those combinations can change

r:
    bounds the width of the communication channel between them

The columns of $B$ span at most $r$ output directions. The rows of $A$ define at most $r$ input-sensitive coordinates. This is stronger and more useful than saying only that the adapter is “small.”

0x02C SVD Is Motivation, Not The Training Algorithm

Any matrix has a singular value decomposition:

$$\Delta W = U\Sigma V^\top$$

The best rank-$r$ approximation under the Frobenius norm keeps the leading $r$ singular triplets:

$$\Delta W_r = U_r\Sigma_rV_r^\top$$

One possible factorization is:

$$B = U_r\Sigma_r^{1/2}$$ $$A = \Sigma_r^{1/2}V_r^\top$$

This explains why two factors can represent a rank-$r$ matrix. Vanilla LoRA does not normally compute the dense task update first and then perform this SVD. It learns $A$ and $B$ directly by gradient descent while $W_0$ remains frozen.

That distinction matters:

truncated SVD:
    starts from a known dense matrix and approximates it

LoRA training:
    searches directly inside a low-rank parameterization
    without ever materializing the full dense Delta W

It is also incorrect to infer that the pretrained $W_0$ must itself be low rank. The hypothesis concerns useful downstream updates, not the full information content of the frozen base weight.

0x03 Tensor-Level Forward Pass

Use a batch of token rows:

X: [B, S, d_in]

Most linear kernels treat the leading dimensions as a collection of rows. Flatten:

T = B * S
X2: [T, d_in]

The base projection is:

$$Y_{\text{base}} = X_2 W^\top$$

with:

W:      [d_out, d_in]
W.T:    [d_in, d_out]
Y_base: [T, d_out]

The LoRA branch applies two smaller projections:

$$U = X_2 A^\top$$ $$V = U B^\top$$

with:

A:   [r, d_in]
A.T: [d_in, r]
U:   [T, r]

B:   [d_out, r]
B.T: [r, d_out]
V:   [T, d_out]

The final output is:

$$Y = Y_{\text{base}} + sV$$

where vanilla LoRA commonly uses:

$$s = \frac{\alpha}{r}$$

Renderer-safe form:

U = X2 @ A.T
V = U @ B.T
Y = X2 @ W.T + (alpha / r) * V

The tensor path is:

[T, d_in]
    |
    | base: @ W.T
    v
[T, d_out] ----------------------+
                                     +--> add --> [T, d_out]
[T, d_in]                            |
    |                                |
    | adapter: @ A.T                 |
    v                                |
[T, r]                               |
    |                                |
    | @ B.T                          |
    v                                |
[T, d_out] -- scale alpha/r --------+

0x03A Why The Order Matters

The efficient branch computes:

$$(X_2A^\top)B^\top$$

It does not normally materialize:

$$BA \in \mathbb{R}^{d_{out}\times d_{in}}$$

during every training forward pass.

The two costs are different.

Live low-rank branch:

$$F_{\text{branch}} \approx 2T(d_{in}r + rd_{out})$$

Materialize and use a dense update:

$$F_{\text{dense update use}} \approx 2Td_{in}d_{out}$$

Materializing $BA$ can be useful for a one-time merge after training, but doing it each step would throw away the main computational structure of the parameterization.

0x03B A Shape-Checked Module

class LoRALinear(nn.Module):
    def __init__(self, base, rank, alpha, dropout_p=0.0):
        super().__init__()
        self.base = base
        self.rank = rank
        self.scale = alpha / rank
        self.dropout = nn.Dropout(dropout_p)

        d_out, d_in = base.weight.shape
        self.A = nn.Parameter(torch.empty(rank, d_in))
        self.B = nn.Parameter(torch.zeros(d_out, rank))

        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
        self.base.weight.requires_grad_(False)
        if self.base.bias is not None:
            self.base.bias.requires_grad_(False)

    def forward(self, x):
        assert x.shape[-1] == self.A.shape[1]
        base_y = self.base(x)
        low_rank = F.linear(self.dropout(x), self.A)
        delta_y = F.linear(low_rank, self.B)
        assert delta_y.shape == base_y.shape
        return base_y + self.scale * delta_y

The implementation uses the same row-major linear convention as PyTorch:

F.linear(x, weight):
    output = x @ weight.T

Confusing mathematical $Ax$ notation with framework row tensors is a common source of transposition bugs.

0x03C Dtype And Accumulation

The base weight, adapter factors, input activation, and accumulation dtype do not have to match.

A runtime may hold:

base storage:
    quantized or low precision

adapter storage:
    bf16 or fp16

matmul accumulation:
    fp32 or hardware-specific mixed precision

output:
    model activation dtype

The mathematically simple addition still needs a defined cast policy. Silent promotion can increase memory; premature down-casting can reduce numerical accuracy; unsupported mixed dtypes can force a slower kernel path. M6.4 owns quantized-base variants, but the basic tensor contract begins here.

0x03D Derive The Row Kernel From The Column Equation

The compact paper equation usually treats one activation as a column vector. With the Loop 2 storage convention,

$$y_{\text{live}} = W_0x+b+sB(Ax),$$

where:

x:   [d_in]
A:   [r, d_in]
Ax:  [r]
B:   [d_out, r]
BAx: [d_out]
W0x: [d_out]
b:   [d_out]

Framework linear kernels usually receive a row batch instead. Transpose one column result:

$$y_{\text{live}}^\top = x^\top W_0^\top+b^\top+s x^\top A^\top B^\top.$$

Stacking $T$ row vectors into $X\in\mathbb{R}^{T\times d_{in}}$ gives:

$$Y_{\text{live}} = XW_0^\top+\mathbf{1}_T b^\top+s(XA^\top)B^\top.$$

This derivation explains every transpose in the implementation. F.linear(X, A) consumes stored A:[r,d_in] and computes X @ A.T; the framework is not using a different factorization.

The rank restriction also follows from an image-space argument, not merely a memorized inequality. Every column of $BA$ is $B$ applied to one column of $A$, so every output column lies in the column space of $B$:

$$\operatorname{col}(BA) \subseteq \operatorname{col}(B).$$

Therefore:

$$\operatorname{rank}(BA) \le \operatorname{rank}(B) \le r.$$

Applying the same argument to $(BA)^\top=A^\top B^\top$ also gives $\operatorname{rank}(BA)\le\operatorname{rank}(A)$. Conversely, if a dense update has rank $q\le r$, a thin rank factorization $\Delta W=CR$ with C:[d_out,q] and R:[q,d_in] can be zero-padded to B:[d_out,r] and A:[r,d_in]. Thus rank at most $r$ is both necessary and sufficient for exact representation by this factor shape. It does not say gradient descent will find that factorization or that rank $r$ is sufficient for a task.

Parameter counting now follows from stored entries rather than from the dense product:

$$P_A = rd_{in}, \qquad P_B = d_{out}r,$$

so:

$$P_{\text{adapter}} = P_A+P_B = r(d_{in}+d_{out}).$$

No $d_{out}\times d_{in}$ update tensor is trainable or checkpointed in the live representation. A later merged weight is a derived representation, not an additional trainable parameter in this accounting.

0x03E Separate Recurring Work From One-Time Merge Work

Using one multiply and one add per inner-product term, the two live branch matmuls require:

$$F_{XA^\top}=2T d_{in}r,$$

and:

$$F_{UB^\top}=2T r d_{out}.$$

Therefore:

$$F_{\text{live branch, matmul}} = 2Tr(d_{in}+d_{out}).$$

Scaling and adding the branch are separate elementwise work:

$$F_{\text{scale+add}} \approx 2Td_{out}.$$

The dense base projection remains:

$$F_{\text{base}} \approx 2Td_{in}d_{out}.$$

The logical live schedule also produces two branch intermediates:

$$N_U=Tr, \qquad N_V=Td_{out}.$$

An implementation may fuse scale/add or avoid materializing all of $V$, but that is a physical-kernel optimization over the same logical operator. It must not be used to erase the rank-width intermediate from the derivation.

Merging has a different amortization boundary. Constructing $BA$ once costs approximately:

$$F_{\text{form }BA} \approx 2d_{out}rd_{in},$$

followed by $d_{out}d_{in}$ scalar additions into the chosen merged representation. Paying this on every training forward would replace a rank-structured recurring path with dense update construction. Paying it once for a compatible deployment can remove the live branch from each later forward.

For the Loop 2 teaching fixture:

T=3, d_in=6, d_out=4, r=2

stored adapter parameters:
    2*6 + 4*2 = 20

live branch matmul FLOPs:
    2*3*(6*2 + 2*4) = 120

dense update-use FLOPs:
    2*3*6*4 = 144

one-time B@A construction FLOPs:
    2*4*2*6 = 96

logical branch intermediates:
    U = 3*2 = 6 elements
    V = 3*4 = 12 elements

The tiny fixture is chosen for exact inspection, not to demonstrate speed. Its rank is not very small relative to the matrix dimensions. For the earlier square d=4096,r=8 example, the branch-to-dense-update matmul ratio is $2r/d\approx0.00390625$ before elementwise work and kernel effects.

0x03F Prove Live-To-Merged Equivalence, Then State Its Limits

Define the derived merged weight in exact arithmetic:

$$W_{\text{merged}} = W_0+sBA.$$

Its row-batch forward is:

$$\begin{aligned} Y_{\text{merged}} &=XW_{\text{merged}}^\top+\mathbf{1}_Tb^\top\\ &=X(W_0+sBA)^\top+\mathbf{1}_Tb^\top\\ &=XW_0^\top+sXA^\top B^\top+\mathbf{1}_Tb^\top\\ &=Y_{\text{live}}. \end{aligned}$$

The third line uses transpose distributivity, $(BA)^\top=A^\top B^\top$, and matrix associativity. The bias is unchanged because vanilla LoRA in this fixture adapts only the weight.

The Loop 3 exact-integer oracle uses the frozen Loop 2 shapes with scale=4 and obtains:

U = X @ A.T
  = [[0, 8], [2, 2], [7, -3]]

V = U @ B.T
  = [[0, 8, -8, 8], [2, 2, 0, -2], [7, -3, 10, -17]]

Y_live = Y_merged
       = [[4, 34, -26, 30],
          [14, 7, 2, 0],
          [24, -5, 45, -75]]

That equality is an algebraic contract under the fixture. A deployment claim needs more conditions:

Condition	Why it is required	What breaks it
linear additive branch	associativity can fold the update into one weight	nonlinear bottleneck or input-dependent factors
evaluation-mode dropout	the branch must be deterministic	sampled training dropout mask
fixed adapter and scale	one derived weight represents one fixed update	per-request adapter selection or token-dependent routing
identical base, target, and bias	the merged tensor must modify the same operator	wrong base revision, alias, target, or bias policy
explicit merge dtype	$BA$, addition, cast, and storage must be defined	quantization, overflow, or implicit promotion
preserved artifact lineage	rollback needs the immutable base plus adapter metadata	treating the merged copy as the canonical base

In finite precision, different accumulation orders and casts mean the live and merged paths need not be bitwise identical even when they implement the same real-valued expression. The later executable oracle must therefore declare tolerances and representations; it may not replace this algebraic proof with an unqualified allclose call.

0x04 Backward Pass: Which Factor Learns First?

Let:

$$G = \frac{\partial \mathcal{L}}{\partial Y} \in \mathbb{R}^{T \times d_{out}}$$

and:

$$U = X_2A^\top$$ $$V = UB^\top$$ $$Y = X_2W^\top + sV$$

Because $W$ is frozen, the trainable gradients are for $A$ and $B$.

For $B$:

$$\frac{\partial \mathcal{L}}{\partial B} = sG^\top U$$

Shape check:

G.T: [d_out, T]
U:   [T, r]
dB:  [d_out, r]

For the intermediate $U$:

$$\frac{\partial \mathcal{L}}{\partial U} = sGB$$

Shape check:

G:   [T, d_out]
B:   [d_out, r]
dU:  [T, r]

For $A$:

$$\frac{\partial \mathcal{L}}{\partial A} = \left(\frac{\partial \mathcal{L}}{\partial U}\right)^\top X_2 = s(GB)^\top X_2$$

Shape check:

dU.T: [r, T]
X2:   [T, d_in]
dA:   [r, d_in]

The adapter contribution to the input gradient is:

$$\frac{\partial \mathcal{L}}{\partial X_2}\bigg|_{\text{adapter}} = \frac{\partial \mathcal{L}}{\partial U} A = sGBA$$

The base branch also contributes:

$$\frac{\partial \mathcal{L}}{\partial X_2}\bigg|_{\text{base}} = GW$$

Freezing $W$ means not storing or applying $dW$. It does not stop gradients from flowing through the base projection into earlier activations.

0x04A Derive Every Gradient From One Differential

The formulas above are easy to memorize and easy to transpose incorrectly. Derive them once from the row-batch graph instead.

Let $\widetilde X$ denote the input used by the adapter branch. In evaluation mode, or when branch dropout is disabled, $\widetilde X=X_2$. Define:

$$U=\widetilde X A^\top, \qquad Y=X_2W^\top+sUB^\top+b$$

For a scalar loss, the upstream gradient is $G=\partial\mathcal L/\partial Y$. The differential identity:

$$\mathrm d\mathcal L = \operatorname{tr}\left(G^\top\mathrm dY\right)$$

turns each local differential into a gradient with the same storage axes as its tensor.

Holding $U$ fixed while varying $B$ gives:

$$\mathrm dY_B = sU\,\mathrm dB^\top$$

and therefore:

$$\begin{aligned} \mathrm d\mathcal L_B &= s\operatorname{tr}\left(G^\top U\,\mathrm dB^\top\right)\\ &= s\operatorname{tr}\left((G^\top U)^\top\mathrm dB\right). \end{aligned}$$

Matching this to $\mathrm d\mathcal L=\operatorname{tr}((\partial\mathcal L/\partial B)^\top\mathrm dB)$ yields:

$$\boxed{ \frac{\partial\mathcal L}{\partial B}=sG^\top U }$$

Holding $B$ fixed while varying $U$ gives:

$$\mathrm dY_U=s\,\mathrm dU\,B^\top$$

so cyclic permutation inside the trace yields:

$$\boxed{ \frac{\partial\mathcal L}{\partial U}=sGB }$$

Now vary $A$ through $U=\widetilde X A^\top$:

$$\mathrm dU_A=\widetilde X\,\mathrm dA^\top.$$

Substituting the already-derived $\partial\mathcal L/\partial U$ gives:

$$\begin{aligned} \mathrm d\mathcal L_A &= \operatorname{tr}\left( \left(\frac{\partial\mathcal L}{\partial U}\right)^\top \widetilde X\,\mathrm dA^\top \right)\\ &= \operatorname{tr}\left( \left[ \left(\frac{\partial\mathcal L}{\partial U}\right)^\top \widetilde X \right]^\top \mathrm dA \right), \end{aligned}$$

hence:

$$\boxed{ \frac{\partial\mathcal L}{\partial A} = s(GB)^\top\widetilde X }$$

The branch-input adjoint is:

$$\boxed{ \frac{\partial\mathcal L}{\partial\widetilde X} = sGBA }$$

while the frozen base path still contributes:

$$\boxed{ \frac{\partial\mathcal L}{\partial X_2}\bigg|_{\text{base}} = GW }$$

The Loop 3 matrices make this trace executable. Freeze:

$$G= \begin{bmatrix} 1 & -1 & 2 & 0\\ 0 & 2 & -1 & 1\\ 2 & 0 & 1 & -2 \end{bmatrix}, \qquad s=4.$$

The exact-integer oracle then derives:

U =
[[0, 8],
 [2, 2],
 [7, -3]]

dB = s * G.T @ U =
[[56, 8],
 [16, -16],
 [20, 44],
 [-48, 32]]

dU = s * G @ B =
[[12, -12],
 [-12, 16],
 [28, -12]]

dA = dU.T @ X =
[[68, 12, -52, 60, 76, -68],
 [-36, -8, 44, -8, -64, 44]]

dX_adapter = dU @ A =
[[12, -24, -24, 36, -12, 12],
 [-12, 32, 28, -40, 16, -12],
 [28, -24, -40, 68, -12, 28]]

dX_base = G @ W =
[[5, -3, 3, -3, 1, 3],
 [-3, 5, -1, 4, -1, 0],
 [6, -5, 2, -2, 5, 1]]

With $U$ retained from the forward pass, the four adapter-backward matrix multiplications cost:

$$\begin{aligned} F_{dB} &= 2Td_{out}r,\\ F_{dU} &= 2Td_{out}r,\\ F_{dA} &= 2Trd_{in},\\ F_{d\widetilde X} &= 2Trd_{in}. \end{aligned}$$

For $T=3$, $d_{in}=6$, $d_{out}=4$, and $r=2$, that is $48+48+72+72=240$ matrix-multiplication FLOPs. Recomputing $U$ adds $2Td_{in}r=72$ FLOPs. The base input adjoint $GW$ still costs $2Td_{out}d_{in}=144$ FLOPs, but a frozen $W$ does not require materializing the $d_{out}\times d_{in}$ parameter gradient $G^\top X_2$. These are local operator ledgers, not an end-to-end training-speed claim.

0x04B Branch Dropout, Frozen Bias, And Gradient Ownership

Training-mode LoRA commonly drops only the branch input:

$$\widetilde X = \operatorname{Dropout}(X_2;M,p) = \frac{M}{1-p}\odot X_2,$$

where $M$ is the sampled Bernoulli mask under inverted-dropout semantics. The base projection still consumes the undropped $X_2$.

The total input gradient is therefore:

$$\boxed{ \frac{\partial\mathcal L}{\partial X_2} = GW + J_{\operatorname{Dropout}}(X_2)^\top \left(sGBA\right) }$$

For the elementwise mask above:

$$J_{\operatorname{Dropout}}(X_2)^\top Z = \frac{M}{1-p}\odot Z.$$

This establishes three distinct contracts:

State	Mathematical derivative	Stored trainable gradient	Optimizer update
frozen base weight $W$	$G^\top X_2$ exists	omitted when $W$ is frozen	none
frozen base bias $b$	$\sum_t G_{t,:}$ exists	omitted when $b$ is frozen	none
adapter factors $A,B$	derived above	required	yes
branch input $\widetilde X$	$sGBA$	activation adjoint, not a parameter gradient	propagated through dropout

For the exact fixture:

$$\frac{\partial\mathcal L}{\partial b} = \sum_{t=1}^{T}G_{t,:} = \begin{bmatrix} 3 & 1 & 2 & -1 \end{bmatrix}.$$

Reporting this local derivative does not make the bias trainable. Likewise, evaluation-mode merge equivalence does not merge a sampled training dropout mask into one fixed weight.

0x04C Zero-$B$ Initialization Is Asymmetric

The usual initialization is:

A:
    random

B:
    zero

At step zero:

$$BA = 0$$

so:

$$Y = X_2W^\top$$

The pretrained function is preserved exactly up to dtype and implementation behavior.

But inspect the first gradients.

Since $B=0$:

$$\frac{\partial \mathcal{L}}{\partial A} = s(GB)^\top X_2 = 0$$

Meanwhile:

$$\frac{\partial \mathcal{L}}{\partial B} = sG^\top(X_2A^\top)$$

which is generally nonzero because $A$ is random.

Therefore:

first optimizer step:
    B can move
    A receives zero gradient

later steps:
    once B is nonzero, both factors can learn

This is an implementation-level consequence of the no-op initialization. It is also one motivation for initialization variants such as PiSSA and gradient-aligned methods, which M6.4 compares directly.

0x04D Four Initialization Regimes Have Different First Gradients

A zero initial update is not enough to characterize learnability. The factor that is zero determines which factor can move first.

Initial $A$	Initial $B$	$BA$ at step zero	$dA$	$dB$	Branch $dX_2$	First-order consequence
random/nonzero	zero	zero	zero	generally nonzero	zero	standard LoRA: $B$ moves first
zero	random/nonzero	zero	generally nonzero	zero	zero	reversed asymmetry: $A$ moves first
zero	zero	zero	zero	zero	zero	dead first-order start
nonzero	nonzero	not guaranteed zero	generally nonzero	generally nonzero	generally nonzero	pretrained function is not guaranteed to be preserved

The final row is deliberately qualified. Two nonzero factors can multiply to zero in a degenerate construction, but nonzero/nonzero initialization does not provide the structural no-op guarantee supplied by one zero factor.

For the frozen Loop 3 tensors and the same $G$, the oracle does not merely label these cases. It substitutes zero matrices of the exact stored shapes and proves:

random A, zero B:
    dA = 0
    dB = [[56, 8], [16, -16], [20, 44], [-48, 32]]

zero A, random B:
    dA = [[68, 12, -52, 60, 76, -68],
          [-36, -8, 44, -8, -64, 44]]
    dB = 0

zero A, zero B:
    dA = 0
    dB = 0

0x04E The First Plain-SGD Update Carries The Scale Twice

The scale appears once in the factor gradient and once again when the changed factor is mapped back into $\Delta W=sBA$.

Under the standard $A_0\ne0$, $B_0=0$ initialization and plain SGD with learning rate $\eta$:

$$A_1=A_0$$

because $dA_0=0$, while:

$$\begin{aligned} B_1 &= B_0-\eta\frac{\partial\mathcal L}{\partial B}\bigg|_0\\ &= -\eta sG^\top(\widetilde X A_0^\top). \end{aligned}$$

The first nonzero dense update represented by the factors is therefore:

$$\boxed{ \Delta W_1 = sB_1A_1 = -\eta s^2G^\top(\widetilde X A_0^\top)A_0 }$$

For the exact fixture with $\eta=1$, the coefficient tensors are:

B1 =
[[-56, -8],
 [-16, 16],
 [-20, -44],
 [48, -32]]

DeltaW1 = s * B1 @ A0 =
[[-224, -64, 192, -416, -32, -224],
 [-64, 128, 128, -192, 64, -64],
 [-80, -352, -96, 16, -176, -80],
 [192, -256, -320, 512, -128, 192]]

This $s^2$ statement is narrowly scoped to the first plain-SGD step with fixed $G$, zero $B$, unchanged $A$, and no clipping, regularization, momentum, adaptive preconditioning, loss scaling, or mixed-precision overflow. It is not a general claim that optimization behavior scales quadratically with $s$. Loop 5 owns that sensitivity analysis.

0x04F Gradient Pseudocode

def lora_backward_shapes(X2, X_branch, W, A, B, grad_y, scale, dropout_backward):
    # X2:     [T, d_in]
    # X_branch is dropout(X2) for the adapter branch only.
    # W:      [d_out, d_in], frozen but still used for grad_X_base.
    # A:      [r, d_in]
    # B:      [d_out, r]
    # grad_y: [T, d_out]

    U = X_branch @ A.T
    grad_B = scale * grad_y.T @ U

    grad_U = scale * grad_y @ B
    grad_A = grad_U.T @ X_branch

    grad_X_branch = grad_U @ A
    grad_X_adapter = dropout_backward(grad_X_branch)
    grad_X_base = grad_y @ W
    grad_X = grad_X_base + grad_X_adapter

    local_grad_bias = grad_y.sum(axis=0)  # derivative exists even if bias is frozen

    assert grad_B.shape == B.shape
    assert grad_A.shape == A.shape
    assert grad_X_adapter.shape == X2.shape
    assert grad_X.shape == X2.shape
    return grad_A, grad_B, grad_X, local_grad_bias

The pseudocode exposes the local derivatives but intentionally does not allocate grad_W or an optimizer-visible base-bias gradient. Loss scaling, distributed reductions, saved-tensor policy, and fused-kernel details remain separate implementation decisions.

0x05 Rank, Scale, Dropout, Precision, And Conditioning Need Separate Experiments

The equation

$$Y=X_2W^\top+sX_2A^\top B^\top, \qquad s=\frac{\alpha}{r}$$

contains several controls that are often collapsed into one phrase such as “LoRA rank 16.” That phrase does not determine the executed operator. Changing $r$ changes factor shapes, parameter state, logical work, and the maximum rank of $BA$. Under vanilla $\alpha/r$ scaling, it can also change $s$. Changing $\alpha$ with fixed factors changes the forward branch and all branch adjoints. Dropout changes the training operator but not the evaluation operator. Dtype and cast placement can make two real-arithmetic-equivalent paths numerically different. Finally, many pairs $(A,B)$ represent the same dense update while having very different factor magnitudes and gradients.

This section therefore uses one-axis contracts rather than a single benchmark table:

rank sweep:
    state which factors are truncated, padded, or retrained
    state whether alpha or effective scale is held fixed

alpha sweep:
    hold rank, factors, input, upstream gradient, and optimizer rule fixed

dropout sweep:
    hold the deterministic factors fixed and identify the sampled mask

precision sweep:
    hold the real-valued graph fixed and move only declared cast boundaries

conditioning sweep:
    preserve BA exactly while changing the factor coordinates

The checked-in Loop 5 oracle applies those controls to the exact Loop 3/4 fixture. It is a local algebra and representation experiment, not a training-quality result.

0x05A Rank Changes Four Ledgers, Not One Number

For one adapted matrix,

$$P_{adapter}(r)=r(d_{in}+d_{out})$$

and the two live-branch matrix multiplications require

$$F_{branch}(r) = 2Trd_{in}+2Trd_{out} = 2Tr(d_{in}+d_{out})$$

logical FLOPs under the convention that one multiply-add counts as two FLOPs. The representation ceiling follows from

$$\operatorname{rank}(BA) \le \min(\operatorname{rank}(B),\operatorname{rank}(A)) \le r.$$

These are three different consequences of $r$: stored adapter state, recurring branch work, and an upper bound on dense-update rank. A fourth consequence appears through $s=\alpha/r$.

The exact fixture has $T=3$, $d_{in}=6$, and $d_{out}=4$:

Factor construction	$r$	$\alpha$	$s$	Params	Forward matmul FLOPs	Actual $\operatorname{rank}(BA)$
keep only factor coordinate 1	1	4	4	10	60	1
Loop 3 baseline	2	8	4	20	120	2
append one zero row/column, hold $s$	3	12	4	30	180	2

The padded rank-3 factors represent exactly the same $BA$ as the rank-2 factors. They still allocate 50% more adapter parameters and logical branch matmul work. The unused coordinate does not become useful merely because the declared rank increased. Conversely, the rank-1 truncation changes the represented dense update; it is not a cost-only perturbation.

Now hold $\alpha=8$ while padding from $r=2$ to $r=3$. The factors still represent the same unscaled $BA$, but vanilla scaling changes from $4$ to $8/3$, so the branch output changes. That experiment varies rank and effective scale. Holding $s=4$ instead requires changing $\alpha$ from 8 to 12. A fair rank study must state which of those contracts it uses and whether the added directions are initialized, inherited, or retrained.

This also explains why rank-stabilized scaling is not a footnote inside vanilla LoRA. It changes the rank-to-scale rule itself. M6.4 owns rsLoRA, adaptive rank allocation, and methods that change the baseline training policy.

0x05B Alpha Scales The Local Graph, But Not All Training Dynamics

Hold $X_2$, $A$, $B$, $G=\partial L/\partial Y$, and $r$ fixed. Define the unscaled branch

$$V=X_2A^\top B^\top.$$

Then

$$\Delta Y=sV,$$

and the Loop 4 adjoints become

$$\frac{\partial L}{\partial B} =sG^\top(X_2A^\top),$$ $$\frac{\partial L}{\partial A} =s(GB)^\top X_2,$$ $$\left.\frac{\partial L}{\partial X_2}\right|_{adapter} =sGBA.$$

For that frozen local graph, multiplying $s$ by $c$ multiplies the branch output and those three adjoints by $c$; their squared Frobenius norms multiply by $c^2$. LoRA v2 defines the vanilla $\alpha/r$ scale and gives a narrower optimizer-conditioned motivation for it. The fixture below isolates the algebra before any optimizer claim is made:

$s$	branch checksum	$\|\partial L/\partial B\|_F^2$	$\|\partial L/\partial A\|_F^2$	$\|\partial L/\partial X\vert_{adapter}\|_F^2$
1	7	586	1,929	959
2	14	2,344	7,716	3,836
4	28	9,376	30,864	15,344

This is not a universal loss-curve law. In an actual optimization run, changing $s$ changes logits and therefore usually changes $G$; clipping, weight decay, loss scaling, momentum, adaptive moments, and finite precision can all alter the update. The $s^2$ result from Loop 4 remains narrower still: with random $A_0$, zero $B_0$, fixed $G$, and one plain-SGD step,

$$\Delta W_1 = -\eta s^2G^\top(X_2A_0^\top)A_0.$$

The extra $s$ comes from converting the factor update back into the scaled dense delta. It does not justify treating $\alpha$ as an optimizer-independent branch-strength knob.

0x05C Inverted Dropout Preserves The Mean, Not The Realization Or Variance

Let $m_i\sim\operatorname{Bernoulli}(1-p)$ independently and use inverted dropout on one branch-input row $x$:

$$D_p(x)_i = \frac{m_i}{1-p}x_i.$$

Because $\mathbb{E}[m_i]=1-p$,

$$\mathbb{E}[D_p(x)_i]=x_i.$$

With

$$C=A^\top B^\top \in\mathbb{R}^{d_{in}\times d_{out}}, \qquad z=sD_p(x)C,$$

linearity gives

$$\mathbb{E}[z]=sxC.$$

That expectation does not make one sampled training forward equal to evaluation. The component variance is

$$\operatorname{Var}[D_p(x)_i] = \frac{p}{1-p}x_i^2,$$

and independence yields the output covariance

$$\operatorname{Cov}(z) = s^2\frac{p}{1-p} C^\top\operatorname{diag}(x_1^2,\ldots,x_{d_{in}}^2)C.$$

For the fixture’s first row, $p=1/2$, $s=4$, and all $2^6=64$ masks, exact enumeration gives

$$\mathbb{E}[z]=[0,32,-32,32]$$

and

$$\operatorname{Cov}(z)= \begin{bmatrix} 96&-32&128&-224\\ -32&416&-448&480\\ 128&-448&576&-704\\ -224&480&-704&928 \end{bmatrix}.$$

The off-diagonal terms appear because the same dropped input coordinates feed multiple output coordinates. At $p=0$, the stochastic branch reduces to the deterministic branch. As $p\rightarrow1$, the inverted-dropout multiplier and variance diverge; that limit is not a valid way to disable the adapter.

The base path still consumes the undropped $X_2$:

$$Y_{base}=X_2W^\top, \qquad \Delta Y=sD_p(X_2)A^\top B^\top.$$

The sampled mask is ephemeral forward state needed by backward. It is not adapter checkpoint state, and it cannot be folded into one deterministic merged weight. Evaluation disables dropout before live/merged equivalence is tested.

0x05D Equivalent Dense Updates Can Have Unequal Factor Conditioning

The factorization of one dense delta is not unique. For any invertible

$$R\in\mathbb{R}^{r\times r},$$

define

$$A'=RA, \qquad B'=BR^{-1}.$$

Then

$$B'A'=BR^{-1}RA=BA.$$

The forward function and merged dense update are unchanged in exact arithmetic, but the factor-coordinate gradients are not. From the same $G$,

$$\frac{\partial L}{\partial B'} = \frac{\partial L}{\partial B}R^\top, \qquad \frac{\partial L}{\partial A'} = R^{-\top}\frac{\partial L}{\partial A}.$$

The fixture applies the integer shear

$$R_n= \begin{bmatrix} 1&n\\0&1 \end{bmatrix}, \qquad R_n^{-1}= \begin{bmatrix} 1&-n\\0&1 \end{bmatrix}.$$

Every row below has the same $BA$, the same branch output, and the same merge result:

$n$	$\|A'\|_F^2$	$\|B'\|_F^2$	$\|\partial L/\partial A'\|_F^2$	$\|\partial L/\partial B'\|_F^2$
0	14	9	30,864	9,376
1	15	21	78,672	11,728
4	102	129	479,760	58,144

Thus same BA does not mean same optimizer trajectory. A shared scalar learning rate acts on factor coordinates, not directly on $BA$, and finite-range dtypes see the factor magnitudes before the product cancels them. This is a parameterization symmetry in the forward function, not an optimization symmetry. M6.7 owns the broader factor-geometry and training-dynamics theory; this section records the implementation consequence needed by vanilla LoRA.

0x05E Precision Is A Cast Schedule, Not One Dtype Label

A reproducible LoRA configuration must distinguish at least:

Role	Object or operation	Why it matters
storage	$W$, $A$, $B$	values can be rounded, underflowed, or overflowed before execution
branch input	cast of $X_2$ after dropout	controls values entering the low-rank path
matmul accumulation	$X_2A^\top$ and $UB^\top$ reductions	controls cancellation and reduction error
scale/add	multiplication by $s$ and addition to the base output	changing order can change rounding
merge construction	$sBA$ and addition into $W$	a dense materialization has its own compute and storage dtypes
output	cast after base-plus-branch add	a small branch can disappear next to a large base value

One binary16 teaching probe exposes why cast placement is part of the operator. Let

$$A=2^{-12}, \qquad B=2^{-13}, \qquad s=2.$$

The real branch value is

$$sBA=2\cdot2^{-25}=2^{-24},$$

the smallest positive binary16 subnormal. If $BA=2^{-25}$ is first rounded to binary16, it becomes zero, and multiplying the rounded value by 2 cannot recover it. If the product and scale remain in a wider compute dtype and only $sBA$ is cast, $2^{-24}$ survives. Both paths may be described casually as “fp16 LoRA,” but they are not the same cast schedule.

Even a surviving $2^{-24}$ branch is lost when added to a binary16 base output near 1 and then rounded, because it is far below the local spacing around 1. The correct lesson is not that one fixed dtype always fails. It is that storage, accumulation, scale/add, merge, and output casts must be specified separately, and live/merged parity needs declared tolerances and reference precision. Generic quantized-number theory belongs to M8.1; quantized-base LoRA variants belong to M6.4.

0x05F Sensitivity Pseudocode Keeps The Axes Separate

def vanilla_lora_sensitivity_case(X, A, B, G, alpha, rank, dropout, casts):
    assert A.shape[0] == rank
    assert B.shape[1] == rank
    scale = alpha / rank

    X_branch, mask = dropout.forward(X)       # base X is unchanged
    U = casts.matmul_accum(X_branch, A.T)
    V = casts.matmul_accum(U, B.T)
    delta_y = casts.scale_and_output(V, scale)

    grad_B = scale * G.T @ U
    grad_U = scale * G @ B
    grad_A = grad_U.T @ X_branch
    grad_X_branch = grad_U @ A
    grad_X_adapter = dropout.backward(grad_X_branch, mask)

    return {
        "delta_y": delta_y,
        "grad_A": grad_A,
        "grad_B": grad_B,
        "grad_X_adapter": grad_X_adapter,
        "state_elements": rank * (A.shape[1] + B.shape[0]),
        "logical_forward_flops": 2 * X.shape[0] * rank * (A.shape[1] + B.shape[0]),
    }

The function returns mechanism observables rather than a quality score. A rank experiment must construct new factors explicitly; an alpha experiment reuses the same factors; a dropout experiment enumerates or records masks; a precision experiment changes casts; and a conditioning experiment transforms $(A,B)$ while asserting $B'A'=BA$.

0x05G Merge Is An Algebraic Equivalence With Conditions

For a fixed adapter and compatible real-arithmetic graph:

$$X_2W^\top + sX_2A^\top B^\top = X_2(W+sBA)^\top$$

because:

$$(BA)^\top = A^\top B^\top$$

That is why a standard LoRA branch can be merged.

The equivalence assumes:

the adapter is input-independent
the same scale applies to every row
dropout is disabled
the live and merged paths declare compatible cast and accumulation schedules
no request needs a different adapter for the same base weight

The algebra proves equality over real arithmetic. It does not prove bitwise equality after different low-precision operation orders. Conditional adapters, quantized merge paths, adapter composition, and multi-tenant serving introduce additional constraints. M6.5 owns those runtime cases.

0x06 Target Modules Are A Graph-Rewrite Contract

Applying rank 8 everywhere is not the same experiment as applying rank 8 to two projections. But even a list such as q_proj,v_proj is still incomplete. A production injector must answer four different questions:

selection identity:
    which path strings match the target rule?

object identity:
    do two matched paths resolve to the same module object?

packed-slice identity:
    does one module store Q, K, and V as row slices of one matrix?

parameter/storage identity:
    do distinct modules or Parameters share one underlying storage?

Those identities can coincide, but they are not interchangeable. A path is a lookup route. A module object is one callable node. A Parameter is an autograd leaf. A storage is the memory backing one or more tensor views. A packed slice is a semantic interval inside one parameter. The injector needs all of them before it mutates the graph.

For a decoder block, possible target surfaces include:

attention:
    q_proj, k_proj, v_proj, o_proj
    or semantic slices inside one packed qkv_proj

MLP:
    gate_proj, up_proj, down_proj
    or slices inside a fused gate/up projection

other:
    embedding table, output head, modality projector
    selected expert matrices

Each target changes both the parameter ledger and what transformations can move.

Target surface	Directly changes	Typical consequence
Q projection	query subspace and attention scores	changes which cached keys are selected
K projection	key representation written during prefill/decode	changes future attention compatibility
V projection	content retrieved after attention weighting	changes routed information without directly changing score geometry
O projection	recombination of head outputs	changes how attention output enters residual stream
MLP gate/up	nonlinear feature expansion and gating	can add much more adapter state than Q/V-only
MLP down	projection back to residual width	changes how expanded features update residual stream
embedding/head	token input or output interface	can be vocabulary-sized and may need tying rules
MoE experts	expert-local transformations	adds routing and placement questions from M6.2

This is why “LoRA rank 8” is not a complete configuration. The target matcher, module graph, slice boundaries, alias groups, tie groups, factor orientation, and merge policy are part of the adaptation hypothesis.

0x06A A Path Match Is Not Yet An Injection Plan

Suppose the graph snapshot contains these occurrences:

blocks.0.self_attn.qkv_proj   -> module:qkv:0
blocks.0.cross_attn.qkv_proj  -> module:qkv:0
blocks.0.self_attn.not_qkv_proj -> module:near-match:0
embed_tokens                 -> module:embedding:0 -> storage:vocab:0
lm_head                      -> module:lm-head:0   -> storage:vocab:0

The first two paths are aliases of one module object. The near-match is a different object whose name merely contains qkv_proj. The final two paths are distinct module objects and distinct parameter identities backed by one tied storage group.

An exact-path target rule for the two QKV paths and lm_head must therefore produce:

matched path occurrences:       3
unique target module objects:   2
wrappers to construct:          2
near-match targets:             0

Counting path matches as unique modules would wrap module:qkv:0 twice. Matching by substring would also capture not_qkv_proj. Deduplicating everything by storage would incorrectly collapse the embedding and output-head roles. The Loop 6 fixture makes each mistake observable.

Frameworks expose different match surfaces. The pinned PEFT v0.19.1 injection source snapshots named_modules(), checks target rules, resolves parent/target/name, and performs compatibility checks before replacement. That is framework evidence for one version, not proof that every injector discovers aliases or tied roles identically. Loop 8 will execute the pinned framework surface; this loop defines the graph invariant independently of it.

0x06B Packed QKV Requires A Slice Manifest

A packed projection stores several semantic operators on one output axis. For GQA-like widths, write:

$$W_{qkv} = \begin{bmatrix} W_q\\ W_k\\ W_v \end{bmatrix} \in \mathbb{R}^{(d_q+d_k+d_v)\times d_{in}}.$$

The query width can differ from the key/value widths:

$$d_q \ne d_k=d_v.$$

Let $P_q$ and $P_v$ be fixed binary row-embedding matrices that place a slice-local matrix back into the packed output axis. Adapting Q and V, but not K, gives:

$$\Delta W_{qkv} = P_q\left(s_qB_qA_q\right) + P_v\left(s_vB_vA_v\right).$$

For a row batch $X$:

$$\Delta Y_{qkv} = s_qXA_q^\top B_q^\top P_q^\top + s_vXA_v^\top B_v^\top P_v^\top.$$

The K interval is zero by construction. Its rows are unchanged because neither placement matrix writes there, not because the K output happened to be numerically small.

The parameter ledger is slice-local:

$$P_{qv} = r_q(d_{in}+d_q) + r_v(d_{in}+d_v).$$

The dense packed update obeys:

$$\operatorname{rank}(\Delta W_{qkv}) \le \operatorname{rank}(s_qB_qA_q) + \operatorname{rank}(s_vB_vA_v) \le r_q+r_v.$$

Separate Q and V factors are therefore not merely an implementation of one monolithic rank-$r$ packed branch. They impose separate slice supports, ranks, and scales. A monolithic factor can couple its rank coordinates across Q/K/V rows; slice-local factors cannot write outside their declared intervals.

The exact fixture uses:

T=2, d_in=4
d_q=4, d_k=2, d_v=2
Q: r_q=2, s_q=2
V: r_v=1, s_v=3

P_Q = 2*(4+4) = 16
P_V = 1*(4+2) =  6
P_packed adapter = 22 parameters

Its branch output is:

Delta Y_qkv =
[[ 8, -6, 2, -20, 0, 0,  30, -15],
 [-4, 12, 8,  28, 0, 0, -24,  12]]
                       ^^^^^^
                       K slice remains zero

Adding that branch to the frozen packed projection gives exactly the same output as forming the selectively row-padded dense update and using a derived merged weight. The dense update has actual rank 3 in this fixture, equal to $r_q+r_v$; the inequality does not promise equality for arbitrary factors.

The pinned loralib.MergedLinear is useful implementation evidence: it builds an enabled-row mask, computes grouped low-rank updates, zero-pads disabled rows, and uses the same selective update for live and merged execution. Its contract divides out_features into equal chunks according to enable_lora. That is appropriate for its declared layout; it is not a general unequal-width GQA slice parser. Architecture-aware code must carry explicit start/stop intervals or use separate projections.

0x06C Aliases Must Share One Wrapper And One Adapter Slot

Let $\pi_1$ and $\pi_2$ be two graph paths resolving to the same module object:

$$\operatorname{resolve}(\pi_1) = \operatorname{resolve}(\pi_2) =m.$$

The rewrite must construct one wrapped object $\widehat m$ and rebind both parents:

$$\operatorname{resolve}'(\pi_1) = \operatorname{resolve}'(\pi_2) =\widehat m.$$

Wrapping each path independently creates two adapter parameter sets for one logical base operator and may destroy the alias relation. In the exact fixture, the packed object needs 22 adapter parameters. A path-wise injector would count it twice:

correct unique state:
    packed object once + lm_head once
    22 + 10 = 32 parameters

naive path state:
    packed alias path 1 + packed alias path 2 + lm_head
    22 + 22 + 10 = 54 parameters

This is not only an accounting error. If the two wrappers receive different optimizer states or adapter names, the originally tied call sites can execute different functions.

0x06D Tied Storage Makes Merge A Role-Sensitive Operation

Weight tying is a different graph relation. Let the input embedding and output head use one base matrix $E\in\mathbb{R}^{V\times d}$:

$$X_{embed}=SE, \qquad Z_{base}=HE^\top,$$

where $S$ selects vocabulary rows and $H$ contains hidden states. A live head-only LoRA branch intends:

$$X_{embed}=SE, \qquad Z_{live}=HE^\top+sHA^\top B^\top.$$

If the head adapter is merged in place into the shared canonical storage, both roles change:

$$X'_{embed}=S(E+sBA), \qquad Z'_{head}=H(E+sBA)^\top.$$

That is not the same graph. The input embedding receives an update even though it was not targeted. A safe injector therefore preserves the tied base storage during live execution and rejects an automatic canonical in-place merge when only one role in the tie group is targeted.

A deployment may instead create an explicit role-local derived head tensor, deliberately adapt every compatible role in the tie group, or keep the live branch. Each choice changes artifact identity or model semantics and needs evaluation. M2.2 owns the full embedding/head tying derivation; M6.3 owns the adapter injection and merge consequence.

For the Loop 6 head fixture:

V=6, d=4, r=1, s=2
head adapter state = 1*(4+6) = 10 parameters

Z_base =
[[1, 0, -1,  2, 1, 1],
 [0, 2,  1, -1, 2, 0]]

Delta Z_head =
[[ 8, 0, -8, 16,  8, -16],
 [-4, 0,  4, -8, -4,   8]]

The live branch changes only the output role and leaves storage:vocab:0 tied. The oracle rejects canonical in-place merge because embed_tokens is an untargeted consumer of that storage.

0x06E Injection Should Be A Preflighted Transaction

Graph mutation should happen after discovery and validation, not while target matching is still incomplete:

def plan_lora_injection(model, target_spec, adapter_name):
    # Snapshot occurrences before replacement changes traversal.
    occurrences = snapshot_paths_objects_parameters_and_storage(model)

    matched = exact_match(occurrences, target_spec.paths)
    require_all_requested_paths_matched(matched, target_spec)

    object_groups = group_by_module_object_identity(matched)
    tie_groups = discover_parameter_and_storage_ties(occurrences)

    plans = []
    for object_id, aliases in object_groups.items():
        base = aliases[0].module
        slices = resolve_semantic_slices(base, target_spec.slice_manifest)
        validate_nonoverlap_full_bounds_and_factor_shapes(base, slices)
        validate_tied_role_merge_policy(base, tie_groups, target_spec)
        plans.append(build_wrapper_plan(object_id, aliases, slices, adapter_name))

    audit_unique_parameter_and_checkpoint_ledgers(plans)
    return plans


def commit_lora_injection(model, plans):
    wrappers = {}
    for plan in plans:
        wrapper = wrappers.setdefault(plan.object_id, build_wrapper_once(plan))
        for parent, child_name in plan.parent_slots:
            setattr(parent, child_name, wrapper)

    freeze_declared_base_parameters(model)
    register_adapter_namespace(model, plans)
    assert_all_aliases_rebound_to_one_wrapper(model, plans)
    assert_tied_base_storage_policy(model, plans)
    assert_trainable_parameters_equal_manifest(model, plans)

The split between plan and commit matters. If an unmatched path, overlapping slice, incompatible rank, or unsafe tie is discovered after half the graph has already been replaced, rollback itself becomes part of correctness. A preflight plan can fail before mutation.

The checked-in standard-library oracle does not import PyTorch or PEFT. It executes the immutable graph and matrix fixture, verifies 69 one-field counterexamples, and proves only the declared teaching contract. Loop 8 will trace actual pinned framework symbols; Loop 10 will integrate forward, gradient, injection, checkpoint, and merge oracles.

0x06F Parameter Ledger Across A Block

Assume:

d_model = 4096
d_ff = 14336
r = 8
layers = 32
adapter dtype = bf16

For one square attention projection:

P = 8 * (4096 + 4096)
  = 65,536

For Q and V in every layer:

P_qv = 2 * 32 * 65,536
     = 4,194,304 parameters

M_qv_bf16 = 4,194,304 * 2 bytes
           = 8 MiB

Now include the three MLP projections.

For gate or up:

P_gate = 8 * (4096 + 14336)
       = 147,456

The down projection has reversed input/output dimensions but the same count:

P_down = 8 * (14336 + 4096)
       = 147,456

Three MLP adapters per layer:

P_mlp_per_layer = 3 * 147,456
                = 442,368

P_mlp_32_layers = 14,155,776
M_mlp_bf16      = 27 MiB

The MLP adaptation surface alone is more than three times the Q/V-only adapter state in this example. The exact model may use fused projections, GQA-specific shapes, tied weights, or different MLP widths; the ledger must follow the actual checkpoint and deduplicate module-object aliases.

0x06G Attention Cache Consequence

Changing Q and changing K/V are not runtime-equivalent.

For a request whose prefix K/V was computed without an adapter:

Q-only adaptation:
    existing base K/V may still describe the same K/V projection
    queries change against that cache

K/V adaptation:
    cached rows should reflect adapted K/V projections
    reusing base-only K/V can be semantically wrong

The exact cache-reuse contract depends on when the adapter becomes active and which projections it changes. M5 owns KV state; M6.5 owns adapter-aware serving. M6.3 establishes why target-module metadata is part of that contract.

0x06H Injection Failure Lab

Failure	Shape checks pass?	Semantic break	Required check
substring matcher captures not_qkv_proj	often	unintended operator receives trainable state	exact or explicitly declared regex match set
two paths wrap one module object twice	often	duplicate adapters and broken alias behavior	group by object identity; rebind every parent slot
equal-chunk packed mask used for unequal GQA widths	possibly	Q/K/V rows are assigned to the wrong semantic slice	explicit start/stop slice manifest
Q/V slice update writes K rows	yes	supposedly untargeted K projection changes	zero-delta assertion on the K interval
path count used as parameter count	yes	state, optimizer, and checkpoint ledgers overcount aliases	count unique adapter parameter identities
head-only delta merged into tied vocabulary storage	yes	input embeddings change unexpectedly	tie-group-aware merge admission
graph mutated before all targets validate	yes	failure leaves a partially injected model	preflight plan, then commit transaction
wrapper created with wrong storage orientation	sometimes	transposed or misplaced dense update	orientation and factor-shape contract

0x07 Training State, Checkpoint, Resume, And Merge Are Different Lifecycles

Freezing the base changes parameter-gradient ownership. It does not sever the computation graph. Activations and input gradients are still needed wherever a trainable adapter earlier in the graph depends on them, and the frozen weights remain resident for the forward and backward operators. A22 makes this distinction in its resource discussion; the retained diagram below comes from that explanation and ultimately from the Ladder Side-Tuning paper, whose comparison motivates a different side-network design precisely because ordinary PEFT still backpropagates through the backbone path.

The more subtle mistake happens after the backward pass: one training run can produce three artifacts with different purposes and state sets.

portable adapter:
    enough to reconstruct and execute the adapted function

exact-resume bundle:
    enough to continue the same optimization and data trajectory

merged release:
    a derived deployment representation with evaluation and rollback lineage

They overlap, but none is a substitute for the other two.

0x07A Derive The State Ledger By Lifetime And Owner

Let $\Theta_a$ be the set of unique trainable adapter parameter identities after the alias reconciliation in 0x06, and let

$$P_a = \sum_{\theta\in\Theta_a}\operatorname{numel}(\theta).$$

Counting matched paths would count the same adapter twice when two paths alias one wrapped object. Training state follows parameter identity, not path occurrence.

For Adam-like optimization, separate persistent state from the gradient buffer that is live only between backward and zeroing:

$$M_{\text{persistent}} = P_a\left( b_{\text{param}} +b_m +b_v +\mathbf{1}_{\text{master}}b_{\text{master}} \right),$$ $$M_{\text{gradient-live}} = P_a b_{\text{grad}},$$

and therefore

$$M_{\text{parameter-state peak}} = M_{\text{persistent}} +M_{\text{gradient-live}}.$$

This is not the whole training peak. It excludes the frozen base, activations, saved dropout masks, input gradients, allocator fragmentation, communication buckets, and temporary workspaces. Those belong in separate ledgers because their lifetimes and scaling variables differ.

For the Loop 6 graph, $P_a=32$ unique factor scalars while naive path counting gives 54. With bf16 parameters and gradients, fp32 first/second moments, and an fp32 master copy:

State	Bytes per scalar	Unique $P_a=32$	Naive path count $54$
adapter parameter	2	64 B	108 B
live gradient	2	64 B	108 B
first moment $m$	4	128 B	216 B
second moment $v$	4	128 B	216 B
master parameter	4	128 B	216 B
persistent subtotal	-	448 B	756 B
subtotal while gradients are live	-	512 B	864 B

The alias error overstates this tiny fixture by 352 bytes. At model scale the same identity mistake corrupts optimizer allocation, checkpoint size, and distributed ownership rather than merely a report.

If no master copy is used, subtract $4P_a$ bytes. If the parameter itself is stored in fp32, do not silently count a second fp32 master copy. Quantized optimizers, factored states, offload, and sharding replace the byte roles; they do not justify one universal multiplier.

For the earlier 4,194,304-parameter Q/V example, bf16 parameter/gradient plus two fp32 moments is about 48 MiB while gradients are live. Adding an fp32 master copy makes the same subtotal about 64 MiB. The frozen base and activation peak remain separate and can still dominate the job.

0x07B AdamW State Belongs To Factor Coordinates

Flatten the unique trainable factors into one conceptual vector $\theta_t$. For gradient $g_t$, Adam-style moments are

$$m_t = \beta_1m_{t-1} +(1-\beta_1)g_t,$$ $$v_t = \beta_2v_{t-1} +(1-\beta_2)g_t\odot g_t.$$

Starting from zero moments, their bias-corrected forms are

$$\widehat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t = \frac{v_t}{1-\beta_2^t}.$$

An AdamW-style decoupled update can then be written as

$$\theta_t = (1-\eta_t\lambda)\theta_{t-1} - \eta_t \frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}.$$

The AdamW paper motivates decoupling weight decay from the adaptive loss-gradient update. Here the important checkpoint consequence is simpler: $m_t$, $v_t$, the step $t$, the learning-rate schedule position, and any master copy are state. Reloading only $A$ and $B$ preserves the current function but resets the optimizer trajectory.

These moments belong to the factor coordinates. Two gauge-related pairs $(A,B)$ and $(RA,BR^{-1})$ can represent the same $BA$ while having different factor gradients and moments, as 0x05D showed. A checkpoint cannot replace factor optimizer state with one optimizer state for the dense product $BA$ without changing the optimization problem.

0x07C The Training Step Has A Canonical Save Boundary

def lora_train_step(model, batch, optimizer, scheduler, scaler=None):
    optimizer.zero_grad(set_to_none=True)

    # Frozen base parameters remain in the differentiable graph.
    logits = model(
        input_ids=batch.input_ids,
        attention_mask=batch.attention_mask,
    ).logits
    loss = cross_entropy(logits[:, :-1], batch.input_ids[:, 1:])

    if scaler is None:
        loss.backward()
        clip_grad_norm_(model.unique_adapter_parameters(), max_norm=1.0)
        optimizer.step()
        step_applied = True
    else:
        scaler.scale(loss).backward()
        scaler.unscale_(optimizer)
        clip_grad_norm_(model.unique_adapter_parameters(), max_norm=1.0)
        step_applied = scaler_step_with_explicit_overflow_result(scaler, optimizer)
        scaler.update()

    if step_applied:
        scheduler.step()

    # Canonical boundary: all state transitions for step t are complete,
    # and the next sample/dropout draw has not happened.
    optimizer.zero_grad(set_to_none=True)
    return loss.detach()

The scaler update and scheduler order are part of the run contract. If overflow causes the optimizer step to be skipped, a step-counted scheduler should not silently advance as though parameters changed. A different scheduler convention may step at another point; the checkpoint must record the convention rather than infer it from a scalar learning rate.

The ownership assertions from 0x06 still apply:

trainable_ids = {id(p) for p in model.unique_adapter_parameters()}
optimizer_ids = {
    id(p)
    for group in optimizer.param_groups
    for p in group["params"]
}

assert optimizer_ids == trainable_ids
assert no_frozen_base_parameter_has_grad(model)
assert every_alias_resolves_to_one_adapter_slot(model)

A model can look frozen while an optimizer created earlier still holds stale base parameters. Conversely, deduplicating by path can omit or duplicate parameter identities. requires_grad flags, optimizer groups, and graph aliases must reconcile to the same set.

0x07D Step-Boundary And Mid-Accumulation Checkpoints Differ

At the canonical boundary above, gradients are empty and the next stochastic event has not occurred. A resume bundle does not need gradient buffers for that policy. With gradient accumulation over $K$ microbatches, a mid-accumulation checkpoint is different:

$$G_j = \sum_{i=1}^{j} g_i, \qquad 0<j<K.$$

Exact continuation from that point needs $G_j$, the microstep $j$, loss-scaling state, and the data/RNG positions after microbatch $j$. Discarding $G_j$ and restarting the accumulation window changes the next optimizer update. Saving only at optimizer-step boundaries is simpler, but it can repeat work after a failure. The policy belongs in the manifest.

The PyTorch AMP recipe explicitly includes GradScaler.state_dict() for bitwise-accurate AMP resume and recommends saving either before a new iteration or after the scaler update. That is official framework guidance, not a claim that every hardware/software environment can replay bitwise.

0x07E Three Artifact ABIs

Artifact	Must carry	Intentionally omits	Valid question it answers
portable adapter	exact base identity/revision, graph and target manifest, adapter config, factor tensors, declared additional modules	optimizer, scheduler, data cursor, transient RNG state	can this adapted function be reconstructed?
exact-resume bundle	portable adapter plus optimizer states/steps, scheduler, scaler, global/microstep, RNG streams, sampler/data state, accumulation state, determinism environment	a derived deployment merge unless separately recorded	can this training trajectory continue from the boundary?
merged release	derived model weights, base+adapter lineage, merge/cast manifest, evaluation record, rollback target	factor optimizer state and training-control state	can this tested representation be deployed and rolled back?

Official PEFT checkpoint documentation describes adapter weights and adapter configuration as the portable PEFT artifact and notes that the original base remains required. Official PyTorch checkpoint guidance separately includes model and optimizer state for resuming training. Combining those two facts does not make a PEFT adapter directory an exact-resume bundle by itself.

0x07F Exact Resume Restores Control State Before The Next Draw

def save_step_boundary(run):
    assert run.representation == "live_unmerged"
    assert run.accumulation.micro_step == 0
    assert all(p.grad is None for p in run.unique_adapter_parameters())

    return atomic_save({
        "portable_adapter": export_adapter_and_graph_abi(run),
        "optimizer_state": run.optimizer.state_dict(),
        "scheduler_state": run.scheduler.state_dict(),
        "global_step": run.global_step,
        "grad_scaler_state": optional_state_dict(run.scaler),
        "rng_state": capture_all_declared_rng_streams(run),
        "sampler_state": run.sampler.state_dict(),
        "accumulation_state": run.accumulation.state_dict(),
        "software_and_determinism_manifest": environment_manifest(run),
    })


def resume_step_boundary(bundle):
    model = load_exact_base(bundle.base_identity)
    model = inject_from_graph_manifest(model, bundle.graph_manifest)
    load_adapter_values(model, bundle.portable_adapter)

    optimizer = build_optimizer(model.unique_adapter_parameters())
    scheduler = build_scheduler(optimizer)  # construct before state loading
    optimizer.load_state_dict(bundle.optimizer_state)
    scheduler.load_state_dict(bundle.scheduler_state)
    load_optional_scaler_state(bundle.grad_scaler_state)

    restore_accumulation(bundle.accumulation_state)
    restore_sampler(bundle.sampler_state)
    restore_rng_streams(bundle.rng_state)    # before next sample/dropout draw
    assert_reconciled_trainable_and_optimizer_ids(model, optimizer)
    return model, optimizer, scheduler

The base and injection graph are recreated before optimizer values because optimizer slots must resolve to the intended parameter identities. Distributed checkpoints may encode names or sharded identities rather than process-local object IDs, but they still need an explicit mapping.

Exact replay also has a bounded claim. PyTorch’s reproducibility note warns that complete reproducibility is not guaranteed across releases, platforms, or CPU/GPU execution. An article may prove state-complete resume for one declared environment; it should not promote that to universal cross-platform bitwise identity.

0x07G An Interrupted AdamW Fixture Makes Missing State Observable

The checked-in standard-library oracle uses four representative adapter scalars, three AdamW steps, an explicit learning-rate sequence, a stateful sample cursor, and a deterministic dropout RNG. It interrupts after step 1, serializes the complete boundary state, restores it, and runs steps 2 and 3.

Step	sample	dropout mask	learning rate	resulting parameter vector
1	2	[0,0,1,1]	0.1	[0.990000, -1.980000, 0.592561, 2.870990]
2	0	[1,0,1,1]	0.05	[0.942366, -1.970100, 0.589598, 2.811167]
3	3	[1,1,1,1]	0.025	[0.947991, -1.986362, 0.572161, 2.796129]

The uninterrupted and interrupted/resumed final states agree exactly under the oracle’s decimal arithmetic, including parameters, both moments, optimizer and scheduler steps, scaler tracker, RNG state, and sample cursor. The fixture is deliberately narrow: it does not import PyTorch or PEFT, execute AMP/distributed checkpointing, benchmark memory, validate task quality, or prove cross-release replay.

0x07H Merge Is A Release Branch, Not A Training Checkpoint

For compatible evaluation-mode linear LoRA, 0x03 proved

$$W_{\text{derived}} = W_0+sBA.$$

That algebra says how to derive a representation. It does not transfer the factor-coordinate optimizer states $(m_A,v_A,m_B,v_B)$ into a meaningful optimizer state for $W_{\text{derived}}$. Nor does it make the derived tensor the canonical base.

live training state
    immutable base + adapter + optimizer/control state

derive release candidate
    copy base -> apply admitted merge -> record casts and checksums

validate candidate
    parity tolerances + task/safety/runtime evaluation

deploy or reject
    retain immutable base, adapter, manifest, and previous release target

The tied-head case from 0x06 remains decisive: a head-only delta cannot be merged in place into storage also consumed by an untargeted embedding role. A role-local derived tensor may be valid, but it is a new artifact with its own lineage. Quantized merge and multi-adapter composition introduce further conditions owned by M6.4/M6.5.

Unmerge is also not an archival strategy. Subtracting a low-precision materialized delta can accumulate rounding error, and a framework may discard adapter machinery after merge_and_unload. Rollback should load an immutable prior artifact, not assume arithmetic reversal reconstructs it bit for bit.

0x07I Distributed Ownership Changes Placement, Not Logical Identity

Replicated data parallelism can all-reduce only adapter gradients, but every rank may still hold the frozen base. ZeRO/FSDP-style strategies may shard adapter parameters, gradients, and optimizer states. The global ledger remains over unique logical adapter identities:

$$P_a = \sum_{r=1}^{R}P_{a,r}^{\text{owned}}$$

only when each logical parameter element has one owner in that accounting. Replicated copies must not be summed as new model parameters; shards must not be mistaken for complete tensors.

PyTorch’s Distributed Checkpoint documentation describes in-place loading, per-rank checkpoint files, and load-time resharding support. Those are versioned framework capabilities. This Loop 7 oracle records only the framework-independent requirements: a stable parameter namespace, one logical value per unique adapter parameter, a save ownership rule, and an explicit resharding plan when topology changes.

0x07J Training-State Failure Lab

Failure	What was saved	First observable divergence	Required repair
adapter tensors only, then “resume”	current $A,B$	moments and bias-correction step reset	label it warm-start or restore optimizer state
scheduler cursor omitted	optimizer moments preserved	next learning rate differs	save/load scheduler state and ordering convention
dropout RNG restored after forward	nominal RNG bytes	next mask already consumed from wrong state	restore every RNG stream before next stochastic event
sampler cursor omitted	model/optimizer complete	next training example differs or repeats	checkpoint sampler/data-pipeline state
AMP scaler omitted	model/optimizer complete	overflow/step-skip trajectory can differ	save scaler at a valid boundary
mid-accumulation gradients discarded	$j$ microsteps already consumed	next optimizer update omits $G_j$	save gradient/accumulation state or checkpoint only at step boundaries
optimizer groups count aliased paths	same adapter object appears twice	duplicate or conflicting state ownership	reconcile unique parameter identities
merged release reused as resume base	dense derived weight only	factor optimizer state has no matching coordinates	resume from immutable base + live adapter bundle
arithmetic unmerge used as rollback	merged low-precision tensor	recovered base drifts after subtraction	load immutable rollback artifact
topology changed without resharding plan	rank-local shards	missing/duplicated optimizer values	stable names plus framework-supported resharding

0x07K From LoraConfig To A Changed Module Graph

The equation $W'=W_0+sBA$ does not explain how a real model acquires hundreds of factor pairs. A PEFT library must interpret configuration, discover graph owners, choose a backend wrapper, create named state, change trainability, serialize only the selected artifact, and later reconstruct that graph.

This section is tied to PEFT v0.19.1 at commit ba6a1906.... It is official implementation evidence for that revision, not a redefinition of the 2021 LoRA paper and not a promise about later PEFT releases.

Source surface	Pinned symbol	Contract observed here	What it does not prove
public mapping helper	get_peft_model	records base identity metadata and returns a generic or task-specific wrapper while mutating the supplied graph	that every base model or task wrapper follows one identical forward API
low-level mapping helper	inject_adapter_in_model	instantiates the mapped tuner and returns the mutated base model without PeftModel	prompt-method support or portable weight population by itself
wrapper lifecycle	PeftModel.__init__, save_pretrained, from_pretrained	maps PEFT type to tuner, saves selected adapters, and reconstructs a wrapper plus adapter	exact optimizer resume, which remains a Loop 7 bundle
generic tuner injection	BaseTuner.inject_adapter	snapshots named modules, checks targets, invokes method-specific replacement, and marks trainable state	complete storage-alias preservation for arbitrary graphs
LoRA model rewrite	_create_and_replace, _create_new_module	updates an existing LoRA wrapper or chooses the first matching backend dispatcher	equivalent semantics across quantized/custom/TP backends without testing them
LoRA layer state and operator	update_layer, Linear.forward, get_delta_weight, merge	owns named A/B/dropout/scaling slots, active forward additions, and merge materialization	mixed-request serving or variant behavior owned by M6.4/M6.5
adapter state translation	get_peft_model_state_dict, set_peft_model_state_dict	filters adapter state, removes a local adapter name on save, and inserts the requested name on load	semantic compatibility from tensor shapes alone

The public wrapper workflow is conceptually:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
    bias="none",
)

model = get_peft_model(
    base_model,
    config,
    adapter_name="legal",
)

The pinned call path is more specific than the public snippet:

get_peft_model
  -> PeftModel.__init__
  -> PEFT_TYPE_TO_TUNER_MAPPING[LORA]
  -> LoraModel / BaseTuner.__init__
  -> BaseTuner.inject_adapter
  -> LoraModel._create_and_replace for each match
       existing LoraLayer -> update_layer(new adapter name)
       plain base module  -> _create_new_module -> first backend match
  -> _mark_only_adapters_as_trainable

The low-level entry starts at the same tuner mapping but returns peft_model.model, not the PeftModel wrapper. It also explicitly rejects prompt learning and unsupported PEFT types. Wrapper choice, graph mutation, and weight loading are therefore separate operations even when a high-level API composes them.

0x07K1 Discovery Is A Snapshot, Then A Mutation

The pinned BaseTuner.inject_adapter first materializes list(model.named_modules()). It uses that snapshot to configure targets, discover existing adapter prefixes, match modules or state-dict-derived owners, and obtain parent/target/child triples before invoking method-specific replacement. This avoids iterating a live module generator while the graph changes.

The durable teaching algorithm is:

def pinned_injection_shape(model, config, adapter_name, optional_state_keys=None):
    named_modules = list(model.named_modules())
    validate_new_adapter_config(config)
    inspect_tied_modules(model, config)
    configure_default_or_all_linear_targets(config, model)

    matches = []
    for path, module in named_modules:
        decision = match_target(path, module, config, optional_state_keys)
        if decision.excluded:
            continue
        if decision.matched:
            parent, target, child_name = get_submodules(model, path)
            matches.append((path, parent, target, child_name))

    if no_valid_match(matches, config):
        raise ValueError("target contract did not match")

    for path, parent, target, child_name in matches:
        lora_model_create_or_update(
            config, adapter_name, target, child_name, parent, path
        )

    mark_only_declared_adapter_state_trainable(model, config)
    return matches

The source attempts several checks before replacement, but it is still not a transaction over every framework-independent identity from Loop 6. Custom module hooks, parameter targeting, lazy/meta initialization, tied modules, and backend-specific wrappers complicate the mutation boundary. A production preflight should compare the expected graph manifest against observed owners before permitting any irreversible change.

The diagnostic targeted_module_names also needs careful interpretation. It appends names as adapters are injected. In the Loop 8 two-adapter run, it is:

[q_proj, v_proj, q_proj, v_proj]

That is useful injection history, but the graph still contains only two target wrappers. Canonical graph accounting should deduplicate by module identity and then inspect the named slots owned by each wrapper.

0x07L The Injected Layer Owns Named Adapter Slots

LoraModel._create_and_replace has two materially different branches. If a target is already a LoraLayer, the method calls update_layer and adds a slot. Otherwise it calls _create_new_module; that method evaluates custom, quantized, tensor-parallel, and default dispatchers in a fixed order and accepts the first non-None wrapper. Backend dispatch is therefore part of the runtime contract, not an implementation footnote.

For an ordinary linear target, the transformed object has this state:

LoRA-wrapped linear
    base_layer.weight                 frozen

    lora_A["legal"].weight           [r, d_in]
    lora_B["legal"].weight           [d_out, r]
    dropout["legal"]
    scaling["legal"]

    lora_A["support"].weight         optional second adapter
    lora_B["support"].weight
    ...

Module dictionaries are important. The rank, scaling, factors, and activation state belong to an adapter name, not just to the base layer. Adding a second adapter should create another slot instead of overwriting the first pair.

The pinned update_layer validates positive rank, chooses dropout or identity, creates

$$A_a\in\mathbb{R}^{r_a\times d_{\text{in}}}, \qquad B_a\in\mathbb{R}^{d_{\text{out}}\times r_a},$$

and records per-adapter scale

$$s_a= \begin{cases} \alpha_a/r_a, & \text{vanilla LoRA},\\ \alpha_a/\sqrt{r_a}, & \text{rsLoRA variant}. \end{cases}$$

The second branch is mentioned here only to make the source conditional visible. M6.4 owns rsLoRA’s changed scaling contract.

The pinned linear forward has four control paths:

Runtime state	Forward behavior	Owner
adapters disabled	unmerge if necessary, then execute base	M6.3 control state
adapter_names supplied per example	mixed-batch helper	M6.5
layer already merged	execute materialized base	M6.3 representation state
live unmerged adapters	base result plus every active adapter delta	M6.3 vanilla operator

The last path is the tensor path derived in 0x03:

def peft_linear_forward(layer, x, adapter_name):
    y = layer.base_layer(x)

    if layer.adapters_disabled or adapter_name is None:
        return y

    x_adapter = x.to(layer.lora_A[adapter_name].weight.dtype)
    z = layer.lora_A[adapter_name](layer.dropout[adapter_name](x_adapter))
    delta = layer.lora_B[adapter_name](z)
    return y + layer.scaling[adapter_name] * delta

The actual pinned source loops over all active_adapters, casts the input to each adapter A dtype, accumulates each delta into the base result, and casts the final result back to the original base-result dtype. Production implementations handle more cases: mixed-adapter batches, variants, quantized layers, tensor-parallel hooks, and specialized merge behavior. M6.4/M6.5 own those branches. The M6.3 invariant is narrower: the injected vanilla object must implement the same axis convention and scaling contract as its saved factors.

0x07L1 The Tiny Fixture Executes The Exact Operator

The Loop 8 runtime fixture uses two base matrices, two rank-one LoRA paths, and input

$$x=[1,-2,0.5].$$

The frozen paths are

$$W_qx=[-1.5,-3], \qquad W_vx=[3.5,0.75],$$

so the frozen sum is $[2,-2.25]$. With $r=1$, $\alpha=2$, and $s=2$:

$$sB_qA_qx=[0.44,-0.22], \qquad sB_vA_vx=[-0.625,-1.875].$$

Therefore

$$y_{\text{live}} =[2,-2.25]+[0.44,-0.22]+[-0.625,-1.875] =[1.815,-4.345].$$

After adding a second adapter called support and reselecting legal, both wrappers own legal and support A/B slots, but only the four legal factor tensors have requires_grad=True. This executes the slot distinction rather than inferring it from key names.

After injection, inspect the graph and optimizer surface rather than trusting the configuration:

expected = {name for name, _ in model.named_modules() if is_expected_target(name)}
actual = set(model.targeted_module_names)
assert actual == expected

trainable = {
    name for name, parameter in model.named_parameters()
    if parameter.requires_grad
}
assert trainable
assert all(is_adapter_parameter(name) for name in trainable)

Some configurations intentionally save or train additional modules, such as an output head. In that case the last assertion must admit the declared modules_to_save; it should not be weakened into accepting arbitrary trainable base tensors.

0x07M Adapter-Only State Dict And Reload

A normal model.state_dict() can include the complete transformed model. A portable PEFT artifact should select adapter-owned values and save the configuration needed to recreate their owners.

Official PEFT checkpoint documentation describes two central files for a LoRA adapter:

adapter_model.safetensors (or adapter_model.bin):
    adapter tensors and declared additional saved modules

adapter_config.json:
    PEFT type, rank/scaling/target metadata,
    base-model identity, and reconstruction options

Runtime parameter names may contain an adapter slot such as:

...q_proj.lora_A.legal.weight
...q_proj.lora_B.legal.weight

The pinned get_peft_model_state_dict first selects PEFT-owned values according to method and bias policy, includes declared auxiliary modules and selected embeddings when required, and then removes the local adapter name from adapter-owned keys. PeftModel.save_pretrained writes each selected non-default adapter under its own subdirectory, clones storage aliases that safetensors cannot encode, makes tensors contiguous, writes the tensor artifact, and writes configuration with inference-mode metadata.

The Loop 8 run makes the namespace transformation explicit:

Stage	Example key	Meaning
live graph	base_model.model.q_proj.lora_A.legal.weight	factor belongs to local slot legal
portable artifact	base_model.model.q_proj.lora_A.weight	local slot name removed
reloaded graph	base_model.model.q_proj.lora_A.renamed.weight	same portable factor inserted into requested local slot renamed

The portable directory contained exactly:

README.md
legal/adapter_config.json
legal/adapter_model.safetensors

The subdirectory is a consequence of saving a non-default selected adapter in this pinned version. It is not a universal storage layout for every PEFT release or hosting system.

The high-level reload path is:

base = load_exact_base(base_id, revision=base_revision)
model = PeftModel.from_pretrained(
    base,
    adapter_path,
    adapter_name="legal",
    is_trainable=False,
)

The lower-level injection API separates graph construction from weight population:

model = inject_adapter_in_model(config, base, adapter_name="legal")
outcome = set_peft_model_state_dict(model, adapter_state, adapter_name="legal")

assert not outcome.unexpected_keys
assert no_required_adapter_keys_missing(outcome.missing_keys)

PeftModel.from_pretrained loads configuration, constructs the wrapper and tuner graph, then delegates to load_adapter. The pinned set_peft_model_state_dict reinserts the requested adapter name, applies method-specific translations, optionally filters size mismatches, and calls model.load_state_dict(..., strict=False). A non-strict load result is evidence to inspect, not permission to ignore arbitrary missing keys.

This distinction matters for debugging. Successfully creating the low-rank modules does not prove their trained values were loaded. Conversely, loading a dictionary into shape-compatible factors does not prove the target modules, scaling convention, fan-in/fan-out orientation, or base revision are correct.

Treat the adapter as a typed artifact:

adapter ABI =
    base identity and revision
    model architecture
    target-module paths
    factor shapes and orientation
    rank/alpha/scaling convention
    dtype and quantization assumptions
    optional saved modules
    tensor values

The small tensor file is only one component of that ABI.

0x07M1 Reload And Merge Reconcile Different Representations

The real PEFT run loaded the legal artifact under local name renamed. The live and reloaded outputs agreed exactly in Torch fp32:

live output:     [ 1.8150000572, -4.3450002670 ]
reloaded output: [ 1.8150000572, -4.3450002670 ]
max |live - reloaded| = 0

For vanilla Linear, the pinned get_delta_weight computes

$$\Delta W_a = \operatorname{transpose}_{\text{fan-in/out}}(B_aA_a)s_a.$$

safe_merge=True clones the base weight, applies that delta, checks finiteness, and then records the merged adapter. merge_and_unload walks wrappers, merges admitted adapters, replaces each wrapper with its base layer, and removes peft_config from the returned graph. The fixture observed:

reloaded output: [ 1.8150000572, -4.3450002670 ]
merged output:   [ 1.8150001764, -4.3449997902 ]
max |reloaded - merged| = 4.76837158203125e-7
merged state keys containing "lora_" = 0

This is tolerance-level representation parity for one tiny fp32 torch.nn.Linear graph. It does not prove bitwise merge parity, quantized merge, tied-role safety, cross-release loading, or archival rollback. Loop 6’s tied-storage preflight and Loop 7’s immutable rollback lineage remain stronger admission requirements than a successful framework call.

0x07N What Loop 8 Actually Executed

source:
    PEFT v0.19.1 tag
    commit ba6a19060d6ab54a87538a6e77e3e4d5a907375b
    8 exact PEFT files, 20 AST symbol ranges

runtime:
    Python 3.11.15
    PEFT 0.19.1 from the exact checkout
    Torch 2.13.0, Transformers 5.14.1, safetensors 0.8.0
    two ordinary fp32 torch.nn.Linear targets
    two named adapter slots; one active/trainable
    selected save, renamed reload, safe merge and unload

not executed:
    a Transformers architecture
    quantized/custom/tensor-parallel backend dispatch
    distributed or sharded state
    mixed-adapter request batches
    training quality or convergence
    production serving or cross-release compatibility

The checked oracle verifies the exact Git revision and all eight file digests before parsing the 20 source symbols. The runtime imports PEFT from that checkout, not from an unrelated site package. Its report and 49 one-field invalid contracts are article-local evidence; they are not paper benchmarks.

0x07O PEFT Source-To-Runtime Failure Lab

Failure	Same-looking surface	First decisive check	Repair owner
wrapper helper and low-level injection conflated	both mutate the base graph	returned object type and prompt-method support differ	caller/API contract
second adapter overwrites first	same target path	inspect A/B dictionary keys and object identities	injection implementation
diagnostic target list counted as graph owners	repeated q_proj/v_proj strings	deduplicate module objects, then inspect slots	graph manifest
broad backend dispatcher selected	wrapper still has A/B names	record concrete wrapper class and dispatcher order	runtime integration
adapter saved under wrong namespace	tensor shapes still match	compare live, portable, and reload key maps	artifact ABI
non-strict load silently accepted	load call returns	classify missing/unexpected/mismatched keys against required owners	loader/admission
active adapter differs after reload	all factors resident	inspect active list and trainable parameter IDs	runtime control state
merged graph treated as resume checkpoint	output parity may pass	optimizer/factor coordinates and graph identity are gone	Loop 7
quantized or tied merge generalized from fp32 fixture	algebra looks identical	backend/storage-role admission was not executed	M6.4/M6.5

0x07P One End-To-End Oracle Must Preserve Every Intermediate Contract

The focused oracles above answer deliberately narrow questions. Loop 3 proves one linear operator in exact arithmetic. Loop 4 proves its local adjoints. Loop 6 proves graph injection, aliases, packed slices, and tied-storage admission. Loop 7 proves checkpoint and resume state. Loop 8 follows one real PEFT release through source, named slots, portable save, reload, and merge. None of those results alone proves that one unchanged adapter identity survives the complete sequence.

The end-to-end owner question is stricter:

Can one immutable base and target manifest pass through injection, live forward, backward, adapter-only save, renamed reload, dense merge, and rollback while preserving the correct identity and parity predicate at every boundary?

The checked-in Loop 10 fixture answers that question for one exact teaching graph. It has two independent targets, one true module alias, and one untargeted control:

input X: [T=2, d_in=3]
graph output: q_proj(X) + v_proj(X)

q_proj:
    d_out=2, rank=2, alpha=4, scale=2
    canonical path: model.layers.0.self_attn.q_proj
    alias path:     model.layers.0.attn_q_alias

v_proj:
    d_out=2, rank=1, alpha=3, scale=3
    canonical path: model.layers.0.self_attn.v_proj

k_proj:
    untargeted control; no adapter state

The ranks and scales differ intentionally. A loader that silently applies the query configuration to both targets can still create shape-valid tensors, but it will not implement this graph.

Transformer System executable teaching fixture. The diagram records six different parity domains. It is not a paper benchmark, production-framework trace, or finite-precision merge result.

0x07Q Snapshot, Match, Deduplicate, Then Rebind

The graph contains four path occurrences but only three module objects. The canonical query path and attn_q_alias refer to the same query object. Injection therefore follows a transaction, not an in-place name loop:

snapshot = list(named_module_occurrences(base))

matched = exact_match(
    snapshot,
    ["model.layers.0.self_attn.q_proj",
     "model.layers.0.self_attn.v_proj"],
)
assert no_unmatched_targets(matched)

unique_objects = deduplicate_by_module_object(matched)
assert len(unique_objects) == 2

wrappers = preflight_and_construct(unique_objects, adapter_slot="legal")
rebind_every_alias_path(snapshot, wrappers)

assert wrapper_at("q_proj") is wrapper_at("attn_q_alias")
assert not has_adapter("k_proj")

The exact plan reconciles as follows:

Graph count	Value	Why it is not another count
path occurrences	4	includes a query alias and the untargeted key control
requested paths	2	canonical query and value targets
unique target objects	2	query alias does not create a third object
wrappers constructed	2	one wrapper per target object
paths rebound	3	both query paths plus the value path

Matching by suffix would weaken the contract: a near-match added by a later architecture could become silently eligible. Wrapping every matched path would break alias identity and double the query adapter state. Rebinding only the canonical path would leave the alias executing the old base object. Loop 10 rejects all three substitutions before any output parity check.

0x07R Compose The Two Live Branches Before Comparing A Merge

Let the target set be $\mathcal{M}=\{q,v\}$. Each target owns its own frozen matrix, bias, rank, factors, and scale. The graph output is

$$Y = \sum_{m\in\mathcal{M}} \left[ XW_m^\top + \mathbf{1}_Tb_m^\top + s_m(XA_m^\top)B_m^\top \right].$$

This is not one adapter applied twice. The query target uses $r_q=2,s_q=2$; the value target uses $r_v=1,s_v=3$. With

$$X= \begin{bmatrix} 1&2&-1\\ 0&-1&3 \end{bmatrix}, \qquad G= \begin{bmatrix} 1&-2\\ 2&1 \end{bmatrix},$$

the exact intermediates are:

Target	Base output $XW_m^\top+b_m$	Low-rank coordinate $U_m=XA_m^\top$	Scaled branch $s_mU_mB_m^\top$	Live target output
query	[[0,-3],[7,4]]	[[2,1],[-3,2]]	[[4,2],[-6,4]]	[[4,-1],[1,8]]
value	[[4,5],[-6,1]]	[[-2],[4]]	[[-12,6],[24,-12]]	[[-8,11],[18,-11]]

The sum and scalar teaching loss are therefore

$$Y= \begin{bmatrix} -4&10\\ 19&-3 \end{bmatrix}, \qquad L=\langle G,Y\rangle=11.$$

The untargeted key projection is present in the graph manifest but not in this teaching output. That makes it a graph-mutation control, not a hidden third term in the loss.

0x07S Backward Parity Must Cover Every Factor And The Shared Input

For each target $m$, define $U_m=XA_m^\top$. Differentiating the composed graph gives

$$\frac{\partial L}{\partial B_m} = s_mG^\top U_m, \qquad \frac{\partial L}{\partial A_m} = s_m(GB_m)^\top X.$$

Both branches also contribute to the same input adjoint:

$$\frac{\partial L}{\partial X} = \sum_{m\in\mathcal{M}} \left( GW_m+s_mGB_mA_m \right).$$

The exact fixture derives

dA_q = [[  2,   0, 10],
        [ -4, -10, 10]]

dB_q = [[ -8, 10],
        [-14,  0]]

dA_v = [[12, 15, 15]]
dB_v = [[18],
        [24]]

dX   = [[13, -14,  5],
        [16,  -3, 10]]

An implementation can produce the right live output while still attaching one factor to the wrong target, transposing one saved factor, or dropping one branch from $dX$. The oracle therefore perturbs $A_q$, $B_q$, $A_v$, $B_v$, and $X$ independently. For direction $D$ it compares the manual contraction with the exact centered probe

$$\langle \nabla_ZL,D\rangle = \frac{L(Z+D)-L(Z-D)}{2}.$$

The five results are [-18, 6, 33, 66, 12], and all five agree exactly. This remains a local exact-integer derivative oracle. It does not reproduce optimizer dynamics, stochastic dropout, convergence, or task quality.

0x07T Portable Tensor Identity Is Not Local Namespace Identity

The training graph stores both targets under local slot legal. A portable adapter removes that local slot from tensor keys while retaining the target/rank/alpha/orientation manifest and immutable base digest:

local training key:
    base_model.model.layers.0.self_attn.q_proj.lora_A.legal.weight

portable key:
    base_model.model.layers.0.self_attn.q_proj.lora_A.weight

reloaded local key:
    base_model.model.layers.0.self_attn.q_proj.lora_A.renamed.weight

The artifact contains four factor tensors: A and B for query, then A and B for value. It does not contain base weights or exact-resume optimizer state. Its manifest includes:

base digest:  fd7709aa03cf...
target paths: q_proj, v_proj
ranks:        q=2, v=1
alphas:       q=4, v=3
orientation:  out_in

Consequently, three digests are expected to differ:

local legal state:    d219b4132191...
portable tensor state:f4f7e0faf8df...
local renamed state:  937d4341e628...

The differing local-state digests are not corruption; the key namespace changed by contract. The tensor payload and manifest must still reconstruct the same two target operators. The reloaded graph produces the exact same Y as the live training graph. A partial load, target swap, rank reuse, or non-strict missing key is a failed checkpoint even if every tensor that did load has a plausible shape.

0x07U Merge Creates A New Release; It Does Not Rewrite History

Only after live and reload parity pass does the oracle construct a separate dense release:

$$W_m^* = W_m+s_mB_mA_m, \qquad m\in\{q,v\}.$$

For the fixture,

$$W_q^*= \begin{bmatrix} 3&0&0\\ 0&1&3 \end{bmatrix}, \qquad W_v^*= \begin{bmatrix} 6&-4&5\\ -2&4&-3 \end{bmatrix}.$$

The merged graph reproduces each target output and the final sum exactly. Its release state has zero lora_ keys and digest 4211d4c1c912.... The immutable base artifact remains fd7709aa03cf... before and after the transaction.

That gives six explicit parity checkpoints:

Checkpoint	Compared identity	Exact result	What a pass does not prove
P0 manifest	base, target, alias, axis, slot, dtype	frozen	forward correctness
P1 live forward	injected graph versus explicit branch reference	equal	gradient correctness
P2 backward	four factor adjoints and shared input versus directional probes	5/5 equal	optimizer trajectory
P3 portable reload	legal save versus renamed graph	output equal	cross-release compatibility
P4 dense merge	reloaded live graph versus separate merged release	output equal; zero LoRA keys	quantized/tied merge safety
P5 rollback	post-transaction base versus immutable base digest	equal	arithmetic unmerge safety

Rollback is therefore an artifact operation:

def rollback(release_record):
    base = load_immutable_base(release_record.base_revision)
    adapter = load_portable_adapter(release_record.adapter_revision)
    return inject_after_full_admission(base, adapter)

It is not merged_weight -= cast(delta). Subtraction may not reconstruct a low-precision base bit for bit, and a merged graph no longer owns the optimizer, slot, or wrapper state needed for exact training resume.

The standard-library oracle rejects 59 one-field substitutions spanning axes, paths, aliases, factors, target-specific scales, gradient probes, artifact fields, namespace translation, merge representation, rollback, and claim scope. It does not replace the Loop 8 real-PEFT run: Loop 8 proves one pinned framework path, while Loop 10 proves the entire semantic transaction under exact arithmetic. Quantized and tied-storage merge admission remains M6.4/M6.5 work; mixed-request adapter scheduling remains M6.5; distributed state remains a separate deployment contract.

0x08 Why Low Rank Can Work, And Why It Can Fail

The LoRA paper reports empirical rank-deficiency in adaptation updates and finds that useful task adaptation can often be represented with ranks much smaller than the matrix width. Earlier intrinsic-dimension work also motivates the idea that successful optimization may live in a lower-dimensional subspace than the raw parameter count suggests.

The paper does more than report a rank ablation. It compares the leading singular subspaces learned with rank 8 and rank 64 for query and value updates in one GPT-3 layer. The strongest directions overlap substantially even though the larger adapter has many more available directions:

Source: LoRA v2, Figure 3. The heatmaps compare normalized subspace similarity for rank-8 and rank-64 query/value adaptation updates in the reported 48th GPT-3 layer. The result supports a low-dimensional-update hypothesis for that experiment; it does not establish one universal rank for every layer, model, task, seed, or target matrix.

This evidence should be read at the level of the effective update subspace, not as a claim that the raw factors are uniquely identifiable. The later gauge-non-uniqueness section shows why two pairs $(A,B)$ can represent the same dense update $BA$.

The careful claim is:

Many observed downstream adaptations admit effective low-dimensional update structure.

The claim is not:

Every downstream task, layer, projection, and training regime has the same intrinsic rank.

Several factors can increase the required adaptation surface:

large domain shift
new vocabulary or output interface
multimodal projector changes
long or diverse training corpus
continued pretraining rather than narrow task tuning
strict factual or behavioral retention constraints
few target modules
small base model with less reusable representation

Rank also interacts with target coverage. A small rank on many modules can outperform a larger rank on a narrow surface, or the reverse, depending on where the needed update lives.

0x08A Gauge Non-Uniqueness

The factorization is not unique. For any invertible:

$$R \in \mathbb{R}^{r\times r}$$

we have:

$$BA = (BR)(R^{-1}A)$$

The dense update is unchanged while the factor values differ.

Consequences:

factor norms are not a unique description of Delta W
optimizer dynamics depend on the chosen parameterization
regularizing A and B separately is not equivalent to regularizing BA
merging adapters should compare effective updates, not raw factors alone

This helps explain why variants that alter initialization, scaling, or per-factor learning rates can change training even though the final representational family still consists of low-rank updates.

0x09 Alternative Adaptation Mechanisms

LoRA is one PEFT design, not the definition of PEFT. The methods below all reduce some task-specific state, but they do not reduce the same state and do not leave the same executable model behind. A comparison that reports only a trainable-parameter percentage erases the distinctions that matter in training and deployment.

The owner question for this section is therefore:

If two methods train the same number of scalars, do they own the same backward path, inference graph, sequence state, cache identity, and release artifact?

The answer is no. We will derive that answer from the original paper contracts rather than rank methods by one benchmark table.

0x09A Four State Locations, Four Runtime Contracts

The book’s prefix- and prompt-tuning figures are useful because they ask where the learned state enters. The exact answer matters more than the shared label “parameter efficient.” The following reconstruction separates four mechanisms that are often drawn with the same colored blocks:

Reconstructed from the comparison idea in book Figures 9.5 and 9.6. The state locations and runtime ledgers are a corrected Transformer System interpretation grounded in the original Prefix-Tuning v1, Prompt Tuning v2, Houlsby adapter v2, and LoRA v2 papers. The book figures are conceptual references, not implementation specifications.

The first comparison axis is where task state enters:

Method family	Task-specific object	Entry point	Persistent inference consequence
full fine-tuning	selected dense base parameters	existing model weights	task owns a transformed base checkpoint
prompt tuning	learned embedding rows	input embedding sequence	virtual positions traverse the frozen stack
prefix tuning	layer-wise continuous activations	attention context at adapted layers	materialized prefix state extends attention context
bottleneck adapter	nonlinear down/up module	block residual path	serial per-token module remains live
IA3	multiplicative channel vectors	K, V, and FFN activations	live per-request scales or a single-task fold
BitFit	selected existing biases	named base-model parameters	no new operator, but task owns selected base values
LoRA	low-rank weight update	selected linear maps	live two-matmul branch or compatible dense merge

This location is not cosmetic. It determines which tensors require gradients, which dimensions belong in the artifact ABI, and whether serving changes sequence length, cache contents, or graph topology.

0x09B One Parameter Count Is Not A Fair Contract

For method $a$, separate at least six ledgers:

$$\mathcal{C}_a= \left( P_{train,a}, P_{deploy,a}, S_{opt,a}, G_{runtime,a}, Q_{sequence,a}, B_{backward,a} \right)$$

where:

• $P_{train}$ counts directly optimized parameters, including training-only reparameterizers;
• $P_{deploy}$ counts task-specific state needed by the chosen release representation;
• $S_{opt}$ is gradient and optimizer state, not merely model weights;
• $G_{runtime}$ records added, replaced, or folded operators;
• $Q_{sequence}$ records added input positions or layer-wise attention state;
• $B_{backward}$ records how far activation gradients must traverse.

Two methods can have equal $P_{train}$ and disagree on every other coordinate. Prefix-Tuning is a direct example: its training MLP can dominate $P_{train}$, but the paper discards that MLP after materializing the effective prefix. LoRA has the opposite deployment option: its trained factors can remain live or be represented as a dense merged delta. Full fine-tuning adds no operator at inference, yet it owns the largest task-specific change to the base checkpoint.

This is also why paper percentages are not automatically comparable. They may include or exclude a task head, task-specific layer normalization, a training-only reparameterizer, tied parameters, or frozen resident state. The denominator may be BERT, T5, GPT-2, or another architecture. This article uses paper-reported values only as claims about the paper’s own experiment and derives its teaching ledger from one explicit fixture later in this section.

0x09C Full Fine-Tuning: No New Graph, Full Task Weight Ownership

Let the pretrained model own unique parameter vector $\theta_0\in\mathbb{R}^{P_{base}}$. Full fine-tuning optimizes the selected dense coordinates directly:

$$\theta_{task}=\theta_0+\Delta\theta_{task}$$

If every base coordinate is trainable, then:

$$P_{train,FT}=P_{base}$$

For an Adam-like optimizer, every trainable dense coordinate can also own a gradient, first moment, and second moment. The important comparison is not a fixed bytes-per-parameter slogan, because weight, gradient, master-weight, and optimizer dtypes are implementation choices. It is the ownership relation:

full fine-tuning task state:
    dense task weights
    + dense gradients while live
    + optimizer state for selected dense parameters
    + scheduler / scaler / RNG / data-progress state for exact resume

Inference does not gain a new adapter branch. The task-specific function is already represented by ordinary model weights. The operational price is that the task owns a full transformed checkpoint, or a dense delta plus an exact reconstruction contract. Serving many tasks therefore means loading, swapping, or sharing at a much coarser boundary than a small adapter artifact.

Full fine-tuning is the unrestricted update baseline, not the automatically strongest or fairest experimental baseline. Its result depends on optimizer, learning rate, regularization, data size, and checkpoint selection just as PEFT does.

0x09D Prompt Tuning: Input Positions Become Trainable State

The Prompt Tuning paper studies frozen T5 encoder-decoder models. It learns $m$ input-side embeddings of width $d$:

Let $m$ be the learned prompt length and $d$ the model width. Prompt tuning learns:

$$P_{prompt}\in\mathbb{R}^{m\times d}$$

and concatenates it once, at the embedding boundary:

$$H_0'=[P_{prompt};H_0] \in\mathbb{R}^{(m+T)\times d}$$

The frozen Transformer then processes those $m$ positions normally. In a decoder-only model they acquire ordinary hidden states and per-layer K/V rows just like other preceding positions. In the encoder-decoder setting studied by the Prompt Tuning paper, the learned prompt is prepended to the encoder input. The architectural placement therefore determines which attention path grows; “adds $m$ decoder-cache rows” is not a model-independent statement.

# One input-side insertion; P_prompt is the only trainable tensor.
h = concat([P_prompt, token_embedding(input_ids)], dim="sequence")
for block in frozen_blocks:
    h = block(h)

Parameter state is $md$. Runtime sequence work grows because the prompt positions traverse the frozen stack. They are not vocabulary IDs and need not be nearest to any readable word embedding.

For a causal decoder teaching profile with $T$ real positions, the token-only causal attention matrix contains:

$$N_{score,base}=\sum_{t=1}^{T}t=\frac{T(T+1)}{2}$$

When $m$ virtual prompt positions become ordinary preceding positions, the stack computes $(m+T)$ query rows:

$$N_{score,prompt}=\frac{(m+T)(m+T+1)}{2}$$

Subtracting the baseline gives:

$$\Delta N_{score,prompt} =mT+\frac{m(m+1)}{2}$$

The first term is real-token attention to prompt keys. The second is the prompt positions’ own causal work. More importantly, those positions also traverse every projection, normalization, and FFN in the frozen stack. Counting only $md$ task parameters therefore misses the dominant runtime path.

Training is similarly not shallow. Although only $P_{prompt}$ receives parameter gradients, the loss gradient must propagate from the output through the frozen stack to the embedding-boundary prompt leaf. Frozen base weights avoid dense parameter-gradient and optimizer ownership; they do not erase activation-gradient propagation.

The paper also shows that prompt length, initialization, model scale, and the frozen model’s pretraining objective interact. Its default T5 experiment used class-label-informed initialization and a longer prompt, while large models were more robust to initialization and length. These are source-reported experimental findings, not a universal rule that a particular prompt length transfers to a causal decoder.

0x09E Prefix-Tuning: Layer-Wise Attention State And A Training Reparameterizer

The original Prefix-Tuning paper defines each prefix position’s state as a concatenation of activations across Transformer layers. For a decoder-serving implementation that materializes per-layer attention state, the effective K/V view is:

$$K_{prefix}^{(\ell)},V_{prefix}^{(\ell)} \in\mathbb{R}^{m\times H_{kv}\times d_h}$$

and attention consumes:

$$\widetilde K^{(\ell)}=[K_{prefix}^{(\ell)};K_{tokens}^{(\ell)}], \qquad \widetilde V^{(\ell)}=[V_{prefix}^{(\ell)};V_{tokens}^{(\ell)}]$$

This K/V notation is a decoder-serving view of the paper’s more general layer-activation formulation. For BART, the paper introduces prefixes in the relevant encoder and decoder stacks; exact tensors therefore depend on architecture and implementation.

Directly optimizing the full prefix activations was unstable in the paper’s experiments. Training instead used a smaller learned table and an MLP reparameterization:

$$P_{effective}=f_{\phi}(P_{small})$$

The reparameterization network is training state, not required serving state: after training, the generated effective prefix can be stored and the MLP discarded. At inference, each adapted layer attends over $m$ additional prefix positions, but those positions are not ordinary input embeddings propagated through every lower block.

# Effective prefix state may be materialized once per task/request.
prefix_state = prefix_mlp(prefix_table)  # training path
for layer, block in enumerate(frozen_blocks):
    k, v = block.project_kv(h)
    k = concat([prefix_state[layer].k, k], dim="sequence")
    v = concat([prefix_state[layer].v, v], dim="sequence")
    h = block.attend_and_mlp(h, k, v)

Let the materialized prefix width per position be:

$$d_{prefix}=2L H_{kv}d_h$$

for a decoder-only K/V teaching representation. Then direct deployed prefix state is:

$$P_{deploy,prefix}=m d_{prefix}=2LmH_{kv}d_h$$

Suppose training uses a table $P'\in\mathbb{R}^{m\times k}$ and a two-layer MLP with hidden width $h_p$ and output width $d_{prefix}$. Including biases:

$$P_{train,prefix} =mk+(kh_p+h_p)+(h_p d_{prefix}+d_{prefix})$$

The final two parenthesized terms can be much larger than the deployed prefix. That is not a contradiction: the MLP is an optimization coordinate system, while the materialized prefix is the release representation.

For $T$ causal token queries, a direct K/V prefix adds no prefix-query rows. It adds $m$ keys to each token query:

$$N_{score,prefix} =\frac{T(T+1)}{2}+mT, \qquad \Delta N_{score,prefix}=mT$$

This differs from prompt tuning by exactly the prompt positions’ own triangular work in this decoder profile:

$$\Delta N_{score,prompt}-\Delta N_{score,prefix} =\frac{m(m+1)}{2}$$

That equation still understates the difference because prompt positions traverse full blocks, whereas a materialized K/V prefix enters at each attention layer. BART introduces prefixes in architecture-specific encoder and decoder paths, so a single decoder-only K/V count must not be copied into every Prefix-Tuning implementation.

0x09F Bottleneck Adapters: Nonlinear Capacity And Placement Variants

The Houlsby design inserts serial adapters after both the attention projection and feed-forward sublayer in each Transformer block. With bottleneck width $r_a$:

$$Adapter(h)=h+W_{up}\,\sigma(W_{down}h)$$

where:

$$W_{down}\in\mathbb{R}^{r_a\times d}, \qquad W_{up}\in\mathbb{R}^{d\times r_a}$$

Including down- and up-projection biases, one insertion owns:

$$P_{adapter,insert}=dr_a+r_a+r_a d+d =2dr_a+r_a+d$$

With $n_a$ insertions per block and $L$ blocks:

$$P_{adapter,module}=Ln_a(2dr_a+r_a+d)$$

The Houlsby paper uses $n_a=2$ and also learns new task-specific layer-normalization parameters and an output layer. Calling only $2dr_a$ “the Houlsby task state” therefore omits paper-owned state. For $T$ rows, the two dense adapter projections add approximately:

$$MAC_{adapter}=Ln_aT(2dr_a)$$

before bias, activation, residual, and launch costs. There is no sequence-length or direct attention-context growth, but the nonlinear down-activation-up path remains serial. In general it cannot be folded into one existing dense weight because:

$$W_{up}\sigma(W_{down}h)$$

is not a fixed linear map when $\sigma$ is nonlinear.

Placement is part of the method. Houlsby’s BERT graph places two adapters after sublayer projections and before the outer skip addition and layer normalization. A pre-norm decoder has a different residual schedule; copying the name “Houlsby adapter” without freezing the insertion edge is not an implementation specification. Later adapter frameworks support different placements and compositions, so benchmark rows labeled only “adapter” can describe different graphs.

Near-identity initialization is also a training contract. The adapter’s internal residual lets small projection outputs begin near the frozen function. The Houlsby paper reports instability when initialization departs too far from that regime. LoRA’s zero-$B$ initialization and adapter near-identity initialization share a purpose, but their first-gradient algebra and runtime graphs are not the same.

0x09G IA3 And BitFit: Multiplicative Versus Selective Existing State

IA3 modifies activations rather than adding sequence positions or a nonlinear bottleneck. The T-Few paper introduces one learned vector for attention keys, one for values, and one for the FFN intermediate activation in each Transformer block:

$$\operatorname{Attn}_{IA^3}(Q,K,V) =\operatorname{softmax}\!\left( \frac{Q(\ell_k\odot K)^T}{\sqrt{d_k}} \right)(\ell_v\odot V)$$ $$FFN_{IA^3}(x) =W_2\left(\ell_{ff}\odot\gamma(W_1x)\right)$$

where:

$$\ell_k\in\mathbb{R}^{d_k}, \qquad \ell_v\in\mathbb{R}^{d_v}, \qquad \ell_{ff}\in\mathbb{R}^{d_{ff}}$$

For an $L$-block encoder, the paper’s direct count is:

$$P_{IA^3,encoder}=L(d_k+d_v+d_{ff})$$

For its encoder-decoder decoder, both self-attention and cross-attention receive key and value scales:

$$P_{IA^3,decoder}=L(2d_k+2d_v+d_{ff})$$

The vectors initialize to ones, making the inserted operation the identity. They can vary by request in a mixed-task batch because each row can select task-specific scales. For a single fixed task, the paper observes that the scales can be absorbed into adjacent matrices. In column notation, for example:

$$\ell\odot(Wx)=\operatorname{Diag}(\ell)Wx$$

so the scaled output channels can be represented by $W'=\operatorname{Diag}(\ell)W$. The FFN intermediate scale can similarly be absorbed into the input-channel dimension of $W_2$. Folding is therefore a representation choice with the same base-identity and dtype caveats as any release-time transform; it is not a free mixed-task serving solution.

The primary paper’s Figure 1 pins the exact K, V, and FFN insertion points. Its right half shows the broader T-Few loss recipe; those losses are not part of the IA3 operator contract discussed here.

Source: Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning, Figure 1, released with the paper under its linked Creative Commons license. The left panel verifies IA3 placement; the right panel is retained to avoid presenting a cropped paper figure as the original composition. T-Few’s additional losses are out of scope for this operator comparison.

BitFit takes another route: it changes selected bias vectors already present in the base graph. The original paper studies BERT-family masked-language-model encoders and trains the task classifier plus all biases or a selected bias subset. Its basic operator remains:

$$y=Wx+b_{task}$$

No new matrix multiplication or sequence state is required. The task artifact instead depends on exact base parameter names and shapes. The paper reports that query and middle-MLP biases formed a useful reduced subset in its BERT experiments. That is a source-reported architecture-specific result, not a command to tune equivalently named tensors in every decoder. Some modern linear layers omit biases, and normalization families differ, so BitFit’s parameter budget must be counted from the actual graph.

0x09H LoRA Revisited: A Weight-Space Update With Two Release Forms

LoRA attaches trainable factors to selected linear maps:

$$W_0x \longrightarrow W_0x+sBAx$$

It does not increase sequence length and does not create a new layer-wise prefix state. Targeting a K/V projection changes the values and compatibility of the resulting cache, not its row count or tensor shape. A live adapter adds two linear operations at each targeted map; a compatible deployment can instead merge $sBA$ into $W_0$. That merge path is the decisive difference from a nonlinear bottleneck adapter.

This does not make LoRA uniformly cheaper. A live mixed-adapter batch needs an adapter-aware kernel or segmentation by adapter identity. A merged adapter removes the branch but gives up cheap per-request selection unless separate merged model instances are resident. The detailed scheduler and residency consequences belong to M6.5; this article owns only the operator and artifact distinction.

0x09I One Decoder Fixture Separates State From Work

To compare the contracts without mixing paper architectures, freeze this Transformer System teaching model:

decoder blocks L = 2
model width d = 4
SwiGLU width d_ff = 8
query heads H_q = 2
KV heads H_kv = 2
head width d_h = 2
vocabulary V = 16
real input positions T = 3
prompt / prefix length m = 2
adapter bottleneck r_a = 2
LoRA rank r = 1, targeting q_proj and v_proj
prefix reparameterizer k = 2, hidden width h_p = 6
two affine LayerNorms per block plus one final affine LayerNorm
token embedding and output projection share one storage

Every attention and SwiGLU projection includes a bias. The base unique-parameter count is:

$$\begin{aligned} P_{base} ={}&Vd\\ &+L\left[ 4(d^2+d) +2(dd_{ff}+d_{ff}) +(d_{ff}d+d) +4d \right]\\ &+2d \end{aligned}$$

The four terms inside the block are Q/K/V/O, SwiGLU gate and up projections, the down projection, and two LayerNorm gamma/beta pairs. Substitution gives:

$$P_{base}=64+2(80+80+36+16)+8=496$$

The resulting state ledger is:

Method representation	Directly trainable task scalars	Deployed task scalars	What the count includes
full fine-tuning	$496$	$496$	all unique base parameters; tied output storage counted once
prompt tuning	$md=8$	$8$	learned input embeddings
prefix tuning	$134$	$32$	$4$ table + $130$ MLP during training; materialized $2LmH_{kv}d_h$ K/V state at release
Houlsby-style adapter	$120$	$120$	$88$ adapter parameters + $32$ task-specific LayerNorm parameters; task head excluded
IA3, decoder-only self-attention profile	$32$	$32$ live or folded	$L(d_k+d_v+d_{ff})$ with $d_k=d_v=H_{kv}d_h=4$
BitFit, all fixture biases	$92$	$92$	projection and LayerNorm biases in this graph; no task head
LoRA on Q and V	$32$	$32$ live, or zero separate scalars after dense merge	$L\times2$ targets, each $r(d+d)$

The table does not claim equal accuracy, equal expressive power, or equal paper settings. It demonstrates that task-scalar count itself is representation-dependent. Prefix training state and prefix release state differ. Merged LoRA has no separate factor artifact but changes the dense base artifact. Full fine-tuning and BitFit modify existing state rather than append modules.

Now compare recurring work for the same $T=3$ input:

Method	Added recurring work in the fixture	Sequence / K/V consequence
full fine-tuning	no new inference operator	same shape, task-specific values
prompt tuning	full frozen-block work for $m=2$ extra positions; $9$ extra causal score pairs per layer	two ordinary preceding positions and their per-layer K/V rows
prefix tuning	$mT=6$ extra score pairs per layer plus value aggregation	two direct K/V prefix rows per layer; no prefix query rows
Houlsby adapter	$192$ extra projection MACs before activation/bias/launch costs	no extra sequence rows
IA3	$96$ live elementwise multiplies, or no added operator after a valid single-task fold	no extra rows; K/V values and FFN activations change
BitFit	no new operator relative to the biased base graph	no extra rows; existing projected values change
LoRA Q/V	$96$ live branch MACs, or no added operator after merge	no extra rows; cached K/V values depend on adapter identity

For prompt tuning, $\Delta N_{score}=2\cdot3+2\cdot3/2=9$. For direct prefix K/V, $\Delta N_{score}=2\cdot3=6$. For live LoRA, four targeted maps each execute $T[r d+d r]=24$ MACs, giving $96$. For two adapter insertions in each of two blocks, $Ln_aT(2dr_a)=2\cdot2\cdot3\cdot16=192$ MACs. These are logical operator counts, not latency predictions.

Transformer System teaching ledger. Counts are replayed by the Loop 9 oracle for the frozen decoder fixture; the visual does not compare method quality.

0x09J Training And Backward Paths Are Not Ranked By Trainable Scalars

For a layer whose output $h_{\ell+1}=F_{\ell}(h_{\ell};\theta_{\ell})$ feeds the loss, backward still propagates:

$$g_{\ell}=g_{\ell+1} \frac{\partial F_{\ell}}{\partial h_{\ell}}$$

even when $\theta_{\ell}$ is frozen and $\partial\mathcal{L}/\partial\theta_{\ell}$ is not stored. The closest trainable leaf to the model input determines how much of that activation-gradient chain remains necessary:

Method	Base weight gradients	Activation-gradient path	Distinct training consequence
full fine-tuning	selected dense base weights	full selected graph	dense gradient and optimizer state
prompt tuning	none for frozen base	output to input prompt across the full stack	prompt is an input-boundary leaf
prefix tuning	none for frozen base	from outputs to layer-wise prefix insertions	reparameterizer also receives prefix gradients
adapter	none for frozen base	through downstream blocks to every inserted module	nonlinear adapter activations are live for backward
IA3	none for frozen base	through downstream blocks to scale vectors	elementwise scales still depend on backbone activations
BitFit	selected bias gradients	through graph to each selected bias	existing parameter subset owns optimizer state
LoRA	factor gradients only	through downstream blocks and local low-rank branches	base weight-gradient buffers are omitted, not activation gradients

This is the distinction A22’s resource section and retained Ladder Side-Tuning figure introduce earlier in 0x07. LST changes the graph itself by training a separate side network so backbone backward can be avoided. Ordinary in-backbone PEFT should not inherit that property merely because it updates few scalars.

Training recipes also differ beyond the operator:

• Prompt Tuning’s paper studies prompt initialization, prompt length, optimizer settings, T5 scale, and LM adaptation of the frozen model.
• Prefix-Tuning uses an MLP reparameterizer because direct prefix optimization was unstable in its experiments, then discards the MLP after training.
• Houlsby adapters rely on near-identity initialization and train task-specific normalization and output state in addition to bottleneck modules.
• IA3 initializes scales to one and T-Few combines the method with pretraining of IA3 state and additional task losses; those recipe components are not part of the bare IA3 operator.
• BitFit trains a named bias subset and task head in BERT-family models.
• LoRA initializes one factor so the initial dense update is zero and exposes rank, scaling, target, dropout, and merge choices derived earlier in this article.

An empirical comparison is valid only after these recipe differences are declared.

0x09K Artifact, Cache, And Serving Identity

Each method needs a different release manifest:

Method	Minimum task-artifact identity	Cache / request identity consequence
full fine-tuning	exact transformed checkpoint or dense delta, base revision, tokenizer/config	all generated state belongs to that task model revision
prompt tuning	prompt tensor, $m$, $d$, input side, insertion order, base revision	prompt-conditioned hidden/K/V rows belong to prompt identity
prefix tuning	materialized vs reparameterized format, layer/stage map, $m$, K/V head geometry, dtype, base revision	each layer’s prefix state and dependent token K/V belong to prefix identity
bottleneck adapter	insertion edges, pre/post-norm schedule, width, activation, task norms/head, base revision	downstream hidden state and later-layer K/V depend on adapter identity
IA3	K/V/FFN scale targets, channel axes, live vs folded representation, base revision	scaled K/V and dependent hidden state belong to IA3 identity
BitFit	exact bias parameter paths, shapes, values, task head, base revision	projected and downstream state belongs to BitFit identity
LoRA	factor orientation, rank, scale, target paths/slices, live vs merged state, base revision	K/V-targeted adapters directly alter cached rows; all targets alter model identity

A cached prefix cannot be keyed only by token IDs once task-specific state changes the model function. Even when an adapter does not change K/V shape, it can change K/V values directly or through earlier hidden states. Reusing cache across mismatched prompt, prefix, adapter, IA3, BitFit, LoRA, or full-model identities silently conditions future decoding on another function.

Mixed-task batching also follows the state location:

for request in batch:
    contract = resolve_adaptation_contract(request.task_id)
    validate_base_and_cache_identity(contract)

    if contract.kind == "prompt":
        prepend_virtual_embeddings(request, contract.prompt)
    elif contract.kind == "prefix":
        bind_layer_prefix_state(request, contract.materialized_prefix)
    elif contract.kind in {"adapter", "lora", "ia3"}:
        bind_request_specific_module_or_scale_state(request, contract)
    elif contract.kind in {"bitfit", "full_ft"}:
        route_to_compatible_weight_instance(request, contract.model_revision)

This pseudocode states identity, not a universal kernel. Prompt lengths may require padding or grouping; prefixes require layer-state plumbing; live LoRA, adapters, and IA3 require request-aware operators or batch segmentation; folded or dense-updated methods generally select a compatible weight instance. M6.5 owns the scheduler, residency, and fragmentation analysis.

0x09L Paper-By-Contract Comparison

The source papers answer different experimental questions on different architectures:

Primary paper	Frozen model and task context	Directly trained state	Paper-specific detail that must survive retelling	Claim that must not be generalized
Houlsby adapters v2	BERT transfer over classification and QA tasks	two bottleneck adapters per layer, task LayerNorms, output layer	post-sublayer placement and near-identity initialization	one adapter parameter percentage is not every adapter architecture
Prefix-Tuning v1	GPT-2 table-to-text and BART summarization	small prefix table plus MLP reparameterizer	all-layer prefix activations; discard reparameterizer after materialization	decoder-only K/V layout is not the only prefix layout
Prompt Tuning v2	frozen T5 encoder-decoder models	input prompt embeddings	prompt length, initialization, scale, and frozen-model pretraining interact	T5 source-reported quality does not establish causal-decoder cost or quality
LoRA v2	RoBERTa, DeBERTa, GPT-2, and GPT-3 experiments	low-rank factors on selected weights	mergeable linear delta and target/rank choices	no-latency claim assumes a valid merge representation
BitFit v2	BERT-family masked-LM encoders	biases or selected biases plus task classifier	existing-state selection is architecture named	query/MLP bias subset is not universal across decoder graphs
IA3 / T-Few v2	T0/T5 encoder-decoder few-shot tasks	K, V, and FFN scaling vectors	separate self/cross-attention counts, one initialization, mixed-task scales, optional fold	T-Few losses and IA3 operator are not the same contract
LST v2	T5 and CLIP-T5 transfer	separate ladder side network	changes the backward topology to avoid backbone backpropagation	ordinary LoRA/adapter/prompt training does not inherit LST memory behavior

Benchmark claims stay attached to those rows. This article does not use one paper’s score to rank methods under another paper’s model, data, objective, or parameter budget.

0x09M Alternatives Failure Lab

Failure	Broken contract	Observable consequence
equal trainable scalars treated as equal cost	training, runtime, and sequence ledgers collapsed	method choice reverses under latency or memory measurement
prompt and prefix used as synonyms	input embeddings confused with all-layer state	wrong parameter count, mask, cache shape, and artifact layout
prefix reparameterizer shipped as mandatory inference graph	training and release representations conflated	avoidable serving state and compute
direct prefix K/V counted as ordinary prompt tokens	prefix query rows invented	attention work is overcounted and masks are wrong
Houlsby count includes only $2dr_a$ once per block	biases, second insertion, task norms/head omitted	task artifact and optimizer state are understated
adapter graph copied from post-norm BERT into a pre-norm decoder by name	insertion edge omitted	a different function is trained
IA3 called three vectors for every architecture	encoder, decoder, and cross-attention structure ignored	target channels and parameter count are wrong
IA3 fold used while retaining per-request mixed-task selection	representation and batching contracts conflict	requests need incompatible weights
BitFit budget inferred without graph enumeration	absent or extra biases ignored	checkpoint paths fail or parameter count drifts
LoRA called latency-free while kept live	merge precondition omitted	extra branch work and adapter-aware batching remain
frozen weights interpreted as no backbone backward	activation-gradient path omitted	training memory and time are understated
cache keyed only by tokens	adaptation identity omitted	later decode uses state produced by a different function

Choosing an adaptation method now becomes a contract decision: what transformation family is needed, what training path is affordable, what task artifact can be released, what request identity the runtime can select, and what cache state remains valid. Popularity and parameter percentage are insufficient selectors.

0x0A Failure Lab

Failure	Broken assumption	Observable symptom
LoRA called model compression	base residency omitted	deployment memory is underestimated
$W$ assumed low rank because $\Delta W$ is factorized	wrong object analyzed	misleading theory and target selection
framework linear orientation ignored	row/column convention mismatch	shape error or transposed update
$BA$ materialized every training step	low-rank execution path discarded	unnecessary dense compute and memory
zero-$B$ initialization described as symmetric learning	first-step gradient path ignored	incorrect optimizer analysis
rank reported without alpha/scaling	training contract incomplete	irreproducible rank comparison
declared rank treated as actual $\operatorname{rank}(BA)$	factor directions can be dependent or zero	capacity is overstated and padded-rank experiments are misread
alpha held fixed but rank study called scale-controlled	$s=\alpha/r$ changes with rank	branch and gradient magnitude changes are attributed only to capacity
dropout expectation treated as one deterministic forward	mean preservation confused with realization equality	stochastic variance and mask state disappear from the explanation
equal $BA$ treated as equal optimization state	factor-coordinate gauge ignored	equivalent functions follow different factor-gradient trajectories
one dtype label used without cast boundaries	storage, accumulation, scale/add, and output precision collapsed	live/merged parity and underflow failures cannot be reproduced
rank reported without target modules	adaptation budget incomplete	parameter and quality comparisons do not transfer
target path counted as module identity	aliases are wrapped independently	duplicate adapter state and broken shared behavior
substring match used as an undeclared target rule	name near-match is treated as semantic identity	unintended modules become trainable
equal packed chunks assumed for unequal GQA slices	storage layout is inferred from names	Q/K/V deltas land on wrong rows
only one path in an object-alias group is rebound	parent slots no longer resolve to one module	call sites execute different wrappers
tied head delta merged into an untargeted embedding storage	role identity is collapsed into storage identity	token embeddings change during head-only adaptation
replacement begins before graph preflight closes	mutation and validation are interleaved	failed injection leaves a partially rewritten model
base weights removed from memory ledger	frozen confused with absent	training OOM despite small optimizer state
adapter state dict lacks base revision	semantic compatibility unchecked	adapter loads but produces invalid behavior
portable adapter mislabeled exact-resume checkpoint	optimizer and control state omitted	resumed trajectory diverges immediately
scheduler advances after an AMP-skipped optimizer step	step counters use different events	learning-rate schedule drifts from parameter updates
RNG or sampler restored after the next draw	save values are correct but restore order is wrong	next mask or sample differs
path-local optimizer slots created for one aliased parameter	path occurrence confused with parameter identity	duplicate or conflicting moments
merged release relabeled as canonical base	derivation lineage discarded	rollback and later adapter compatibility become ambiguous
base K/V reused after K/V projections change	target-module state ignored	generation conditions on incompatible cache

0x0A1 A Failure Record Needs Five Coordinates

A three-column prose table is useful for recognition, but it is not yet an operational recovery contract. A replayable failure record needs five coordinates:

failure id and category:
    stable identity for the alarm class

invariant:
    the exact semantic statement that must remain true

detector:
    executable assertion, digest, tensor oracle, or lifecycle replay

consequence:
    the first observable state or value that becomes invalid

recovery owner:
    the article or release gate allowed to repair the contract

The owner field prevents a common integration error. M6.5 may change how live adapters are scheduled, but it may not silently redefine M6.3 factor orientation, path matching, checkpoint ABI, or merge semantics. M8 may define quantized deployment representations, but it may not promote a quantized merge to an exact vanilla-LoRA identity without a representation-specific proof.

0x0A2 Same Shape Does Not Mean Same Semantics

The most dangerous substitutions in this article preserve tensor shapes:

Shape-compatible substitution	Detector	First invalid state	Recovery owner
transpose $A$ and $B$ conventions consistently in metadata but not execution	exact row/column forward oracle	live output	M6.3
normalize a selected target with another target’s scale	target-local scale assertion	branch output and all local adjoints	M6.3
wrap both paths of one aliased module independently	object-identity and unique-parameter ledger	adapter state and optimizer ownership	M6.3
split a packed GQA projection into equal thirds	explicit row-slice manifest	Q/K/V row ownership	M6.3
load portable factor tensors into a near-matching base revision	base/config/target digest admission	model and cache identity	M6.3, then M6.5 at admission
commit a merged dense artifact as if it were the immutable base	lineage and representation-state assertion	rollback and later compatibility	release gate
compare prompt, prefix, adapter, IA3, BitFit, and LoRA by scalar count alone	seven-method paper-contract oracle	method-selection claim	release gate

Output equality at one point is also insufficient. The Loop 10 transaction therefore checks object identity, live values, factor and input derivatives, portable namespace identity, merged representation identity, and rollback lineage at six different checkpoints. A later checkpoint cannot stand in for an earlier one.

0x0A3 Failures Belong To Different Alarms

The checked failure manifest groups 30 records from Loops 2-10 into five categories. Each category has a different first detector:

Category	Representative invariants	First detector	Typical recovery
identity and state	path, object, parameter, storage, adapter slot, representation state	immutable manifest and digest reconciliation	rebuild the M6.3 injection or artifact plan
algebra and gradient	orientation, rank, scale, dropout, cast schedule, local adjoints	exact tensor and directional-derivative oracle	repair the M6.3 operator or derivation
graph and target	exact path, alias group, packed slices, tied storage, transactional rebind	snapshot-plan-commit graph oracle	abort mutation and regenerate a complete plan
lifecycle and runtime	optimizer/RNG order, portable namespace, cache identity, merged lineage	interrupted resume, renamed reload, and merge parity	restore canonical artifacts; M6.5 owns request/runtime recovery
evidence and scope	paper revision, code revision, toy/runtime distinction, source-reported result	source, figure, URL, and claim-firewall audit	release gate blocks publication or narrows the claim

Transformer System static closure map. It records failure ownership and publication inventory; it is not browser-layout or production-runtime evidence.

0x0A4 Complete Source, Link, Visual, And Negative-Test Surface

Loop 11 closes the static publication surface rather than sampling it:

source records:
    3 local + 11 revision-pinned primary + 18 official/project = 32

article external URLs:
    29, all covered by the source manifest

local owner routes:
    5 normalized routes, all resolved in src/content/posts

A22 figures:
    53 source figures, each retained, replaced, or deferred exactly once

article image routes:
    33, all present, attributed, digest-pinned, and nonempty in alternative text

negative executable surface:
    9 invalid-fixture families
    566 one-field mutation cases across Loops 2-10

typed recovery surface:
    30 records covering every mechanism loop from 2 through 10

The 32 source records do not prove that every external endpoint will remain available. The 33 static image routes do not prove that text is legible in a browser. SVG roles, titles, descriptions, view boxes, file existence, digests, and alternatives can be checked without rendering; MathJax layout, responsive selection, horizontal overflow, theme contrast, console errors, and local-route acceptance require a browser. Loop 12 owns normalized desktop/mobile and light/dark browser evidence; the completed report below closes that narrower publication claim.

0x0A5 Final Formula, Rendering, Accessibility, And Boundary Closure

The final audit reconciles the source article, generated route, and browser as three different surfaces:

Surface	Exact closure	What it does not prove
source	189 source-ordered display formulas, 38 tables, 98 fenced code blocks, 33 image routes, and five normalized local post routes	formal correctness of every derivation or future availability of external URLs
generated route	every source display formula becomes one display MathJax source; all inline and display sources render without a MathJax error	compatibility with every renderer or assistive technology
responsive images	11 picture elements select their mobile source below the declared breakpoint and their desktop source above it; the remaining 11 images remain single-source	perceptual quality under every zoom level, font override, or color-vision condition
four browser profiles	desktop/mobile and light/dark profiles load every image and local route, preserve heading targets, contain article-owned overflow, and produce no article console exception	production adapter execution, GPU behavior, quality, latency, or live third-party endpoint health
boundaries	concrete M3.4, M5.1, M6.4, M6.5, M6.7, M6.9, and M8.1 owners resolve without long exact prose duplication	that those neighboring articles are already complete

Browser acceptance does not promote a teaching oracle into framework or production evidence. It proves that this built M6.3 snapshot exposes the declared formulas, links, tables, alternatives, and responsive figures without article-local publication defects under the four normalized profiles. The source manifest still owns claim provenance, the executable fixtures still own their narrow tensor and lifecycle contracts, and later articles still own variant and serving behavior.

0x0B Boundary To M6.4 And M6.5

M6.3 ends with the vanilla mechanism fully specified.

M6.4 asks how major variants alter that mechanism:

QLoRA:
    base storage and training-memory path

AdaLoRA:
    rank budget allocation over modules and time

rsLoRA:
    rank-dependent scaling

LoRA+:
    separate learning rates for A and B

PiSSA and LoftQ:
    initialization around pretrained or quantized weights

DoRA:
    magnitude/direction parameterization

M6.5 then treats the adapter as a serving object:

loaded or evicted
merged or live
selected per request
executed in heterogeneous batches
co-resident with request KV
observed under latency and memory SLOs

0x0C Publish Gate

Check	Status
Fine-tuning, PEFT families, rank, factorization, SVD, and target modules are established as prerequisites	pass
Dense update and low-rank update parameter counts are derived	pass
Rank is explained geometrically and algebraically	pass
SVD motivation is separated from direct LoRA training	pass
Forward tensors are traced with explicit shapes	pass
Backward gradients for $A$, $B$, and $X$ are derived	pass
Zero-$B$ first-step asymmetry is explained	pass
Rank, alpha, dropout, precision, and factor conditioning are separated into controlled contracts	pass
Target modules are compared by behavioral and state consequence	pass
Path, object, packed-slice, parameter, and storage identities are separated	pass
Unequal packed Q/K/V adaptation is derived and K-slice zero-delta is executed exactly	pass
Object aliases share one wrapper and tied-storage merge admission is role-aware	pass
Injection is split into preflight planning and graph commit	pass
A realistic multi-layer parameter ledger is included	pass
Training and optimizer state are separated from frozen base residency	pass
Persistent adapter, gradient, optimizer-moment, master-copy, activation, and runtime ledgers are not collapsed	pass
AdamW factor-coordinate moments and bias-correction state are derived	pass
Step-boundary and mid-accumulation checkpoint contracts are separated	pass
Portable adapter, exact-resume bundle, and merged release ABIs are distinct	pass
Exact resume restores optimizer, scheduler, scaler, RNG, sampler, and counters before the next stochastic event	pass
Interrupted and uninterrupted teaching runs reconcile exactly under the declared oracle	pass
Merge is a derived release path with evaluation and immutable rollback lineage	pass
Distributed placement is separated from unique logical adapter identity	pass
Adapter checkpoint metadata is explicit	pass
PEFT target discovery and module replacement are traced from configuration to injected factors	pass
Runtime adapter slots are separated from portable checkpoint keys	pass
Reload validates graph reconstruction and tensor population as separate operations	pass
Low-rank evidence is qualified rather than universalized	pass
Competing PEFT mechanisms are compared	pass
Prompt, prefix, bottleneck-adapter, and LoRA state locations are separated at tensor and runtime level	pass
Full fine-tuning, Prompt Tuning, Prefix-Tuning, Houlsby adapters, IA3, BitFit, LoRA, and LST remain distinct paper contracts	pass
Trainable, deployed, optimizer, runtime-graph, sequence-state, and backward-path ledgers are not collapsed	pass
Prompt and direct-prefix causal attention work are derived from different query-row contracts	pass
Prefix training reparameterizer state is separated from materialized release state	pass
Houlsby module, bias, placement, task-normalization, initialization, and runtime contracts are explicit	pass
IA3 encoder, encoder-decoder, mixed-task live, and single-task folded forms are separated	pass
BitFit bias selection is counted from an explicit graph rather than generalized from BERT names	pass
The seven-method decoder teaching ledger reconciles through the Loop 9 oracle	pass
Every adaptation artifact participates in model and cache identity	pass
QLoRA/DoRA variants and serving runtime have separate owner articles	pass
Thirty typed failures bind invariants to detectors, consequences, and recovery owners	pass
All 32 sources, 29 external URLs, five local routes, 53 A22 figure dispositions, and 33 image routes reconcile statically	pass
Nine invalid-fixture families contain 566 one-field failures across Loops 2-10	pass
Static Loop 11 closure remains distinct from normalized browser evidence	pass
All 189 display formulas reconcile through the source registry and generated MathJax surface	pass
Desktop/mobile and light/dark profiles preserve images, local links, headings, tables, themes, and article-local overflow containment	pass
Browser closure is not promoted to live external availability, production runtime, quality, or latency evidence	pass

0xFF References

• Transformer Series A22 - LoRA
• M6.1 - Routed Capacity
• M6.2 - Expert Parallelism and DeepSeek MoE
• LoRA: Low-Rank Adaptation of Large Language Models, arXiv v2
• Prefix-Tuning: Optimizing Continuous Prompts for Generation, arXiv v1
• The Power of Scale for Parameter-Efficient Prompt Tuning, arXiv v2
• Parameter-Efficient Transfer Learning for NLP, arXiv v2
• Few-Shot Parameter-Efficient Fine-Tuning Is Better and Cheaper than In-Context Learning, arXiv v2
• BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language Models, arXiv v2
• AdapterHub: A Framework for Adapting Transformers, arXiv v3
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning, arXiv v1
• Decoupled Weight Decay Regularization (AdamW), arXiv
• LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning, arXiv
• Low-Rank Adaptation for Foundation Models: A Comprehensive Review, arXiv v2
• Microsoft LoRA repository at commit c4593f0
• Microsoft loralib.MergedLinear source at commit c4593f0
• Hugging Face PEFT v0.19.1 LoRA reference source
• Hugging Face PEFT v0.19.1 adapter injection guide source
• Hugging Face PEFT v0.19.1 BaseTuner.inject_adapter source
• Hugging Face PEFT v0.19.1 checkpoint guide source
• Hugging Face PEFT checkpoint format documentation
• PyTorch 2.8 Linear documentation
• PyTorch saving and loading models tutorial
• PyTorch automatic mixed precision recipe
• PyTorch 2.11 reproducibility note
• PyTorch 2.13 Distributed Checkpoint documentation