M6.4 - The Low-Rank Adaptation Design Space: QLoRA, AdaLoRA, rsLoRA, LoRA+, LoRA-FA, PiSSA, LoftQ, and DoRA

0x00 Overview: Why LoRA Became A Design Space

M6.3 specified vanilla LoRA:

$$Y = XW^\top + sXA^\top B^\top$$

with frozen $W$, trainable $A$ and $B$, and usually:

$$s = \frac{\alpha}{r}$$

That baseline contains several independent design decisions:

base representation:
    full precision or quantized

rank:
    fixed per matrix or allocated dynamically

scaling:
    alpha/r or another rank-dependent rule

optimization:
    same or different learning rates, or one frozen factor

initialization:
    random/zero, spectral, gradient-aligned, or quantization-aware

weight parameterization:
    additive update only or magnitude/direction decomposition

Vanilla LoRA solves one problem: it replaces an unrestricted dense update with a low-rank trainable branch. It does not decide how the frozen base should be stored, whether every module deserves the same rank, how update magnitude should behave as rank changes, how the two factors should be initialized or optimized, or whether weight magnitude and direction should be adapted together.

Those unresolved choices are why “a LoRA variant” is not one category of modification. Each method should first be located in the training and execution path.

The owner question is:

Which limitation of vanilla LoRA does each major variant change, and what does that change do to tensors, gradients, training memory, mergeability, and runtime integration?

0x00A LoRA Solved One State Problem, Not The Whole Training System

The original LoRA result was a decisive change in where a task update is stored. Instead of optimizing every entry of a dense pretrained matrix, it freezes that matrix and optimizes a low-rank product. But a training run is not described by trainable parameter count alone. It also owns a base representation, an initialization, a scale rule, an optimizer, retained activations, a checkpoint ABI, and an export path.

One useful contract is:

adaptation system =
    base representation
  + update parameterization
  + initialization
  + rank and scale policy
  + backward and optimizer state
  + checkpoint and merge representation
  + runtime execution path

Vanilla LoRA changed the update parameterization. The methods in this article exist because the other coordinates remained open. A method that quantizes the frozen base and a method that changes the two factor learning rates can both be called “LoRA variants,” yet they do not solve the same bottleneck and do not produce the same artifact.

0x00B The Design Space Grew As Distinct Bottlenecks Became Visible

The relevant papers form a problem-response sequence rather than one linear replacement ladder:

Paper lineage	Limitation made explicit	Object changed
AdaLoRA	equal rank at every target spends a global budget uniformly	singular-component allocation, importance state, and budget schedule
QLoRA	frozen weights still dominate training residency at 16 bits	frozen-base representation, dequantization path, quantization metadata, and paging policy
LoRA-FA	few trainable parameters do not eliminate activation retention	trainable-factor set and backward liveness
LoftQ	ordinary LoRA starts from a quantized base with an uncorrected representation error	coupled quantized base and low-rank initialization
rsLoRA	the usual rank divisor can weaken learning as rank grows	rank-dependent branch scale
DoRA	one additive low-rank delta couples weight magnitude and direction	trainable magnitude plus a normalized low-rank directional update
LoRA+	equal factor learning rates need not preserve efficient feature learning at large width	optimizer groups and factor learning-rate ratio
PiSSA	random/zero factors spend early optimization discovering a useful subspace	spectral factor initialization and frozen residual base

Dates establish lineage, not superiority. Each paper evaluates a different model/task/precision surface, and later revision numbers matter because titles, algorithms, and implementation guidance changed. Benchmark values remain source-reported until an article-local reproduction says otherwise.

The handwritten Transformer Series A22 already exposes the broader family: decomposition, pruning or freezing of adapter state, quantized bases, rank refinement and augmentation, spectral initialization, and continual composition. M6.4 narrows that survey into eight mechanism owners whose tensor, gradient, optimizer, and artifact contracts can be compared directly. Other A22 variants remain registered in the figure and source manifests rather than disappearing from the lineage.

0x00C Compare The Changed Object Before Comparing Results

The first comparison key is not benchmark score. It is the object whose state transition differs from M6.3:

Method	Frozen or baseline object consumed	New mutable or derived object	First parity question
QLoRA	full-precision pretrained weight identity	packed codes, block metadata, dequantized compute view, floating adapter	does the declared quantized operator match the training graph?
AdaLoRA	fixed-rank $BA$ target	SVD-like triplets, importance history, active mask, budget phase	does masked/final state reconstruct the exported rank allocation?
rsLoRA	fixed rank with $\alpha/r$ scale	alternative rank-dependent scale	are forward and gradient comparisons made at a declared rank/alpha control?
LoRA+	one optimizer group for both factors	factor-specific learning-rate groups	do saved optimizer groups preserve factor identity on resume?
LoRA-FA	both factors trainable	frozen projection-down factor and trainable projection-up factor	does backward omit the full-rank activation solely because the frozen factor permits it?
PiSSA	immutable original base plus zero initial delta	trainable principal factors plus frozen spectral residual	does residual plus factors reproduce the original function at initialization?
LoftQ	independently quantized base and ordinary adapter initialization	alternating quantization/error-compensation state	does the exported pair reproduce the final initialization objective?
DoRA	additive $W_0+BA$	magnitude vector and normalized directional weight	does merge preserve the exact column-norm convention and epsilon policy?

This prevents a common category error: two methods may leave the eventual dense matrix mergeable while requiring different training state to reach it. Conversely, two methods may reduce memory in different ledgers: QLoRA reduces frozen-base storage, while LoRA-FA targets adapter-side activation liveness.

0x00D Evidence Classes, Claim Status, And Limits

M6.4 keeps four evidence classes separate:

Evidence class	What it can establish	What it cannot establish alone
A22 source-local material	how the handwritten series introduced and connected variant families	correctness of every survey statement or current framework behavior
pinned primary paper	the method, assumptions, equations, figures, and source-reported experiments at one revision	universal quality, memory, or speed claims
pinned official implementation	behavior of named files and symbols at one commit/release	equivalence to every paper revision or backend
Transformer System derivation and oracle	exact consequences under the declared tiny tensor/state contract	reproduction of a paper-scale run or production kernel

An equation derived here is marked as a Transformer System derivation unless it is tied to a paper equation. An implementation trace names a commit. A paper figure is retained with provenance and a semantic job; it is not decorative evidence.

0x00E Owner, Boundary, And Loop Contract

M6.4 consumes the vanilla operator, gradients, factor identities, checkpoint ABI, and merge limits frozen in M6.3. It owns only the state that changes those contracts:

owned here:
    QLoRA training representation
    AdaLoRA allocation and schedule state
    rsLoRA scaling
    LoRA+ and LoRA-FA optimization/backward state
    PiSSA and LoftQ initialization state
    DoRA magnitude-direction parameterization
    compatibility and paper-by-contract comparison

deferred:
    serving many adapter identities -> M6.5
    general low-rank theory and diagnostics -> M6.7
    routed/shared/continual adapters -> M6.9
    generic numerical formats and deployment quantization -> M8

forbidden collapse:
    do not rederive vanilla LoRA as a variant
    do not turn NF4 into a general quantization tutorial
    do not infer a production serving path from mergeability
    do not rank papers by incomparable source-reported benchmarks

The twelve-loop evidence program follows the changed object through source, tensor, gradient, optimizer, checkpoint, compatibility, failure, and browser contracts. Loop 1 freezes the paper/code/figure surface and all 53 A22 figure dispositions. Later loops may deepen or correct prose, but they must not change an owner or source revision silently.

The point is not to produce a list of acronyms. Two papers can use the same $BA$ shape while changing different objects, consuming different state, and requiring different runtime support.

0x01 Prerequisites: The Independent Axes Inside A LoRA System

0x01A Freeze The Vanilla Baseline Before Comparing Variants

For one target matrix:

$$W_0\in\mathbb{R}^{d_{out}\times d_{in}}, \quad A\in\mathbb{R}^{r\times d_{in}}, \quad B\in\mathbb{R}^{d_{out}\times r}$$

vanilla LoRA computes:

$$Y=XW_0^\top+sXA^\top B^\top$$

with the usual baseline contract:

W_0:
    frozen, commonly stored in fp16 or bf16

A and B:
    trainable low-rank factors

rank r:
    fixed for the module

scale s:
    commonly alpha/r

initial effect:
    commonly zero because B starts at zero

A method comparison is meaningful only if it states which line changes. Otherwise QLoRA, AdaLoRA, and DoRA appear to be competing answers to the same question when they are not.

0x01B Separate Base State, Adapter State, And Optimizer State

The forward equation hides three storage classes:

State	Example objects	Typical owner
base state	$W_0$, quantization codes/scales, frozen residual	checkpoint/runtime
adapter state	$A$, $B$, magnitude vector, rank mask	task adaptation
optimizer state	moments, importance statistics, schedules	training only

QLoRA primarily reduces base-state memory during training. AdaLoRA adds optimizer-side importance and rank-allocation state. DoRA adds a trainable magnitude object. None of these conclusions can be read from adapter parameter count alone.

0x01C Quantization Changes Representation And Compute Precision

Quantization maps a block of higher-precision weights to compact codes plus metadata:

full-precision block W
    -> normalize by block scale
    -> map values to finite codebook
    -> store codes + scale metadata

During a matrix multiplication, the runtime dequantizes or otherwise consumes that packed representation in a supported kernel. Three dtypes may therefore differ:

storage dtype:
    how frozen weights are packed

compute dtype:
    precision used for matrix multiplication/accumulation

adapter dtype:
    precision of trainable A/B and their optimizer state

QLoRA is not “LoRA with 4-bit gradients.” It keeps a quantized frozen base, performs differentiable computation through its dequantized values, and trains floating-point adapter factors.

0x01D Rank Has Allocation, Scaling, And Optimization Consequences

For one module, LoRA parameters scale as:

$$P_{adapter}=r(d_{in}+d_{out})$$

But a fixed total rank budget can be distributed uniformly or concentrated in selected modules. Rank also changes branch magnitude and gradient statistics. This creates three independent design questions:

allocation:
    which modules receive how much rank?

scaling:
    should s vary as 1/r, 1/sqrt(r), or another rule?

optimization:
    should A and B use the same learning rate?

AdaLoRA addresses allocation, rsLoRA scaling, LoRA+ factor-specific learning rates, and LoRA-FA factor freezing. They can share the same visible factor shapes at one instant while implementing different training dynamics and activation-retention requirements.

0x01E Initialization Chooses The Starting Function And Subspace

Standard random-$A$/zero-$B$ initialization preserves the base function at step zero:

$$BA=0$$

Spectral methods instead initialize from structure in a weight or residual. Quantization-aware methods initialize against quantization error. These choices affect:

whether the initial model exactly matches W_0
which directions receive early gradients
whether W_0 is redefined as a residual
what must be saved to reconstruct or merge the model

PiSSA and LoftQ therefore cannot be reduced to alternative random seeds.

0x01F Training Efficiency, Mergeability, And Serving Are Different Tests

A method can reduce training memory while making deployment more specialized. Ask three separate questions:

training:
    which weights, gradients, optimizer states, and activations exist?

artifact:
    can the adaptation be represented as an additive dense delta?

runtime:
    can current kernels execute or merge that representation correctly?

An algebraically mergeable update may still be kept live to support many adapters. A quantized training base does not force one inference quantization format. A dynamic rank schedule may end in a fixed deployment rank, while a magnitude/direction parameterization may need a specialized merge path.

0x01G Comparison Map

Method	Primary changed mechanism	What remains LoRA-like	New state or constraint
QLoRA	frozen base storage and quantized training path	trainable low-rank factors	quantization metadata, dequantization compute, paged optimizer behavior
AdaLoRA	rank allocation across modules and time	low-rank update family	importance scores, singular components, rank budget schedule
rsLoRA	rank-dependent branch scaling	same factor shapes and optimizer interface	scale $\alpha/\sqrt{r}$ rather than $\alpha/r$
LoRA+	optimization rate of the two factors	same forward equation	separate optimizer groups and learning-rate ratio
LoRA-FA	trainable-factor set and adapter activation retention	same two-factor forward equation	freeze random projection-down $A$; train projection-up $B$
PiSSA	initialization and frozen residual decomposition	same low-rank branch architecture after setup	SVD-derived factors and residual base weight
LoftQ	joint quantization and low-rank initialization	quantized base plus trainable factors	alternating quantization/factor initialization
DoRA	weight magnitude/direction parameterization	low-rank directional update	trainable magnitude and column normalization

These methods are not mutually exclusive in principle. Implementations may combine, for example, quantized-base training with rank-stable scaling or DoRA with quantization. Compatibility must be checked at the actual framework and kernel level.

0x02 QLoRA: Quantize The Frozen Base, Not The Learning Signal

QLoRA addresses a different memory object from vanilla LoRA.

Vanilla LoRA:

base:
    frozen, commonly fp16/bf16

adapter:
    trainable

QLoRA:

base storage:
    frozen 4-bit quantized weights

base compute:
    dequantized into a higher compute dtype for matrix multiplication

adapter:
    trainable low-rank factors, usually in a floating dtype

gradient:
    passes through the dequantized base computation into activations
    and into adapter parameters

The defining equation is conceptually:

$$Y = X\,\operatorname{Dequant}(W_q, q)^\top + sXA^\top B^\top$$

where:

W_q:
    packed quantized values

q:
    scale, zero-point, block, or codebook metadata

Dequant(W_q, q):
    compute-time approximation of the original frozen W

QLoRA does not train the 4-bit codes as ordinary dense parameters. It backpropagates through the frozen quantized-base computation to the trainable adapters.

Source: QLoRA, Figure 1. The diagram separates the 4-bit frozen base, higher-precision adapter path, and paged optimizer state. It describes the training-memory design, not a required inference format.

0x02A Freeze The QLoRA Representation Before Discussing Memory

“The model is four bit” is not a sufficient representation manifest. It does not identify the codebook, block boundaries, scale hierarchy, compute dtype, adapter dtype, optimizer owner, or page residency. For one adapted linear operator, freeze at least:

Object	Logical shape	Stored representation	Trainable?	Lifetime
base matrix	$[d_{out},d_{in}]$	two NF4 codes per byte	no	persistent
first-level block scales	$[\lceil d_{out}d_{in}/G\rceil]$	direct floating scales or nested codes	no	persistent
second-level scales and offset	quantizer-dependent	higher-precision metadata	no	persistent
dequantized base view	tile or $[d_{out},d_{in}]$	compute dtype	no	forward/backward ephemeral
adapter $A$	$[r,d_{in}]$	floating training dtype	yes	persistent
adapter $B$	$[d_{out},r]$	floating training dtype	yes	persistent
adapter gradients	same as $A,B$	gradient dtype	optimizer input	backward ephemeral or accumulated
optimizer moments	two states per $A,B$ scalar for AdamW	optimizer-state dtype	updated	logically persistent; residency may move

The QLoRA paper’s single-layer operator makes the storage/compute split explicit:

$$Y^{\mathrm{BF16}} = X^{\mathrm{BF16}} \operatorname{doubleDequant} \left(c_1^{\mathrm{FP32}},c_2^{k\text{-bit}},W^{\mathrm{NF4}}\right) + X^{\mathrm{BF16}}L_1^{\mathrm{BF16}}L_2^{\mathrm{BF16}}$$

BF16 is the paper’s common compute contract, not a universal implementation law. The reference script derives compute_dtype from its fp16/bf16 flags and passes it separately from load_in_4bit, quant_type, and double_quant. A reproducible run therefore records all four fields rather than inferring compute from storage.

The pinned 2023 implementation path is:

QLoRA qlora.py @ 7f4e95a
    BitsAndBytesConfig(
        load_in_4bit = bits == 4,
        bnb_4bit_compute_dtype = compute_dtype,
        bnb_4bit_use_double_quant = double_quant,
        bnb_4bit_quant_type = quant_type,
    )
    -> Transformers 4.31.0 creates bitsandbytes Linear4bit modules
    -> bitsandbytes 0.40.0 Params4bit packs W and retains quant_state
    -> PEFT 0.4.0 freezes base parameters and injects trainable LoRA factors
    -> Trainer selects paged_adamw_32bit for adapter optimizer state

This is versioned implementation evidence. It is not a promise that a modern Transformers, PEFT, or bitsandbytes release has the same internal tuple layout.

0x02B Why NF4 Uses Nonuniform Levels

The QLoRA paper motivates NormalFloat 4-bit for weights that are approximately normally distributed after normalization.

Uniform 4-bit quantization allocates equally spaced levels:

$$q_i = q_{\min} + i\Delta$$

This spends as many representable levels in low-density tails as in the high-density region near zero.

A quantile-oriented codebook instead chooses boundaries so intervals carry more nearly equal probability mass under a reference distribution. If $Q_X=\Phi^{-1}$ is the standard-normal quantile function, the paper first defines midpoint representatives:

$$q_i = \frac{1}{2} \left[ Q_X\!\left(\frac{i}{2^k+1}\right) + Q_X\!\left(\frac{i+1}{2^k+1}\right) \right]$$

This formula explains the nonuniform spacing: equal probability intervals in the normal reference distribution do not have equal numerical width. It does not yet solve exact zero.

The resulting levels are denser where normalized Gaussian-like weights are dense. QLoRA also needs an exact representation of zero. A naively symmetric even-sized quantile codebook has no single central zero level, so the paper constructs negative and positive quantile sets separately, joins them, and removes one duplicate zero. The resulting NF4 codebook is asymmetric even though the reference weight distribution is zero-centered.

That construction is why this shorthand is incomplete:

NF4 = 16 ordinary standard-normal quantiles

The full contract includes:

reference distribution:
    zero-centered normal

block normalization:
    absolute-maximum scaling into the codebook range

codebook:
    quantile-derived, normalized, with an exact zero

stored metadata:
    packed 4-bit codes plus block scales

At the pinned bitsandbytes 0.40.0 implementation, the normalized NF4 table is:

[-1.0000000, -0.6961928, -0.5250731, -0.3949175,
 -0.2844414, -0.1847734, -0.0910500,  0.0000000,
  0.0795803,  0.1609302,  0.2461123,  0.3379152,
  0.4407098,  0.5626170,  0.7229568,  1.0000000]

The table is asymmetric around the exact zero. Deriving 16 values from a different probability grid can produce a different datatype even if both are called “normal four bit.” Codebook identity is checkpoint state.

The safe claim is not that every weight tensor is perfectly normal. It is that NF4 is designed for the empirical distributional structure QLoRA targets and was validated in that training system.

0x02C Blockwise Quantization Tensor Trace

Suppose a flattened weight block contains $G$ values:

W_block: [G]

Compute a block scale:

$$c = \max_i |W_i|$$

Normalize:

$$\hat{W}_i = \frac{W_i}{c}$$

Map each normalized value to a 4-bit code:

$$z_i = Q_{NF4}(\hat{W}_i)$$

Store:

codes:
    [G] values at 4 bits each

scale:
    one higher-precision value c for the block

During compute:

$$\widetilde{W}_i = c \cdot Q_{NF4}^{-1}(z_i)$$

The quantization residual is:

$$E_q = W - \widetilde{W}$$

The LoRA update must learn the downstream task while operating on $\widetilde{W}$, not the original full-precision $W$. That is why quantization-aware initialization methods such as LoftQ become relevant.

The Loop 2 executable fixture uses a deliberately small matrix so every stored nibble can be inspected. It uses row-major flattening, $G=4$, and three blocks:

Block	Full-precision values	absmax $c_j$	NF4 code indices	Packed bytes, low nibble first
0	$[-1,-0.55,0,0.72]$	$1.00$	$[0,2,7,14]$	$[32,231]$
1	$[0.15,-0.05,0.30,-0.50]$	$0.50$	$[11,6,13,0]$	$[107,13]$
2	$[0.05,0.25,-0.20,0.75]$	$0.75$	$[8,11,4,15]$	$[184,244]$

For example, byte $107=0x6b$ stores code $11$ in the low nibble and code $6$ in the high nibble. Reversing the nibble convention preserves the byte count but reconstructs the wrong values. Shape, flatten order, block membership, codebook, and nibble order are therefore one representation contract.

The teaching fixture uses $G=4$ only to make the trace readable. The paper and pinned reference path use a base-weight block size of 64 for the reported double-quantization ledger.

0x02D Double Quantization

Blockwise quantization introduces many scale constants. If each scale is stored at high precision, metadata becomes visible for small block sizes.

For:

P:
    number of base parameters

G:
    values per quantization block

b_c:
    bytes per scale

scale storage is approximately:

$$M_{\text{scales}} \approx \frac{P}{G}b_c$$

Double quantization quantizes those quantization constants as well. The second level adds its own grouping metadata but reduces average scale storage. The mechanism is not “quantize the LoRA adapter twice”; it is quantization of the first-level quantization constants.

Let the positive first-level scales be $c_j$. The paper centers them before the second quantizer because a symmetric 8-bit quantizer otherwise wastes half its signed range:

$$\mu_c=\frac{1}{N_c}\sum_{j=1}^{N_c}c_j, \qquad \bar c_j=c_j-\mu_c$$

For a second-level group $g$ with absmax $a_g$ and signed range $[-q_{max},q_{max}]$, a transparent teaching quantizer is:

$$z_j^{(c)} = \operatorname{round}\!\left(q_{max}\frac{\bar c_j}{a_g}\right), \qquad \widetilde c_j = \mu_c+a_g\frac{z_j^{(c)}}{q_{max}}$$

The executable fixture has $c=[1,0.5,0.75]$, so $\mu_c=0.75$, $a_g=0.25$, and $z^{(c)}=[127,-127,0]$. It reconstructs those three scales exactly. This small signed-int8 example verifies the metadata dependency graph; it does not reproduce the CUDA kernel’s 8-bit code map.

The QLoRA paper’s source-reported ledger makes the metadata cost concrete. With a first-level block size of 64 and 32-bit scale constants:

$$\frac{32\ \mathrm{bits}}{64\ \mathrm{weights}} =0.5\ \mathrm{bit/weight}$$

With 8-bit first-scale codes, 256 first scales per second-level FP32 scale, and 64 weights per first-level scale, the metadata becomes:

$$b_{DQ} = \frac{8}{64} + \frac{32}{64\cdot256} = 0.126953125\ \text{bit/weight}$$

and the derived saving is:

$$\Delta b = \frac{32}{64}-b_{DQ} = 0.373046875\ \text{bit/weight}$$

The reported “about 0.373 bit” follows from this geometry; it is not an intrinsic constant of nested quantization. A complete storage ledger also includes packed-container padding, the offset, codebook identity, tensor shape, and any serialization metadata.

The nested dequantization path is:

second-level codes + second-level scales
    -> reconstruct first-level block scales
        -> reconstruct base weight blocks from NF4 codes
            -> cast/use in the matrix-multiplication compute dtype

0x02E Paged Optimizers

LoRA reduces persistent optimizer state, but long sequences and activation peaks can still create transient memory spikes. QLoRA uses paged optimizers backed by CUDA unified-memory behavior to move optimizer pages when needed.

This changes the failure mode:

without paging:
    transient peak exceeds device memory
    -> allocation failure

with paging:
    some optimizer pages migrate
    -> training may survive
    -> transfer and page-fault latency can appear

Paged memory is pressure relief, not free capacity. A training profile should still record page migration, step-time variance, and peak resident memory.

The accounting identity is:

$$M_{opt}^{logical} = M_{opt}^{GPU\ resident}(t) + M_{opt}^{backed}(t)$$

Paging can change the two terms on the right over time. It does not reduce the logical Adam moment tensors on the left. For $P_A+P_B$ adapter scalars and two FP32 Adam moments:

$$M_{opt}^{logical}=2(P_A+P_B)\cdot4\ \text{bytes}$$

The Loop 2 fixture has 14 adapter scalars, so its logical moments occupy 112 bytes. A teaching snapshot with 56 bytes resident on the GPU still has 112 logical bytes; the other 56 are backed elsewhere. The paper did not establish a universal paging-latency bound, so migration frequency and step-time tails remain measured runtime properties.

0x02F QLoRA Training Step

def qlora_linear(x, packed_weight, quant_meta, A, B, scale):
    # Dequantization may be fused into the base matmul kernel.
    base_y = quantized_linear(
        x,
        packed_weight,
        quant_meta,
        compute_dtype=torch.bfloat16,
    )

    adapter_hidden = F.linear(x, A)
    adapter_y = F.linear(adapter_hidden, B)
    return base_y + scale * adapter_y

Trainability assertion:

assert not packed_weight.requires_grad
assert A.requires_grad
assert B.requires_grad

Memory ownership:

GPU persistent:
    packed base
    quantization metadata
    adapter parameters
    currently resident optimizer pages

autograd:
    activations needed for adapter and input gradients

CPU/unified backing:
    optimizer pages that may migrate under pressure

The backward path still needs the input gradient through the frozen base:

$$\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y}\widetilde{W} + s\frac{\partial L}{\partial Y}BA$$

but does not allocate a trainable gradient for the packed base weights:

computed:
    dX
    dA
    dB

not computed as an optimizer parameter gradient:
    dW_q

That distinction explains why QLoRA reduces persistent model/optimizer memory without eliminating activation memory or backward compute through the base operator.

For $U=XA^\top$, $Y=X\widetilde W^\top+sUB^\top$, and upstream $G=\partial L/\partial Y$, the local adapter gradients are:

$$\frac{\partial L}{\partial B}=sG^\top U, \qquad \frac{\partial L}{\partial A}=s(G B)^\top X$$

and the input gradient is:

$$\frac{\partial L}{\partial X} = G\widetilde W+s(GB)A$$

The dequantized base participates in $dX$ even though packed codes and scale metadata are not optimizer parameters. That is the precise meaning of “backpropagating through the quantized model”: the graph differentiates the input-side computation while optimizer ownership stops at $A$ and $B$.

The liveness consequence is configuration-dependent:

persistent:
    packed codes, quantization metadata, A, B, logical optimizer moments

forward/backward ephemeral:
    dequantized weight tile or matrix, X, adapter hidden U, outputs

optimizer gradients:
    dA, dB

not optimizer gradients:
    d(packed codes), d(first-level scales), d(second-level scales)

A fused 4-bit matrix multiplication may reconstruct only tiles rather than a full dense $\widetilde W$. The algebra is identical; peak temporary memory and traffic are not. The executable oracle therefore proves the operator and state ownership, not one production kernel’s liveness schedule.

The QLoRA experiments also change the adaptation surface relative to the original LoRA paper’s common query/value setup. The paper reports that applying LoRA to all linear Transformer-block layers was important in its evaluated configurations for matching its 16-bit baselines. This is source-reported evidence, not a theorem that every task needs every linear layer, but it blocks an oversimplified recipe:

QLoRA = quantize base + always adapt q_proj and v_proj only

Target modules remain a training design choice with parameter, memory, and quality consequences.

0x02G Executable Representation And Failure Oracle

The checked evidence bundle at docs/numinens-evidence/m6-4-adaptation-design-space/ executes this contract without requiring a GPU. It:

1. assigns the exact bitsandbytes 0.40.0 NF4 levels to a $3\times4$ matrix;
2. packs two 4-bit codes into each byte and reconstructs nested scales;
3. executes the dequantized base plus floating LoRA forward path;
4. checks the equivalent dense update in the dequantized compute domain;
5. checks analytical $dX,dA,dB$ against finite differences;
6. derives packed, metadata, adapter, gradient, optimizer, and page-residency ledgers;
7. rejects one-field mutations that collapse storage, compute, optimizer, or deployment state.

The oracle’s exact result is a teaching proof, not a paper reproduction. Its $G=4$ and second-level group of three scales make every code auditable; paper geometry remains separately frozen at $64$ and $256$.

Invalid implementation	Why it can appear plausible	Broken contract
mark packed base codes trainable	they are represented by a parameter wrapper	optimizer must own only adapter factors
quantize $A$ and $B$ with the base	”everything is four bit” sounds memory-efficient	QLoRA keeps the learning signal in floating factors
change flatten or nibble order	byte counts and tensor shapes still match	reconstructed weights differ
omit scale centering or offset	nested codes still decode to numbers	first-level scales reconstruct incorrectly
count paged-out moments as removed	GPU residency falls	logical optimizer state is unchanged
commit a merged delta directly into packed nibbles	packed storage looks mutable	merge requires an explicit dequantize/update/requantize path
generalize all-linear targeting to every task	it matched the reported configuration	target scope is an empirical design choice

0x02H QLoRA Is Not An Inference Format Decision

QLoRA specifies a training-memory path. After training, deployment can choose:

keep the supported quantized base and live adapters
merge through a supported high-precision path and requantize
export a different quantization format
serve the adapter separately

The best training representation and the best inference representation need not be identical. M8 owns the final compression decision; M6.5 owns live adapter serving.

0x03 AdaLoRA: Allocate Rank Where The Update Matters

Vanilla LoRA usually assigns the same rank to every selected matrix. AdaLoRA asks whether that uniform allocation is the best use of a global rank budget. Its owner question is narrower than “how should LoRA be improved?”:

How does a training loop estimate the importance of individual low-rank components, reduce one global budget over time, project that budget back into module-local tensors, and preserve enough state to resume or export the result correctly?

This section owns the SVD-like triplet, score state, cubic budget, global mask, and masked-versus-compact artifact boundary. It does not own rank-stable scaling, live heterogeneous-rank serving, or general structured pruning; those belong to 0x04, M6.5, and M6.11 respectively.

0x03A Uniform Rank Is An Allocation Policy

For $n$ adapted matrices, fixed-rank LoRA sets:

$$r_k=r,\qquad \sum_{k=1}^{n}r_k=nr$$

before observing a task gradient. That is a valid baseline, not a consequence of low-rank algebra. The AdaLoRA paper motivates a different policy with module- and layer-selection experiments: under one reported 0.28M-parameter budget, different DeBERTaV3-base modules and layers did not contribute equally on MNLI-m.

Source: AdaLoRA v2, Figure 1. These are source-reported ablations for one model, task, and budget. They motivate adaptive allocation; they do not prescribe a universal decoder-only rank pattern.

AdaLoRA defines the scheduled budget as the number of retained singular values:

$$b^{(t)}=\sum_{k=1}^{n}r_k^{(t)}$$

That unit must not silently become an exact scalar-parameter budget. A retained triplet for a matrix $W_k\in\mathbb{R}^{d_{out,k}\times d_{in,k}}$ physically contains:

$$c_k=d_{out,k}+d_{in,k}+1$$

factor scalars. After physical compaction, the adapter scalar count is:

$$P_{compact}=\sum_{k=1}^{n}r_k^{(T)} \left(d_{out,k}+d_{in,k}+1\right)$$

Therefore equal rank units have equal scalar cost only when their matrix dimensions match. The paper’s budget-matching tables choose average ranks for the evaluated adaptation surface; the rank counter alone is not a universal byte ledger.

0x03B One Triplet Is Three Coupled Tensor Slices

The paper parameterizes one update as:

$$\Delta W_k=P_k\Lambda_kQ_k$$

with:

$$P_k\in\mathbb{R}^{d_{out,k}\times r},\qquad \lambda_k\in\mathbb{R}^{r},\qquad Q_k\in\mathbb{R}^{r\times d_{in,k}}$$

where $\Lambda_k=\operatorname{diag}(\lambda_k)$. The $i$-th allocation unit is the triplet:

$$\mathcal{G}_{k,i}=\{P_{k,*i},\lambda_{k,i},Q_{k,i*}\}$$

The pinned reference code names these tensors differently:

Paper object	Reference field	Shape	Rank axis
$Q_k$	lora_A	$[r,d_{in}]$	rows
$\lambda_k$	lora_E	$[r,1]$	rows
$P_k$	lora_B	$[d_{out},r]$	columns

That mapping determines every reduction and export axis. The live branch in the reference implementation is:

hidden = dropout(x) @ (lora_A * lora_E).T
delta_y = hidden @ lora_B.T
y = base_linear(x) + delta_y * lora_alpha / (ranknum + 1e-5)

lora_E broadcasts over the rows of lora_A. Masking entry $i$ of lora_E removes exactly $P_{*i}\lambda_iQ_{i*}$ from the update without zeroing the two vector factors. The pinned reference keeps ranknum at the initialized rank in this core path, so the scale is not automatically recomputed from the current number of nonzero entries. Rank-dependent scale alternatives belong to 0x04.

Initialization also differs from ordinary two-factor LoRA:

lora_A / Q:
    random Gaussian

lora_B / P:
    random Gaussian

lora_E / lambda:
    zeros

result:
    Delta W = 0 even though both vector factors are nonzero

Keeping the vector factors nonzero matters later: a masked singular value can become nonzero again after subsequent optimizer and allocation steps.

0x03C Orthogonality Makes Triplet Projection Better Defined

The parameterization resembles an SVD but is not an exact SVD computed after every optimizer step. AdaLoRA instead adds:

$$R(P_k,Q_k)= \left\|P_k^\top P_k-I\right\|_F^2+ \left\|Q_kQ_k^\top-I\right\|_F^2$$

and trains with:

$$\mathcal{L}= \mathcal{C}(\mathcal{P},\mathcal{E},\mathcal{Q})+ \gamma\sum_{k=1}^{n}R(P_k,Q_k)$$

Why add this term? If columns of $P$ and rows of $Q$ are strongly dependent, zeroing one factor pair can perturb directions also represented by neighboring pairs. Orthogonality pressure reduces that coupling, making a singular-value mask closer to an interpretable rank projection. It does not prove the learned factors are the exact singular vectors of the dense task update.

The alternative of recomputing an exact SVD of every dense update has cost on the order of:

$$O\!\left(\min(d_{out},d_{in})d_{out}d_{in}\right)$$

per matrix and allocation event. AdaLoRA pays instead for trainable triplets, orthogonality loss, and score state.

0x03D Derive Sensitivity From The Removal Operation

For one trainable scalar $w$, a first-order loss expansion is:

$$\mathcal{L}(w+\delta w)\approx \mathcal{L}(w)+\frac{\partial\mathcal{L}}{\partial w}\delta w$$

Removing the scalar means $\delta w=-w$, so the magnitude of the predicted loss change is:

$$I^{(t)}(w)= \left|w^{(t)}\nabla_w\mathcal{L}^{(t)}\right|$$

This explains the weight-gradient product. It is a local first-order proxy, not the measured loss after retraining without that scalar. A mini-batch makes the proxy noisy, so the paper maintains two exponential states:

$$\overline I^{(t)}(w)= \beta_1\overline I^{(t-1)}(w)+ (1-\beta_1)I^{(t)}(w)$$ $$\overline U^{(t)}(w)= \beta_2\overline U^{(t-1)}(w)+ (1-\beta_2) \left|I^{(t)}(w)-\overline I^{(t)}(w)\right|$$

The second equation centers the deviation on the newly updated smoothed sensitivity. The element score is:

$$s^{(t)}(w)=\overline I^{(t)}(w)\overline U^{(t)}(w)$$

and one triplet score combines the singular scalar with the means of both vector slices:

$$S_{k,i}^{(t)}= s(\lambda_{k,i})+ \frac{1}{d_{out,k}}\sum_{j=1}^{d_{out,k}}s(P_{k,ji})+ \frac{1}{d_{in,k}}\sum_{j=1}^{d_{in,k}}s(Q_{k,ij})$$

The means prevent a wider vector from winning only because it contains more entries. Summing those slices, omitting $\lambda$, or reducing the wrong axes changes the allocation while preserving superficially plausible tensor shapes.

This state is not ordinary Adam state. The pinned allocator keeps ipt, exp_avg_ipt, and exp_avg_unc for every LoRA factor scalar. If the adapter has $P_{init}$ full-shape scalars, these dictionaries add up to $3P_{init}$ score scalars before considering optimizer moments.

0x03E The Cubic Budget Has Three Phases

The endpoint-consistent expression implemented by both the pinned AdaLoRA reference and PEFT v0.19.1 is:

$$b_{code}^{(t)}= \begin{cases} b^{(0)}, & t\le t_i \\ b^{(T)}+ \left(b^{(0)}-b^{(T)}\right) \left(1-\dfrac{t-t_i}{T-t_f-t_i}\right)^3, & t_i<t\le T-t_f \\ b^{(T)}, & t>T-t_f \end{cases}$$

The code truncates the middle value to an integer and materializes a new mask only when step % deltaT == 0; final fine-tuning reapplies the fixed target pattern. This produces three different phases:

Phase	Budget	Importance state	Mask action
initial warmup	initial	updated	no allocation mask
reduction	cubic, integer	updated	only at deltaT intervals
final warmup	target	frozen	fixed pattern reapplied

There is a source discrepancy worth preserving. AdaLoRA v2 Appendix A prints:

$$1-\frac{t-t_i-t_f}{T-t_i-t_f}$$

inside the cubic term. For $t_f>0$, that expression can exceed one at $t=t_i$ and therefore exceed the initial budget, contradicting the surrounding prose. For the checked fixture $b^{(0)}=6$, $b^{(T)}=3$, $T=10$, $t_i=t_f=2$, the printed expression gives $10.111\ldots$ at $t_i$; the pinned code gives $6$. This article uses the code expression for its executable schedule and records the printed equation rather than silently attributing the correction to the paper.

0x03F Projection Is Global, Periodic, And Threshold-Sensitive

At a mask event, all triplet scores from all adapted matrices are concatenated. For target budget $b^{(t)}$, the projection is conceptually:

$$\lambda_{k,i}^{(t+1)}= \begin{cases} \widetilde\lambda_{k,i}^{(t)}, & S_{k,i}^{(t)}\text{ is globally retained}\\ 0,&\text{otherwise} \end{cases}$$

The pinned implementations obtain the $(b^{(0)}-b^{(t)})$-th smallest score as a threshold and retain scores strictly greater than it:

all_scores = concat(score[module] for module in adapted_modules)
threshold = kth_smallest(all_scores, initial_budget - current_budget)

for module in adapted_modules:
    module.lora_E.masked_fill_(triplet_score[module] <= threshold, 0.0)

This is global top-budget allocation, not one top-$k$ operation per module. It permits rank zero for a module unless a separate floor is added.

The strict comparison also creates a versioned tie behavior. With scores [10, 8, 5, 5] and requested budget 3, the threshold is 5 and strict > 5 retains only two entries. The paper’s mathematical “top-$b$” contract and this implementation’s exact cutoff behavior are therefore identical only when the boundary score is not tied. A production allocator should define whether it accepts underfill or applies a deterministic tie-fill policy.

Source-local visual reused from Transformer Series A22, which combines AdaLoRA Algorithm 1 with the $P\Lambda Q$ and orthogonality context. English reading contract: compute gradients; update sensitivity, smoothed sensitivity, and uncertainty; score every triplet; update $P,Q$; then project $\Lambda$ under the scheduled global budget. The source image’s top annotation is Chinese, so the equations and loop are transcribed in this section rather than left image-only.

0x03G Paper Algorithm And Runtime Call Order Are Not Identical

The paper’s Algorithm 1 presents this logical order:

forward and task plus orthogonality loss
    -> backward gradients
    -> sensitivity and score-state update
    -> parameter update
    -> global projection of lambda

The pinned reference README and PEFT example show:

(task_loss + orthogonal_regularizer).backward()
optimizer.step()
allocator.update_and_mask(model, global_step)
optimizer.zero_grad()

update_ipt then evaluates abs(parameter * parameter.grad): the gradient is from the just-completed backward pass, while the parameter may already contain the optimizer update. This code path is reproducible, but it is not numerically identical to scoring the pre-update parameter in the paper’s pseudocode. A training report must record the implementation and call order, not cite Algorithm 1 as if it uniquely fixes both.

The final phase adds another state transition. PEFT v0.19.1 stops refreshing importance state, forces one target-budget pattern, stores that Boolean pattern in the adapter config, resets allocator score dictionaries, and reapplies the mask on later steps. Calling zero_grad() before allocation, omitting the global step, or resuming with a reset allocator in the reduction phase changes the learned rank pattern.

0x03H Worked Global Allocation Trace

The checked Loop 3 fixture adapts q_proj and v_proj, each with initial rank three. It observes two mini-batch sensitivity snapshots with $\beta_1=\beta_2=0.5$. After the second update, the exact triplet scores are:

Global order	Triplet	Score	Projection at $b=3$
1	v_proj[0]	5.0625	retain
2	v_proj[1]	3.515625	retain
3	q_proj[0]	3.375	retain
4	q_proj[1]	2.25	mask at strict threshold
5	v_proj[2]	1.6875	mask
6	q_proj[2]	0.5625	mask

The rank pattern is therefore:

q_proj: [true, false, false] -> active rank 1
v_proj: [true, true, false]  -> active rank 2
global active rank           -> 3

For the same fixture, the pinned schedule yields:

Step	Budget	Materialize mask?	Phase
0	6	no	initial warmup
2	6	no	initial boundary
3	4	no	reduction between intervals
4	3	yes	reduction and allocation
8	3	yes	end of reduction
9	3	yes	final fixed pattern

The executable oracle in docs/numinens-evidence/m6-4-adaptation-design-space/ derives every score from actual factor values and gradients, checks both EMA states, applies the global threshold, and reconstructs the update from masked-full and compact factors. It is a mechanism proof, not a reproduction of the paper’s task results.

0x03I Masked Training State Is Not A Compact Artifact

In the fixture, each $2\times2$ update starts with:

$$3(2+2+1)=15$$

factor scalars. Two modules therefore keep 30 factor scalars during masked training even though only three of six singular values are active. The allocator adds:

$$3\times30=90$$

sensitivity/EMA/uncertainty scalars, and two Adam moments add another 60 logical scalars. Zeroed lora_E entries do not delete their factor rows, columns, optimizer slots, or checkpoint storage.

Only an explicit compaction that selects retained rows of lora_A, retained entries of lora_E, and retained columns of lora_B reduces this fixture to:

$$1(2+2+1)+2(2+2+1)=15$$

factor scalars. The oracle proves the full-shape masked and compact tensors produce the same $\Delta W$.

The original reference lora_state_dict filters keys containing lora_; it does not slice away masked triplets. Its NLU example writes rank_pattern.json as separate metadata. PEFT v0.19.1 contains resize_modules_by_rank_pattern and resize_state_dict_by_rank_pattern helpers, but its training transition deliberately leaves module resizing commented out and continues to reapply masks. The existence of a resize helper must not be rewritten as automatic training-time compaction.

Resume therefore has three levels:

Resume point	Required state
after final pattern is fixed	factor tensors, rank pattern, scale/config, global step, optimizer/scheduler state, RNG state
during budget reduction	above plus score EMAs, uncertainty state, mask interval phase
compact inference export	compact factors or merged weight plus exact per-module rank metadata

In the pinned implementations, allocator dictionaries are ordinary Python state rather than ordinary adapter parameters. A generic adapter-only save is not proof of exact mid-schedule resumability.

0x03J Alternatives Change Different Contracts

Design	Allocation signal	Projection object	Recovery after masking	Main cost/risk
fixed-rank LoRA	none	none	not applicable	uniform budget may be suboptimal
singular-value magnitude	$	\lambda_i	$	$\lambda_i$
AdaLoRA sensitivity/uncertainty	score over $P,\lambda,Q$	$\lambda_i$	vectors remain trainable	score state, schedule, and global coordination
prune LoRA doublets	score over $A_{i*},B_{*i}$	both factor slices	difficult if both slices are zeroed	non-orthogonal doublets can be coupled
repeated exact SVD	dense update spectrum	exact singular triplet	depends on implementation	prohibitive repeated dense decomposition

These are not interchangeable names for “dynamic rank.” They differ in what is scored, what is zeroed, whether inactive directions keep trainable state, and whether the result is immediately compact.

0x03K Failure And Runtime Closure

Failure	Plausible symptom	Broken contract
rank each module independently	every module receives some capacity	budget is global, not per-module
reduce vector scores by sum	wider matrices dominate	triplet score uses vector means
mask factor rows and columns	active rank appears smaller	recovery path is destroyed
call allocator after zero_grad()	scores vanish or fail	sensitivity requires retained gradients
restore tensors but not step/score state	training resumes without load error	budget and allocation trajectory change
count nonzero $\lambda$ as resident parameters	parameter ledger looks smaller	training tensors and moments remain full-shape
call rank units exact scalar budget	rank totals reconcile	heterogeneous matrix dimensions change scalar cost
assume cutoff ties always fill budget	ordinary random runs look correct	strict threshold can underfill
rescale by current active rank in the pinned path	output remains finite	algorithm changed into another scale policy

After finalization, heterogeneous ranks complicate kernel grouping, adapter batching, hotswap compatibility, and static compilation. M6.5 owns those live serving consequences. AdaLoRA’s result handed to that article must include at least adapter identity, target modules, finalized rank pattern, scale rule, factor dtype, and whether the stored tensors are masked-full or physically compact.

0x04 rsLoRA: Rank Stability Is A Learning-Trajectory Contract

0x04A Scale Is Part Of The Learning Parameterization

For one adapter, write the branch without hiding its rank-dependent scale:

$$f_r(x)=\gamma_r B A x, \qquad A\in\mathbb{R}^{r\times d_{in}}, \quad B\in\mathbb{R}^{d_{out}\times r}.$$

Increasing $r$ changes at least three objects:

capacity:
    the maximum rank of BA

state and live work:
    r(d_in + d_out) factor scalars
    r(d_in + d_out) MACs per token for an unmerged branch

learning parameterization:
    the scale gamma_r multiplying every forward and backward path

The usual LoRA convention and the rank-stabilized convention are:

$$\gamma_r^{\text{LoRA}}=\frac{\alpha}{r}, \qquad \gamma_r^{\text{rsLoRA}}=\frac{\alpha}{\sqrt r}.$$

They have identical factor shapes and parameter counts. They do not define the same rank sweep. Holding $\alpha$ fixed while changing $r$ changes the optimization signal differently under the two rules. A comparison that records rank but omits $\alpha$ and the scale convention is underspecified.

0x04B Define Rank Stability Before Deriving A Rule

The rsLoRA v1 paper does not define “stable” as “higher rank always produces a better benchmark.” Its theorem is an asymptotic moment statement. In expectation over initialization, it asks for two directions of the adapter to remain order one as $r$ grows:

1. entrywise input moments in $\Theta_r(1)$ should produce adapter-output moments in $\Theta_r(1)$;
2. entrywise adapter-output gradients in $\Theta_r(1)$ should produce input-gradient entries in $\Theta_r(1)$.

The argument assumes:

A_0 entries:
    iid, zero mean, variance sigma_A independent of rank

B_0:
    exactly zero

gamma_r:
    tends to zero as rank tends to infinity

inputs and upstream gradients:
    have the stated rank-independent moments

Those assumptions matter. The result is not a distribution-free theorem about arbitrary initializers, optimizers, normalization stacks, data, or trained features. It identifies the scale class that prevents the paper’s leading forward and backward terms from vanishing or exploding under its model.

0x04C Zero-$B$ Initialization Makes The First Update Auditable

Let $v_n=\nabla_{f_r(x_n)}\mathcal L$ at training step $n$. Matrix calculus gives:

$$\nabla_B\mathcal L = \gamma_r v_n x_n^\top A_n^\top,$$ $$\nabla_A\mathcal L = \gamma_r B_n^\top v_n x_n^\top.$$

At exact initialization, $B_0=0$. Therefore:

$$f_r(x;A_0,B_0)=0, \qquad \nabla_A\mathcal L\big|_0=0,$$

but generally:

$$\nabla_B\mathcal L\big|_0 = \gamma_r v_0 x_0^\top A_0^\top \neq 0.$$

This removes one misleading explanation of rsLoRA: its branch is not “larger at initialization” under zero-$B$ initialization. Both branches emit zero. The scale first appears in the nonzero $B$ gradient and then appears a second time when that learned $B$ is used in the scaled forward branch.

For one SGD step with learning rate $\eta$:

$$B_1 = -\eta\gamma_r v_0x_0^\top A_0^\top, \qquad A_1=A_0.$$

The first learned, scaled dense update is therefore:

$$\gamma_rB_1A_1 = -\eta\gamma_r^2 v_0x_0^\top A_0^\top A_0.$$

The two powers of $\gamma_r$ come from different places: one from optimizing $B$, one from executing the resulting adapter.

0x04D Derive The $\gamma_r^2r$ Law

The paper extends the first-step calculation along the SGD trajectory. To leading order:

$$B_n= \left( -\eta\gamma_r\sum_{k<n}v_kx_k^\top +\mathcal O_r(\gamma_r^2) \right)A_0^\top,$$ $$A_n=A_0\left(1+\mathcal O_r(\gamma_r^2)\right).$$

Substitute both into the branch:

$$\gamma_r B_nA_n = -\eta\gamma_r^2 \sum_{k<n}v_kx_k^\top A_0^\top A_0 +\mathcal O_r(\gamma_r^3)A_0^\top A_0.$$

The iid initialization supplies the rank factor:

$$\mathbb E_{A_0}[A_0^\top A_0] = r\sigma_A I.$$

Consequently, the leading learned adapter is:

$$\mathbb E_{A_0}[\gamma_rB_nA_n] = -\eta\gamma_r^2r\sigma_A \sum_{k<n}v_kx_k^\top +\mathcal O_r(\gamma_r^3r).$$

The same coefficient reaches the backward path. For a new upstream gradient $v_n$:

$$\nabla_{x_n}\mathcal L = (\gamma_rB_nA_n)^\top v_n \in \Theta_r(\gamma_r^2r)$$

under the paper assumptions. Forward $m$-th moments scale as:

$$\mathbb E\left[(\gamma_rB_nA_nx_n)^m\right] \in \Theta_r\left((\gamma_r^2r)^m\right).$$

Avoiding both collapse and growth requires:

$$\gamma_r^2r\in\Theta_r(1) \quad\Longleftrightarrow\quad \gamma_r\in\Theta_r\left(\frac{1}{\sqrt r}\right).$$

This is the derivation behind the final square-root rule; it is not a heuristic replacement of one denominator by another.

0x04E One Exponent Separates Collapse, Stability, And Growth

Write a family of rules as:

$$\gamma_r=\alpha r^{-p}.$$

Then the leading learned-signal factor is:

$$\gamma_r^2r = \alpha^2r^{1-2p}.$$

Rule	$p$	Leading factor $\gamma_r^2r$	Asymptotic behavior in this model
LoRA, $\alpha/r$	$1$	$\alpha^2/r$	collapses
rsLoRA, $\alpha/\sqrt r$	$1/2$	$\alpha^2$	rank-stable
weak divisor, $\alpha/r^{1/4}$	$1/4$	$\alpha^2\sqrt r$	grows
over-divisor, $\alpha/r^2$	$2$	$\alpha^2/r^3$	collapses faster

The paper evaluates the $p=1/4$ and $p=2$ alternatives as ablations. Their role is diagnostic: a scale above the stabilized class can become unstable, while a more aggressive divisor can collapse learning faster.

0x04F Exact Rank Sweep: Same Initial Function, Different Learned Signal

The article-local oracle uses a deterministic substitute for the paper’s expectation. For each $r\in\{4,16,64,256\}$, repeat the rows:

$$[1,0],\ [-1,0],\ [0,1],\ [0,-1].$$

Then exactly:

$$A_0^\top A_0=\frac r2I,$$

so there is no sampling noise in the rank comparison. With $\alpha=1$, $\eta=0.125$, $x=[1,-2]^\top$, and $v=[0.5,-1]^\top$, the oracle checks the initial output, both first gradients, one-step dense update, probe output, and input gradient.

Rank	LoRA $\gamma^2r$	rsLoRA $\gamma^2r$	$r^{-1/4}$ rule	$r^{-2}$ rule
4	$0.25$	$1$	$2$	$0.015625$
16	$0.0625$	$1$	$4$	$0.000244140625$
64	$0.015625$	$1$	$8$	$0.000003814697$
256	$0.00390625$	$1$	$16$	$0.000000059605$

The rank-256 to rank-4 ratio is $1/64$ for standard LoRA, $1$ for rsLoRA, $8$ for the weak divisor, and approximately $1/262{,}144$ for the over-divisor. Meanwhile parameter bytes and live branch MACs still grow linearly with rank. Stabilizing the learned signal does not make high rank free.

def one_step_scaled_delta(A, x, output_grad, eta, alpha, rank, p):
    gamma = alpha / rank**p
    B0 = zeros([output_grad.size, rank])

    initial_output = gamma * B0 @ A @ x       # exactly zero
    grad_B = gamma * outer(output_grad, A @ x)
    grad_A = gamma * outer(B0.T @ output_grad, x)  # exactly zero

    B1 = B0 - eta * grad_B
    delta_1 = gamma * B1 @ A
    return initial_output, grad_A, grad_B, delta_1

The checked fixture is an algebra oracle, not a paper-scale training reproduction. Its semantic report hash freezes the exact tensor and scale contract used by this section.

0x04G Initialization Scaling Is Not Adapter Scaling

One alternative is to divide $A_0$ by $\sqrt r$ while retaining another forward scale rule. That changes $A_0^\top A_0$, but it does not apply the same coefficient at every later execution of the learned branch. The rsLoRA paper reports two relevant ablations:

1. scaling only the initialization and using no branch scale equalized an initial gradient comparison but became unstable for the high-rank run;
2. scaling the initialization while retaining $1/r$ branch scaling reproduced the weak high-rank learning behavior.

This is source-reported evidence under the paper’s setup. Mechanically, it also shows why these choices cannot be collapsed:

initialization rule:
    chooses the starting A_0 distribution

adapter scaling rule:
    multiplies every forward and backward use of the branch

PiSSA and LoftQ later in this article own structured initialization. They do not silently replace rsLoRA’s scale contract.

0x04H Paper Evidence And Ablations

The main rsLoRA experiment fine-tunes Llama 2 7B on the first 20,000 OpenOrca examples with AdamW, a constant $5\times10^{-5}$ learning rate, batch size 32, context length 512, and adapters on all linear attention and MLP modules. It sweeps ranks $\{4,8,32,128,512,2048\}$. All results below are source-reported from the pinned v1 paper.

Source: rsLoRA v1, Figure 1. In this experiment, standard-LoRA rank curves converge similarly while higher-rank rsLoRA curves improve. This is evidence for the evaluated configuration, not a universal quality ordering.

Source: rsLoRA v1, Figure 2. Standard-LoRA gradient norms separate by rank; rsLoRA norms remain in a comparable order of magnitude. The plot reports average parameter-gradient norms, not every tensor entry in the theorem.

The appendix checks several alternative explanations: SGD instead of AdamW; GPT-J on GSM8K with Adafactor; attention-only targets; initialization-only scaling; a low-rank learning-rate sweep; and the $p=1/4$ and $p=2$ scale rules. Those ablations broaden the paper’s evidence surface, but they still do not prove that every model, task, rank, or optimizer benefits monotonically from larger $r$.

The paper also reports benign averaged activation moments for both methods and offers architecture, normalization, small-magnitude, and averaging effects as possible explanations. That observation should remain visible: a clean aggregate activation plot does not falsify rank-dependent parameter-gradient collapse, and neither plot alone proves the full theorem inside a Transformer.

0x04I Pinned PEFT Scale, Pattern, Merge, And Hotswap Trace

At PEFT v0.19.1, LoraConfig records use_rslora. During module creation, LoraModel first resolves rank_pattern and alpha_pattern; LoraLayer.update_layer then stores:

rank = rank_pattern.get(module_key, default_rank)
alpha = alpha_pattern.get(module_key, default_alpha)

if use_rslora:
    scaling[adapter_name] = alpha / math.sqrt(rank)
else:
    scaling[adapter_name] = alpha / rank

This order matters. The denominator uses the effective per-module rank, not necessarily the global default. For the checked fixture:

Module	Effective rank	Effective alpha	LoRA scale	rsLoRA scale
q_proj	64	16	0.25	2
v_proj	16	32	2	8
o_proj	4	32	8	16

The live linear path multiplies the low-rank branch by the stored scale. The merge path forms the same scaled delta:

$$\Delta W_{\text{merge}} = (BA)\,\texttt{scaling[adapter]}.$$

Changing the rule only at merge would break live/merged parity. PEFT’s hotswap path also resolves incoming rank and alpha patterns, recomputes $\alpha/\sqrt r$ when use_rslora=True, and updates the module’s stored scale. Therefore use_rslora is artifact and runtime identity, not a disposable training comment.

The same $A/B$ tensors + wrong rule = wrong adapter.

0x04J Conversion, Checkpoint, And Runtime Boundary

PEFT’s initialized-adapter conversion path can double rank when converting a PiSSA-, CorDA-, OLoRA-, or LoRA-GA-style difference into standard LoRA factors. Preserving the scale requires different alpha transformations:

$$\frac{2\alpha}{2r}=\frac{\alpha}{r} \qquad\text{for standard LoRA},$$ $$\frac{\sqrt2\alpha}{\sqrt{2r}} = \frac{\alpha}{\sqrt r} \qquad\text{for rsLoRA}.$$

The pinned code implements exactly those multipliers. It also rejects this rsLoRA conversion when nonempty rank or alpha patterns are combined with the conversion path, rather than pretending a single global adjustment preserves every module-specific scale.

A resumable or serveable rsLoRA artifact therefore needs:

adapter identity
target modules
factor tensors and dtypes
default rank and alpha
per-module rank and alpha patterns
use_rslora scale identity
merge state and base-model identity

M6.5 owns multi-adapter admission, batching, and live hotswap behavior. This section hands it the scale identity and pattern metadata; it does not infer that every runtime supports rsLoRA merely because it supports two LoRA matmuls.

0x04K Failure And Claim Closure

Failure	Plausible symptom	Broken contract
compare ranks while omitting alpha or scale rule	larger rank appears ineffective	rank and learning parameterization were conflated
say rsLoRA has nonzero initial output	zero-$B$ unit test disagrees	stabilization begins in the update path
derive only one power of $\gamma_r$	predicted exponent is wrong	gradient scale and forward scale were collapsed
call $1/\sqrt r$ universally optimal	one benchmark sweep looks favorable	theorem assumptions and feature quality were generalized
scale initialization only	initial norm looks matched	persistent branch scale is a different object
apply global rank in a patterned adapter	some modules have wrong output magnitude	effective per-module rank was ignored
merge $BA$ without stored scale	live and merged outputs differ	scale is part of the operator
hotswap tensors but retain old scale	shapes may still load	incoming adapter identity was not restored
double rank and alpha identically for both rules	converted adapter changes scale	$r$ and $\sqrt r$ denominators transform differently
claim high rank is free under rsLoRA	quality curve improves	parameters, optimizer state, and live MACs still grow with rank

The defensible conclusion is narrow and useful: under the paper’s assumptions, the learned forward and input-gradient moments carry a $\gamma_r^2r$ factor, so $\gamma_r\in\Theta_r(1/\sqrt r)$ is the rank-stable class. Whether the extra rank improves a real task remains an empirical question, and whether that adapter can be served correctly remains a runtime compatibility question.

0x05 LoRA+ And LoRA-FA: Optimizer And Backward-State Contracts

M6.3 established that $A$ and $B$ are asymmetric under the common random-$A$, zero-$B$ initialization. Loop 4 then froze the adapter scale $s$ as part of learning, merge, checkpoint, and runtime identity. Two methods now preserve the low-rank forward form while changing different training objects:

Method	Forward operator	Changed training object	Added experiment identity
LoRA+	unchanged	optimizer groups and factor learning rates	group order, LRs, scheduler, moments
LoRA-FA	unchanged	frozen $A$, trainable $B$, corrected $B$ gradient	frozen $A$, correction, pairing, $B$ moments

Neither method is merely a different rank. LoRA+ changes the trajectory without changing the tensor family. LoRA-FA changes the trainable factor set, backward liveness, and the subspace in which the induced dense update can move.

0x05A LoRA+ Starts From Factor-Asymmetric Feature Updates

For one token representation $z$, define the two LoRA features used by the LoRA+ paper:

$$Z_A=Az, \qquad Z_B=BZ_A=BAz.$$

After an optimizer update, write $A^+=A+\Delta A$ and $B^+=B+\Delta B$. Expanding the product gives

$$\begin{aligned} \Delta Z_B &=B^+A^+z-BAz \\ &=B\Delta A z+\Delta BAz+\Delta B\Delta A z \\ &=\underbrace{B\Delta Z_A}_{\delta_1} +\underbrace{\Delta BZ_A}_{\delta_2} +\underbrace{\Delta B\Delta Z_A}_{\delta_3}. \end{aligned}$$

$\delta_1$ changes the coordinates supplied by $A$ while holding the old $B$ directions fixed. $\delta_2$ changes the directions while holding the old coordinates fixed. $\delta_3$ is their multiplicative interaction. The paper calls learning efficient when both linear contributions remain order one, rather than allowing one factor to become asymptotically inert.

With $\bar g=\partial\mathcal L/\partial Z_B$, the ordinary gradients are

$$\nabla_B\mathcal L=s\,\bar g\,(Az)^\top, \qquad \nabla_A\mathcal L=s\,B^\top\bar g\,z^\top.$$

At $B_0=0$,

$$\nabla_A\mathcal L\big|_{t=0}=0, \qquad \nabla_B\mathcal L\big|_{t=0}=s\,\bar g\,(A_0z)^\top$$

is generally nonzero. The first step is therefore driven by $\eta_B$; changing $\eta_A$ cannot change it. This finite, directly testable result is separate from the infinite-width argument.

0x05B Derive The Width Exponents Before Choosing A Ratio

Let width be $n$, hold rank fixed, and write

$$q=\Theta(n^{\gamma[q]})$$

for the entrywise scale of a quantity. The paper assumes one trainable LoRA layer, stable pretrained features, Adam-type processed gradients with $\Theta(1)$ entries, and an alignment condition on the processed $A$ gradient and layer input. It calls the general theorem “physics level of rigor” and omits technical assumptions; this is not a distribution-free optimizer law.

For the two linear feature contributions to remain order one, its exponent ledger is

$$\gamma[\eta_A]+\gamma[B]+1=0,$$ $$\gamma[\eta_B]+\gamma[Az]=0.$$

The factor-state recurrences reduce to

$$\gamma[B^+]=\max\!\left(\gamma[B],\gamma[\eta_B]\right),$$ $$\gamma[A^+z] =\max\!\left(\gamma[Az],\gamma[\eta_A]+1\right).$$

Combining the efficiency equations gives

$$\gamma[\eta_A]+\gamma[\eta_B]=-1.$$

Equal learning rates assign both exponents $-1/2$. The paper then uses the initialization-specific recurrences to show that this does not keep both linear updates efficient and the factor features stable. Its stable efficient assignment is

$$\boxed{ \eta_A=\Theta(n^{-1}), \qquad \eta_B=\Theta(1) }$$

and therefore

$$\eta_B/\eta_A=\Theta(n).$$

That is an asymptotic class, not a finite value. It does not derive $\lambda=16$, nor say that production code should use embedding width as the literal ratio.

0x05C Finite LoRA+ Is A One-Dimensional Search

The practical method fixes

$$\eta_B=\lambda\eta_A, \qquad \lambda>1,$$

then tunes $\eta_A$. This changes a two-dimensional $(\eta_A,\eta_B)$ sweep into a one-dimensional search while retaining factor asymmetry. The finite $\lambda$ absorbs constants omitted by $\Theta$ notation.

Source: LoRA+, Figure 5 in the pinned source. Each box summarizes $\log_2(\eta_B/\eta_A)$ for the four runs with the best final test loss for one model/task pair. It shows finite optima vary; it does not establish one universal ratio.

The paper reports that its second initialization often favored a ratio near $2^4$ for RoBERTa, standard random-$A$/zero-$B$ initialization often favored roughly $2^2$ to $2^3$, and its Llama experiments roughly $2^1$ to $2^2$. It also reports little benefit on some easier tasks. These are source-reported experiments, not a guarantee or a theorem for $16$.

0x05D Pinned PEFT Uses Four Groups, Not Merely A/B

The pinned create_loraplus_optimizer from PEFT v0.19.1 classifies every trainable parameter:

for name, parameter in model.named_parameters():
    if not parameter.requires_grad:
        continue
    if owning_module_is_peft_embedding(name):
        group = "embedding"
    elif "lora_B" in name or parameter.ndim == 1:
        group = "groupB" if name in decay_parameters else "groupB_no_decay"
    else:
        group = "groupA"

It constructs four groups:

Group	Learning rate	Weight decay	Membership rule
groupA	$\eta_A$	LoRA+ decay	remaining trainable parameters
embedding	dedicated embedding LR	LoRA+ decay	PEFT LoRA embedding parameters
groupB	$\lambda\eta_A$	LoRA+ decay	decayed lora_B and 1-D parameters
groupB_no_decay	$\lambda\eta_A$	$0$	non-decayed lora_B and 1-D parameters

The parameter.ndim == 1 branch makes the high-rate groups broader than the mathematical $B$ factors. A trainable one-dimensional bias or normalization parameter can receive the $B$ learning rate. Reproduction therefore requires more than the phrase “LoRA+ ratio 16.”

The exact fixture uses $\eta_A=10^{-4}$ and $\lambda=16$. It obtains $1.6\times10^{-3}$ for both $B$ groups, places an embedding adapter at $10^{-6}$, sends a trainable non-LoRA bias to groupB_no_decay, and excludes a frozen tensor. For the explicitly named 8-bit optimizer classes, the helper also asks bitsandbytes to keep embedding optimizer state at 32 bits.

0x05E Scheduler And Exact-Resume Identity

If one scheduler multiplies every group by the same factor $c_t$, then

$$\frac{c_t\eta_B}{c_t\eta_A}=\lambda.$$

A scheduler that rewrites groups differently does not preserve the ratio. Exact resume therefore requires:

ordered parameter names
ordered optimizer groups and membership
current LR and weight decay of every group
optimizer moments and step
scheduler class, state, and group mapping
rank, alpha, and Loop 4 scale identity

Restoring identical $A/B$ tensors into equal-LR groups resumes a different training algorithm. The merged operator can be identical at that instant; the future trajectory is not.

0x05F LoRA-FA Freezes A, But That Is Only Half The Method

LoRA-FA keeps the projection-down factor $A$ fixed and trains $B$:

W [d_out,d_in]: frozen
A [r,d_in]:     random, frozen, checkpointed
B [d_out,r]:    usually zero-initialized, trainable

Source: LoRA-FA, Figure 1. The figure shows trainable and frozen tensors. It does not alone prove activation savings or equality with full fine-tuning; those require the following derivations.

Let a flattened token batch be $X\in\mathbb R^{N\times d_{in}}$ and define

$$U=XA^\top\in\mathbb R^{N\times r},$$ $$Y_{adapter}=sUB^\top \in\mathbb R^{N\times d_{out}}.$$

For $G=\partial\mathcal L/\partial Y_{adapter}$, ordinary autograd gives

$$g_B^{raw} =\frac{\partial\mathcal L}{\partial B} =sG^\top U,$$ $$g_A^{raw} =\frac{\partial\mathcal L}{\partial A} =s(GB)^\top X.$$

Freezing $A$ removes $g_A^{raw}$ and its optimizer state. The paper also changes the gradient supplied to the $B$ optimizer. Merely setting A.requires_grad=False implements frozen-factor LoRA, not the complete corrected-gradient LoRA-FA contract.

0x05G Derive The Corrected B Gradient As A Projection

The dense gradient for $W$ under this article’s orientation is

$$g_W=G^\top X \in\mathbb R^{d_{out}\times d_{in}}.$$

If an optimizer applies a matrix $H_B$ to $B$, its corresponding first-order update in dense-weight coordinates is $sH_BA$. LoRA-FA asks for the closest representable dense gradient:

$$H_B^* =\arg\min_H\left\|sHA-g_W\right\|_F^2.$$

Differentiate with respect to $H$:

$$\nabla_H\left\|sHA-g_W\right\|_F^2 =2s(sHA-g_W)A^\top.$$

The normal equation is

$$sH_B^*(AA^\top)=g_WA^\top.$$

When $A$ has full row rank,

$$H_B^* =\frac{1}{s}g_WA^\top(AA^\top)^{-1}.$$

Ordinary autograd already produced

$$g_B^{raw}=sG^\top XA^\top=sg_WA^\top,$$

so the correction can be computed without materializing $g_W$:

$$\boxed{ H_B^* =\frac{1}{s^2} g_B^{raw}(AA^\top)^{-1} }.$$

This is a right preconditioner in rank space. Its induced dense gradient is

$$sH_B^*A =g_WA^\top(AA^\top)^{-1}A =g_WP_A,$$

where

$$P_A=A^\top(AA^\top)^{-1}A.$$

$P_A$ is the orthogonal projector onto the row space of fixed $A$. The unrepresentable component is

$$R=g_W(I-P_A).$$

The corrected update equals the arbitrary dense gradient only when every row of $g_W$ already lies in $\operatorname{row}(A)$. Freezing a random $A$ does not provide universal full-gradient equivalence. Broad paper language about preserving expressivity must be read against this fixed-subspace condition.

The correction remains a descent direction when $AA^\top$ is positive definite because

$$\left\langle g_B^{raw},H_B^*\right\rangle_F =\frac{1}{s^2}\operatorname{tr}\!\left( g_B^{raw}(AA^\top)^{-1}(g_B^{raw})^\top \right)\ge 0.$$

For singular or ill-conditioned $A$, the inverse must be replaced by a pseudoinverse or regularized solve. The result is then tied to that tolerance, solve policy, and dtype.

0x05H Exact Tiny-Tensor Projection Trace

The executable fixture uses

$$A= \begin{bmatrix} 1&0&1\\ 0&2&0 \end{bmatrix}, \qquad AA^\top= \begin{bmatrix} 2&0\\ 0&4 \end{bmatrix},$$

and obtains

$$P_A= \begin{bmatrix} 1/2&0&1/2\\ 0&1&0\\ 1/2&0&1/2 \end{bmatrix}.$$

For its exact $X$ and upstream $G$,

$$g_W= \begin{bmatrix} 1&2.5&0.5\\ -2&-3&5 \end{bmatrix},$$

while the corrected factor update induces

$$g_WP_A= \begin{bmatrix} 0.75&2.5&0.75\\ 1.5&-3&1.5 \end{bmatrix}.$$

The oracle checks exact equality between $sH_B^*A$ and $g_WP_A$, but

$$\left\|g_W(I-P_A)\right\|_F^2=24.625.$$

That nonzero residual is deliberate. A fixture whose dense gradient already lies inside the row space could pass while hiding the method’s main restriction.

0x05I Activation Liveness Is Adapter-Local And Rank-Dependent

The $B$ gradient needs $U:[N,r]$. The $A$ gradient additionally needs $X:[N,d_{in}]$. With $b_a$ bytes per saved activation element:

Method	Trainable adapter parameters	Adapter-local saved activation	Two FP32 optimizer moments
standard LoRA	$r(d_{in}+d_{out})$	$Nb_a(d_{in}+r)$	$8r(d_{in}+d_{out})$ bytes
LoRA-FA	$rd_{out}$	$Nb_ar$	$8rd_{out}$ bytes
LoRA-FB	$rd_{in}$	$Nb_ad_{in}$	$8rd_{in}$ bytes

For the square-layer, 16-bit convention used in the paper, with batch $b$, sequence $s_q$, and width $d$:

$$M_{act}^{LoRA}=2bs_qd+2bs_qr,$$ $$M_{act}^{FA}=2bs_qr, \qquad M_{act}^{FB}=2bs_qd.$$

LoRA-FA removes the need to retain full-width $X$ for the adapter’s $A$ gradient. It does not prove that attention, normalization, residuals, MLP nonlinearities, the base path, or an earlier trainable layer no longer need state. It still retains $U$, so adapter activation grows linearly with rank.

Source: LoRA-FA, Figure 4(a). The paper reports Llama2-7B at rank 64, global batch 8, maximum sequence length 1024, on one 80 GB A800. LoRA-FA is shown at 52.2 GB while compared methods cross the capacity line. This is source-reported system evidence, not a reproduced or hardware-independent memory guarantee.

The forward still computes $U=XA^\top$. Freezing $A$ changes autograd liveness and optimizer work; it does not algebraically delete the projection- down matmul. Any reported kernel compute saving depends on a concrete fused implementation and cannot be inferred from requires_grad=False alone.

0x05J Paper Algorithm And PEFT v0.19.1 Differ

The paper’s Algorithm 1 proposes precomputing

$$K_i=(A_iA_i^\top)^{-1}$$

once because $A_i$ is frozen. The pinned LoraFAOptimizer instead executes this inside each optimizer step:

AA_T = A @ A.T
AA_T_inv = torch.linalg.pinv(AA_T + 1e-8 * eye(rank))
grad_B = (1 / scaling_factor**2) * (grad_B_raw @ AA_T_inv)

Its pinned contract also includes:

1. Every name containing lora_A is frozen.
2. A and B are paired by named-parameter iteration order, not an explicit adapter-key join.
3. Only $B$ receives LoRA-FA moments; non-LoRA trainable parameters follow an ordinary AdamW-style path.
4. The helper receives one global r, lora_alpha, and use_rslora, then computes one scale $\alpha/r$ or $\alpha/\sqrt r$.
5. The correction uses $1/s^2$, so Loop 4’s scale identity is part of the optimizer as well as the forward.
6. The inverse path adds $10^{-8}I$, uses a pseudoinverse, may execute under BF16 autocast, and converts the corrected gradient to the raw dtype.

The global helper arguments do not represent arbitrary per-module rank or alpha patterns. Pairing order, regularization, solve policy, and scale are compatibility and exact-resume state, not incidental code details.

0x05K Frozen Does Not Mean Disposable

LoRA-FA trains only $B$, but frozen $A$ still defines the forward projection $U=XA^\top$, the row-space projector $P_A$, the Gram correction, the merged delta $BA$, and the next optimizer update. A resumable artifact needs:

base-model and adapter identity
frozen A tensor, name, shape, dtype, and order
trainable B tensor, name, shape, dtype, and order
rank, alpha, and standard/rsLoRA scale rule
correction mode, regularization, and solve policy
B optimizer moments and step
non-LoRA trainable parameters and their state

Saving only $B$ is not a smaller equivalent checkpoint. It loses both the operator and the subspace in which the next update is defined.

0x05L Alternatives And Composition Boundaries

LoRA-FB freezes $B$ and trains $A$. Its $A$ gradient still needs full-width $X$, so it does not obtain LoRA-FA’s adapter-local activation advantage. It also needs nonzero frozen $B$: freezing the usual zero-$B$ initialization leaves both adapter output and $A$ gradient at zero.

LoRA+ and LoRA-FA are not naively additive labels. Once $A$ is frozen, $\eta_A$ has no effect and a “LoRA+ ratio” merely renames the effective $B$ learning rate; the two-factor efficiency theorem no longer describes the training algorithm. A meaningful combination needs a new contract and ablation, not string concatenation.

Combination	Required reconciliation
rsLoRA + LoRA-FA	preserve stored $s$ in forward, merge, and $1/s^2$ correction
QLoRA + LoRA-FA	separate quantized-base storage from adapter activation and correction state
AdaLoRA + LoRA-FA	rederive allocation and compaction when $A$ is fixed and rank changes
M6.5 serving	serve the final operator and metadata; do not import optimizer state into request scheduling

0x05M Failure And Claim Closure

Failure	Plausible symptom	Broken contract
call $\lambda=16$ a theorem	one default works	asymptotic width scaling became a finite constant
group only names containing lora_B	optimizer runs	pinned PEFT also assigns trainable 1-D parameters the B rate
restore tensors but rebuild equal-LR groups	checkpoint loads	future trajectory changed
schedule groups differently	initial ratio is right	$\eta_B/\eta_A$ drifts
freeze $A$ but use raw $g_B$	memory drops	corrected LoRA-FA optimizer is absent
claim every dense gradient is recovered	one projection passes	$g_W(I-P_A)$ was ignored
invert singular $AA^\top$	NaNs or failure	full-row-rank assumption broke
discard frozen $A$	$B$ remains	forward, merge, subspace, and correction were lost
claim all activation disappears	adapter ledger shrinks	whole-model liveness was generalized
call LoRA-FA rank-independent	memory seems small	saved $U$ and correction work still grow with rank
say paper and PEFT precompute alike	values look close	PEFT recomputes a regularized pseudoinverse per step
combine LoRA+ and LoRA-FA by name	one LR remains	$A$ is frozen, so the two-factor theorem no longer applies

The useful conclusion is precise. LoRA+ changes how two trainable factors move, making optimizer groups part of exact resume. LoRA-FA removes the adapter’s $A$-gradient liveness and preconditions the $B$ gradient, but the induced dense update remains $g_WP_A$ inside a fixed row space. Both eventually produce $BA$; they are still different training algorithms.

0x06 PiSSA: Initialize From The Principal Subspace

Vanilla LoRA preserves the pretrained function by making one trainable factor zero. That contract also delays one factor’s first update:

vanilla LoRA at step 0:
    A0 = random
    B0 = 0
    s B0 A0 = 0
    gradient_A = 0
    gradient_B is generally nonzero

PiSSA v4 asks a different question:

Can both low-rank factors begin as a meaningful, nonzero part of the pretrained weight while the combined layer still computes exactly the pretrained function?

Its answer changes three identities at once. The original weight is split into a frozen residual and a trainable principal component; both adapter factors start nonzero; and a native checkpoint is now paired with the residual base. This is more than a random-seed choice, even though the live low-rank operator still has the same shapes as LoRA.

0x06A Derive The Principal And Residual Split

Use the M6.3 weight convention:

$$W\in\mathbb{R}^{d_{out}\times d_{in}}, \qquad Y=XW^\top.$$

Let the economy singular value decomposition be

$$W=U\Sigma V^\top,$$

with singular values ordered $\sigma_1\geq\sigma_2\geq\cdots\geq0$. Partition the first $r$ components from the tail:

$$U=[U_r\;U_\perp], \qquad V=[V_r\;V_\perp], \qquad \Sigma= \begin{bmatrix} \Sigma_r&0\\ 0&\Sigma_\perp \end{bmatrix}.$$

Then

$$W= \underbrace{U_r\Sigma_rV_r^\top}_{W_{pri}} + \underbrace{U_\perp\Sigma_\perp V_\perp^\top}_{W_{res}}.$$

The Eckart-Young-Mirsky theorem gives the exact Frobenius contract behind the word “principal”:

$$W_{pri} = \arg\min_{\operatorname{rank}(Z)\leq r}\|W-Z\|_F,$$

and

$$\|W-W_{pri}\|_F^2 = \sum_{i>r}\sigma_i^2.$$

That is a matrix-reconstruction result. It does not prove that the principal subspace is optimal for every downstream loss. The paper’s own limitations request more theory for PiSSA’s advantage, so task-level superiority remains an empirical claim.

0x06B The Adapter Scale Belongs Inside Initialization

The paper writes the PiSSA decomposition with an unscaled product. M6.3 owns a general LoRA branch

$$W_{eff}=W_{base}+sBA, \qquad s>0.$$

Therefore the factors must satisfy

$$sB_0A_0=W_{pri},$$

not merely $B_0A_0=W_{pri}$. Split each singular value symmetrically after dividing by the stored scale:

$$B_0 = U_r\left(\frac{\Sigma_r}{s}\right)^{1/2},$$ $$A_0 = \left(\frac{\Sigma_r}{s}\right)^{1/2}V_r^\top.$$

Then

$$sB_0A_0 = sU_r\frac{\Sigma_r}{s}V_r^\top = W_{pri}.$$

Define the actual frozen base by subtraction:

$$W_{res}=W-sB_0A_0.$$

The initialization identity is now explicit:

$$W_{res}+sB_0A_0=W.$$

The PEFT v0.19.1 implementation does exactly this: it divides retained singular values by the adapter scale, takes square roots, initializes both factors, and subtracts the scaled product from the base weight. The paper’s experiments set $\alpha=r$, so its scale is one; copying the paper factors into a non-unit-scale implementation without this adjustment is wrong.

Source: PiSSA v4, Figure 1. The paper uses an $AB$ naming convention; Transformer System keeps M6.3’s $BA$ convention. The semantic contract is the same: principal factors remain trainable and the residual matrix is frozen.

0x06C Exact Tensor Trace And Spectral Ledger

Use the checked fixture

$$W= \begin{bmatrix} -7.2&9.6&0\\ 5.4&12.8&0\\ 0&0&1 \end{bmatrix}, \qquad \Sigma=\operatorname{diag}(16,9,1),$$

with

$$U= \begin{bmatrix} 0.6&-0.8&0\\ 0.8&0.6&0\\ 0&0&1 \end{bmatrix}, \qquad V= \begin{bmatrix} 0&1&0\\ 1&0&0\\ 0&0&1 \end{bmatrix}.$$

For $r=1$ and $s=4$,

$$B_0= \begin{bmatrix}1.2\\1.6\\0\end{bmatrix}, \qquad A_0= \begin{bmatrix}0&2&0\end{bmatrix}.$$

Therefore

$$sB_0A_0= \begin{bmatrix} 0&9.6&0\\ 0&12.8&0\\ 0&0&0 \end{bmatrix},$$

and

$$W_{res}= \begin{bmatrix} -7.2&0&0\\ 5.4&0&0\\ 0&0&1 \end{bmatrix}.$$

The fixture separates captured energy from residual error:

$$\frac{\|W_{pri}\|_F^2}{\|W\|_F^2} = \frac{16^2}{16^2+9^2+1^2} \approx0.7574,$$

while

$$\|W-W_{pri}\|_F^2=9^2+1^2=82.$$

The machine-checked reconstruction, orthogonality, gradient, dropout, and conversion results live in the M6.4 evidence bundle. This tiny matrix is an algebra oracle, not evidence for paper-scale quality.

0x06D Both Factors Receive A First Gradient

For

$$Y=XW_{res}^\top+sXA^\top B^\top$$

and output gradient

$$G=\frac{\partial L}{\partial Y} \in\mathbb{R}^{N\times d_{out}},$$

the chain rule gives

$$U_A=XA^\top \in\mathbb{R}^{N\times r},$$ $$\frac{\partial L}{\partial B} = sG^\top U_A = sG^\top XA^\top,$$

and

$$\frac{\partial L}{\partial A} = s(GB)^\top X = sB^\top G^\top X.$$

Vanilla zero-$B$ initialization makes $\partial L/\partial A=0$ at the first step. PiSSA generally has $A_0\neq0$ and $B_0\neq0$, so neither gradient is forced to zero.

For the fixture inputs

$$X= \begin{bmatrix}1&2&0\\0&1&1\end{bmatrix}, \qquad G= \begin{bmatrix}1&-1&2\\2&0.5&-1\end{bmatrix},$$

the low-rank feature is

$$XA_0^\top= \begin{bmatrix}4\\2\end{bmatrix},$$

and the checked first gradients are

$$g_B= \begin{bmatrix}32\\-12\\24\end{bmatrix}, \qquad g_A= \begin{bmatrix}-1.6&9.6&12.8\end{bmatrix}.$$

Both are nonzero. This is the precise first-step distinction; it is stronger than saying that PiSSA “starts closer” without naming the computation.

0x06E Principal Initialization Is Not A Permanent Subspace Constraint

Let a finite optimizer step produce

$$B_1=B_0+\Delta B, \qquad A_1=A_0+\Delta A.$$

The effective dense change relative to initialization is

$$\begin{aligned} \Delta W &=s(B_1A_1-B_0A_0)\\ &=s\Delta B A_0+sB_0\Delta A+s\Delta B\Delta A. \end{aligned}$$

The first two terms form the tangent update of the rank-$r$ factorization; the last is the finite-step cross term. Columns of $\Delta B$ need not stay in $\operatorname{col}(U_r)$, and rows of $\Delta A$ need not stay in $\operatorname{row}(V_r^\top)$. PiSSA initializes from principal directions; ordinary factor optimization can rotate away from them.

In the oracle, one gradient step introduces a nonzero third coordinate in $B_1$ and new first and third coordinates in $A_1$. The exact dense update equals tangent plus cross term to machine precision. Calling the method a permanent “principal-only” optimizer would therefore describe the initial state as though it were the entire trajectory.

0x06F Exact SVD, Fast SVD, Dtype, And Function Parity

PiSSA exposes two different questions:

1. Did initialization recover the exact top-$r$ singular subspace?
2. Does the initialized residual-plus-adapter operator reproduce the input weight?

Exact SVD can answer the first under the selected arithmetic. A randomized low-rank SVD gives an approximation whose quality depends on rank, oversampling, iteration count, spectrum, and arithmetic. But if code defines

$$W_{res}:=W-s\widehat{B}_0\widehat{A}_0,$$

then

$$W_{res}+s\widehat{B}_0\widehat{A}_0=W$$

still holds algebraically. Approximate subspace fidelity and initial function identity are different tests.

The paper’s Fast SVD appendix reports a source-specific time/error/loss tradeoff as niter changes. In pinned PEFT:

init_lora_weights = "pissa"
    -> torch.linalg.svd

init_lora_weights = "pissa_niter_N"
    -> torch.svd_lowrank(..., niter=N)

Pinned PEFT accepts source weights in FP32, FP16, or BF16, converts the SVD input to FP32, and casts the residual back to the source dtype. Therefore the parity gate should be dtype-aware:

def pissa_init(weight, rank, scale, niter=None):
    source_dtype = weight.dtype
    assert source_dtype in (torch.float32, torch.float16, torch.bfloat16)
    work = weight.float()

    if niter is None:
        U, S, Vh = torch.linalg.svd(work, full_matrices=False)
        U, S, Vh = U[:, :rank], S[:rank], Vh[:rank, :]
    else:
        U, S, V = torch.svd_lowrank(work, q=rank, niter=niter)
        Vh = V.T

    root = torch.sqrt(S / scale)
    B0 = U * root[None, :]
    A0 = root[:, None] * Vh
    residual = (work - scale * (B0 @ A0)).to(source_dtype)

    reconstructed = residual.float() + scale * (B0 @ A0)
    assert_close(reconstructed, weight.float(), dtype_tolerance(source_dtype))
    return residual, A0, B0

Initialization time, peak workspace, host/device placement, random seed for fast SVD, niter, compute dtype, cast dtype, and reconstruction tolerance are all reproducibility state.

0x06G Training Mode Adds A Dropout Contract

The parity proof assumes the same $X$ reaches the residual and principal paths. If adapter dropout transforms only the low-rank input, then at initialization

$$Y_0 = XW_{res}^\top +s\mathcal{D}(X)A_0^\top B_0^\top.$$

Relative to the pretrained output, the samplewise difference is

$$Y_0-XW^\top = s\left(\mathcal{D}(X)-X\right)A_0^\top B_0^\top.$$

It is generally nonzero for a nonidentity dropout mask. The PiSSA paper’s reported training setup uses lora_dropout = 0 and $\alpha=r$. A framework may allow other values, but “initially the same function” must then be scoped to evaluation mode or to the expectation justified by that dropout rule, not claimed as samplewise training-mode identity.

0x06H Native State And Portable Conversion Are Different Artifacts

After training, let the factors be $B_t,A_t$. A native PiSSA layer computes

$$W_{native}=W_{res}+sB_tA_t.$$

That adapter is not portable by itself to the original $W$. Since

$$W_{res}=W-sB_0A_0,$$

the equivalent update relative to $W$ is

$$\Delta W_{portable} = s(B_tA_t-B_0A_0).$$

Factor it by concatenation:

$$B_\Delta=[B_t\;B_0] \in\mathbb{R}^{d_{out}\times2r},$$ $$A_\Delta= \begin{bmatrix} A_t\\-A_0 \end{bmatrix} \in\mathbb{R}^{2r\times d_{in}},$$

so that

$$sB_\Delta A_\Delta = s(B_tA_t-B_0A_0).$$

The portable adapter has rank at most $2r$, not $r$. Saving only $B_t,A_t$ and applying them to $W$ double-counts the initialized principal component. The oracle’s wrong artifact differs from the native trained weight by squared Frobenius error $256$, exactly the energy of the omitted subtraction in its rank-one principal component.

The PiSSA conversion appendix derives this rank-doubled adapter. Pinned PEFT implements it as subtract_mutated_init, requiring an adapter snapshot taken immediately after initialization. It also preserves the branch scale when rank doubles:

Scale rule	Original	Converted rank	Converted alpha	Scale after conversion
classic LoRA	$\alpha/r$	$2r$	$2\alpha$	$\alpha/r$
rsLoRA	$\alpha/\sqrt r$	$2r$	$\sqrt2\alpha$	$\alpha/\sqrt r$

PEFT v0.19.1 rejects the rsLoRA conversion path when rank or alpha patterns are also active. It also rewrites the converted configuration’s init_lora_weights to ordinary LoRA initialization; otherwise loading the saved initial snapshot would subtract the principal component again.

0x06I Quantized Residuals Break Naive Conversion Equivalence

Quantization is nonlinear. In general,

$$Q(W_{res})+sB_tA_t \neq Q(W)+s(B_tA_t-B_0A_0).$$

Pinned PEFT emits this warning during conversion. A native QPiSSA artifact can pair the quantized residual with trained factors; a portable converted LoRA adapter on the quantized original base can have a numerical gap. This does not turn PiSSA into LoftQ:

Method	Matrix decomposed at initialization	Object optimized
PiSSA	original pretrained $W$	principal factors, with residual frozen
QPiSSA	original $W$, then quantize $W_{res}$	principal factors over quantized residual
LoftQ	quantization residual under a chosen quantizer	coupled quantized base and low-rank compensation

Loop 7 owns LoftQ’s alternating quantization-aware objective. This section only freezes the artifact boundary needed to avoid treating the methods as synonyms.

0x06J What The Paper Figures Establish

Source: PiSSA v4, Figure 2(a). This is a source-reported toy loss-landscape experiment. It visualizes the paper’s optimization hypothesis; it is not a proof that principal directions minimize every task loss.

Source: PiSSA v4, Appendix M. The plot reports one LLaMA-3-8B MetaMathQA setup and supports an early-step comparison under that protocol. It does not establish a universal convergence ordering.

The paper also compares principal, middle, and minor singular components in its appendix. Those experiments motivate the principal choice on the evaluated settings. MiLoRA v3 later studies the opposite design hypothesis: preserve principal components and adapt minor ones. The comparison is empirical and task-dependent; the SVD reconstruction theorem alone cannot choose between their downstream objectives.

0x06K State, Runtime, And Exact-Resume Contract

PiSSA uses the ordinary two-matmul LoRA branch during training, so it does not remove adapter activation state or optimizer moments. Its distinguishing state is initialization and artifact identity:

initialization manifest:
    source checkpoint identity and target module
    rank, alpha, and classic/rsLoRA scale rule
    exact SVD or randomized SVD with niter and seed
    SVD compute dtype, residual storage dtype, device, tolerance
    dropout mode

native training checkpoint:
    residual base identity or residual tensor
    current A and B
    initial A0 and B0 snapshot for portable conversion
    optimizer and scheduler state

portable converted adapter:
    original base identity
    rank 2r factors encoding BtAt - B0A0
    adjusted rank and alpha metadata preserving scale
    ordinary-LoRA initialization metadata

For a merged fixed deployment, materialize $W_{native}$ once and use the dense operator. For live adapter serving, a residual-base PiSSA adapter cannot be mixed casually with adapters that assume the original base. Convert to the original-base delta contract first, or maintain a separate compatible base identity. M6.5 owns registry, batching, and hotswap policy.

0x06L Failure And Claim Closure

Failure	Plausible symptom	Broken contract
omit $s$ from singular roots	reconstruction passes only when $s=1$	scale-aware initialization
call fast SVD exact	function recomposes	subspace fidelity confused with subtraction identity
compare residual after FP16 cast at FP64 tolerance	tiny parity failure	dtype boundary omitted
enable adapter dropout and claim samplewise identity	training output changes at step zero	branch inputs differ
say PiSSA remains in the initial principal subspace	early vectors look aligned	factor gradients can rotate both sides
call principal directions task-optimal by theorem	reconstruction error is minimal	matrix norm theorem generalized to downstream loss
save trained factors against original $W$	model loads but output shifts	initial principal product was not subtracted
convert without initial snapshot	only final factors remain	portable delta is trained minus initial product
keep rank $r$ after portable conversion	shapes look compact	difference of two rank-$r$ products needs up to $2r$
quantize original and residual bases interchangeably	similar file size	nonlinear quantization broke equivalence
call PiSSA and LoftQ the same initializer	both use SVD	original weight and quantization error are different objects
generalize source plots	one setup improves	source-reported experiment became universal claim

The precise conclusion is narrower and stronger than “SVD is a better seed.” PiSSA represents the original layer as a frozen residual plus a nonzero, scale-aware principal adapter; this gives both factors first-step gradients. The factors may rotate during training. Native artifacts require the residual base, while portable original-base adapters encode trained-minus-initial state and can require rank $2r$.

0x07 LoftQ: Quantization-Error Initialization Is A Coupled State Problem

QLoRA and LoftQ can use the same low-bit vocabulary and still initialize two different functions. Let $W\in\mathbb{R}^{d_{out}\times d_{in}}$ be one pretrained matrix. A quantizer does not merely return another dense matrix. It produces representation state

$$C=Q_N(W),$$

such as packed code indices, scales, a codebook or zero points, block metadata, and shape. The matrix consumed by a floating-point execution path is the dequantized value

$$\widehat{Q}=D(C).$$

M8.1 owns how such codes and scales work. This section owns a different question:

Given a fixed low-bit representation family and a rank budget, how should the quantized base and the initial LoRA factors be chosen together so that their initial effective weight remains close to $W$?

Vanilla QLoRA first quantizes $W$ and then installs a zero-output LoRA branch:

$$W_{eff}^{QLoRA}(0)=\widehat{Q}_0+sB_0A_0=\widehat{Q}_0, \qquad B_0A_0=0.$$

Its initial weight error is therefore

$$E_q=W-\widehat{Q}_0.$$

For an input batch $X\in\mathbb{R}^{T\times d_{in}}$, the corresponding output error is

$$\Delta Y = X(W-\widehat{Q}_0)^\top = XE_q^\top.$$

The error is already present before any task gradient reaches the adapter. LoftQ changes that initial condition: it gives the low-rank branch a nonzero job at step zero and, in the full algorithm, lets the chosen quantized base change with that branch.

0x07A Derive The Joint Objective From The Executed Weight

M6.3 defined the repository’s LoRA convention as

$$W_{eff}=\widehat{Q}+sBA,$$

with

$$B\in\mathbb{R}^{d_{out}\times r}, \qquad A\in\mathbb{R}^{r\times d_{in}}, \qquad s>0.$$

The initializer should make this executed weight approximate the original pretrained matrix. That gives the weight-space objective

$$J(C,B,A) = \left\| W-D(C)-sBA \right\|_F^2.$$

Equivalently, if $\mathcal{Q}_N$ denotes all matrices representable by the selected code, scale, grouping, and backend contract,

$$\min_{\widehat{Q}\in\mathcal{Q}_N,\;\operatorname{rank}(BA)\le r} \left\|W-\widehat{Q}-sBA\right\|_F^2.$$

The LoftQ paper writes an unscaled product $AB^\top$. The extra $s$ above is not a new method. It reconciles the paper notation with a framework whose LoRA forward multiplies the branch by $\alpha/r$ or $\alpha/\sqrt{r}$.

This objective is LoRA-fine-tuning-aware in a precise but limited sense. It reserves a rank-$r$ additive component while choosing the low-bit base. It does not include task labels, activations, optimizer dynamics, or serving latency.

0x07B The Two Alternating Blocks Have Different Guarantees

Initialize

$$B_0=0, \qquad A_0=0.$$

At alternating step $t$, hold the previous low-rank product fixed and form

$$Z_t = W-sB_{t-1}A_{t-1}.$$

The quantizer block produces codes and dequantization metadata:

$$C_t=Q_N(Z_t), \qquad \widehat{Q}_t=D(C_t).$$

The ideal block-coordinate update would be

$$\widehat{Q}_t^* \in \arg\min_{\widehat{Q}\in\mathcal{Q}_N} \left\|Z_t-\widehat{Q}\right\|_F^2.$$

But the paper explicitly says its practical $q_N(\cdot)$ is an efficient approximation, not an exact solution of that subproblem. Dynamic absmax scales, block grouping, clipping, lookup tables, and backend rules make the quantizer a discrete algorithm rather than a symbolic nearest-matrix operator.

Now freeze $\widehat{Q}_t$ and define its residual against the original weight:

$$E_t=W-\widehat{Q}_t.$$

Take the economy SVD:

$$E_t = U_t\Sigma_tV_t^\top = \sum_{i=1}^{d} \sigma_{t,i}u_{t,i}v_{t,i}^\top,$$

where $d=\min(d_{out},d_{in})$ and $\sigma_{t,1}\ge\cdots\ge\sigma_{t,d}\ge0$. By Eckart—Young, the exact best rank-$r$ correction for this fixed dequantized base is

$$D_t = (E_t)_r = U_{t,r}\Sigma_{t,r}V_{t,r}^\top.$$

Therefore

$$\min_{\operatorname{rank}(D)\le r} \|E_t-D\|_F^2 = \sum_{i=r+1}^{d}\sigma_{t,i}^2.$$

The two blocks should not be described with the same word “optimal”:

Block	Fixed state	Update	Guarantee
quantizer	$B_{t-1},A_{t-1}$	quantize $W-sB_{t-1}A_{t-1}$	practical, representation-specific approximation
low-rank	$\widehat{Q}_t$	truncated SVD of $W-\widehat{Q}_t$	exact Frobenius-best rank-$r$ matrix

That distinction is why generic convergence claims are unsafe.

0x07C Rank Recovers Spectral Energy, Not Arbitrary Quantization Error

For the first quantized residual $E_1$, define the rank-$r$ recoverable energy fraction

$$\rho_r(E_1) = \frac{\sum_{i=1}^{r}\sigma_i^2} {\sum_{i=1}^{d}\sigma_i^2}.$$

Then the initial quantized-only error is

$$\|E_1\|_F^2 = \sum_{i=1}^{d}\sigma_i^2,$$

whereas a one-step LoftQ correction leaves

$$\|E_1-(E_1)_r\|_F^2 = \sum_{i=r+1}^{d}\sigma_i^2.$$

If the error is concentrated in a few singular directions, a small adapter can remove much of it. If the spectrum is flat, rank $r$ leaves a large floor. A larger parameter count does not by itself prove a useful error geometry.

0x07D Exact Three-By-Three Alternating Trace

The checked Loop 7 oracle uses

$$W= \begin{bmatrix} 1.05 & 3.48 & 3.39\\ -1.38 & 3.91 & -2.50\\ 2.59 & -2.74 & -0.76 \end{bmatrix},$$

rank $r=1$, and a teaching quantizer with normalized codebook

$$\mathcal{C} = \left\{-1,-\frac13,\frac13,1\right\}.$$

Each quantizer call recomputes one matrix-wide absmax scale and chooses the nearest normalized code. This is deliberately small enough to audit; it is not claimed to reproduce bitsandbytes NF4.

At $t=1$, the quantized-only residual has

$$\|W-\widehat{Q}_1\|_F^2 = 5.276988888889.$$

Its rank-one SVD correction captures

$$\rho_1=0.764836830295$$

of that squared Frobenius error. The factors in repository orientation are

$$B_1= \begin{bmatrix} -0.480669894675\\ -0.477533666939\\ 1.244952494712 \end{bmatrix},$$ $$A_1= \begin{bmatrix} 0.876171940883 & 0.827920287789 & 0.745559784238 \end{bmatrix}.$$

The exact SVD tail ledger is

$$\|W-\widehat{Q}_1-B_1A_1\|_F^2 = 1.240953433611.$$

Four complete iterations in this witness give:

$t$	error after quantizer, before new SVD	error after rank-one SVD
1	$5.276989$	$1.240953$
2	$1.574891$	$0.943482$
3	$1.028072$	$0.868030$
4	$0.929800$	$0.855185$

Every SVD update lowers the objective relative to the quantized state from the same iteration. But the $t=2$ quantizer moves the previous $1.240953$ state to $1.574891$ before SVD. Even in a successful trace, the quantizer block is not a descent proof.

The oracle also contains a second valid matrix whose post-SVD objectives are

$$1.089873, \quad 1.451682, \quad 2.078309, \quad 3.147118.$$

That counterexample does not refute the paper’s empirical results. It prevents an invalid theorem: this practical alternating map does not guarantee that every complete iteration lowers the objective for every matrix and quantizer.

0x07E T=0, T=1, Full Alternation, And One-Step Replacement

The iteration count changes artifact identity:

State	Quantized base	Initial branch	What changes
$T=0$ QLoRA baseline	$Q_N(W)$	zero-output LoRA	quantization error is uncompensated
$T=1$ LoftQ	$Q_N(W)$	rank-$r$ SVD of $W-D(Q_N(W))$	base equals QLoRA’s initial quantization
$T>1$ full LoftQ	$Q_N(W-sB_{T-1}A_{T-1})$	SVD residual matched to that final base	both base and adapter change
PEFT replacement utility	an already quantized base	one SVD fit to streamed original weight	base remains fixed

For $T=1$, the paper’s own special-case statement is exact:

$$\widehat{Q}_1=D(Q_N(W)),$$ $$sB_1A_1=(W-\widehat{Q}_1)_r.$$

For $T>1$, loading $A_T,B_T$ on a separately regenerated $Q_N(W)$ is wrong in general because the final base was quantized from $W-sB_{T-1}A_{T-1}$, not from $W$.

0x07F The Adapter Scale Must Match The SVD Product

The paper’s factor split for an unscaled branch is

$$B_t^{paper}=U_{t,r}\Sigma_{t,r}^{1/2},$$ $$A_t^{paper}=\Sigma_{t,r}^{1/2}V_{t,r}^\top,$$

so

$$B_t^{paper}A_t^{paper}=D_t.$$

If the actual forward computes $sBA$, those factors reproduce $D_t$ only when $s=1$. For arbitrary positive $s$, a symmetric scale-aware split is

$$B_t = U_{t,r} \left(\frac{\Sigma_{t,r}}{s}\right)^{1/2},$$ $$A_t = \left(\frac{\Sigma_{t,r}}{s}\right)^{1/2} V_{t,r}^\top.$$

Then

$$sB_tA_t=D_t.$$

In the oracle, using the unscaled SVD factors with a configured forward scale $s=0.5$ leaves squared error

$$2.249962297431,$$

whereas scale-corrected factors recover the best achievable rank-one tail

$$1.240953433611.$$

This is an implementation contract, not a recommendation that every framework must rewrite its initializer. A framework can instead require $s=1$, divide one factor by $s$, or accept a different initial effective weight. It must state which behavior it implements.

0x07G Quantizer State Is More Than A Dense Matrix

An implementation trace should preserve these tensors separately:

full-precision source:
    W                    [d_out, d_in], initializer compute dtype

quantizer working input:
    Z_t = W - s B A      [d_out, d_in], usually floating point

stored low-bit state:
    code indices         packed or backend-native integer representation
    scales / absmax      per tensor, channel, group, or block
    codebook / zero      representation metadata
    shape / layout       unpack and kernel ABI

execution view:
    Qhat_t = dequant(C_t) [d_out, d_in], compute dtype

adapter state:
    A_t                  [r, d_in], floating point
    B_t                  [d_out, r], floating point
    scale s              scalar or adapter metadata

Two artifacts can expose the same dequantized shape and still be incompatible because their group size, packing order, codebook, scales, or backend differ. LoftQ binds its factors to the actual dequantized base produced by that quantizer contract.

The setup workspace is also not free. For one matrix it may hold $W$, the current low-rank product, quantizer working state, $\widehat{Q}_t$, residual $E_t$, and SVD workspace. The persistent training artifact can be small, while initialization still needs a full-precision source tensor and matrix-level SVD workspace. The paper calls this preprocessing cost negligible because matrices can be processed independently and in parallel; that is a source-reported systems claim, not a universal memory or wall-time guarantee.

0x07H Paper Algorithm, Reference Repository, And PEFT Are Three Contracts

The LoftQ v4 paper specifies quantize-then-SVD alternation and stores the final integer weight plus lookup metadata with the final adapters. The pinned reference repository routes its model conversion through PEFT, exposes bits, rank, and iter, saves the modified base and adapter in separate directories, and changes the saved adapter configuration so LoftQ is not applied again on load.

Pinned PEFT v0.19.1 contains two materially different paths:

Contract	loftq_init	replace_lora_weights_loftq
input base	unquantized matrix	already bitsandbytes-quantized PEFT model
original $W$	live source tensor	streamed from a safetensors checkpoint
iterations	configured positive count	one residual SVD
base mutation	yes, assigns returned dequantized LoftQ base	no
bits in pinned code	NF4-like 4-bit or int8 path	bitsandbytes 4-bit only
callback	none	may keep or roll back each module replacement
compute	casts source weight to float32 for initialization	dequantizes base and SVDs float32 residual

The full pinned helper performs:

res = W.float()
for t in range(num_iter):
    Q_state = quantize(res)
    Qhat = dequantize(Q_state)
    E = W.float() - Qhat
    B, A = truncated_svd_factors(E, rank)
    if t + 1 < num_iter:
        res = W.float() - B @ A

return Qhat.cast_as(W), A, B

The replacement utility instead performs:

for each bitsandbytes Linear4bit target:
    W = stream_original_tensor_from_safetensors(name)
    Qhat = dequantize(existing_quantized_module.weight)
    B, A = truncated_svd_factors(W.float() - Qhat, rank)

    install(A, B)
    if callback_rejects(module):
        restore_previous(A, B)

Calling the replacement utility repeatedly is a greedy series of adapter replacements on a fixed low-bit base. It is not equivalent to Algorithm 1, which re-quantizes the base from the previous low-rank state.

The pinned PEFT factor helper returns the unscaled SVD product and does not divide singular roots by the configured LoRA forward scale. Therefore exact paper-objective recomposition in that pinned path requires scale $s=1$ or an explicit scale-aware adjustment. This statement is version-pinned; it is not a claim about all future PEFT releases.

0x07I Training, Dropout, Merge, And Runtime Integration

After initialization, the paper freezes the quantized backbone and optimizes the floating adapter factors. For one target matrix:

def loftq_linear(x, quantized_base_state, A, B, scale, dropout):
    Qhat_x = quantized_linear(x, quantized_base_state)
    adapter_x = dropout(x) @ A.T
    return Qhat_x + scale * (adapter_x @ B.T)

LoftQ does not remove the live LoRA branch. The runtime still owns adapter matmuls, adapter identity, batching constraints, dtype transitions, and K/V state described by M6.3.

The weight-space objective assumes a deterministic additive branch. In training mode with nonidentity adapter dropout,

$$Y_{train} = X\widehat{Q}^\top + s\operatorname{Dropout}(X)A^\top B^\top,$$

which is not samplewise equal to $X(\widehat{Q}+sBA)^\top$. The oracle’s fixed $p=0.5$ mask produces squared output error $15.540252082377$ relative to deterministic reconstruction. The expectation may be scale-correct under inverted dropout; individual training steps are not identical.

Merging also has a representation boundary. In floating point one can form

$$W_{merge}=\widehat{Q}+sBA.$$

But a low-bit serving artifact needs

$$C_{merge}=Q_N(W_{merge}),$$

which introduces a new quantization error and may use different scales from the LoftQ-initialized base. “Mergeable” does not mean bit-exactly mergeable into the same packed codes.

0x07J Full Artifact And Resume ABI

A reproducible initializer manifest should bind:

source:
    original checkpoint identity and tensor name
    orientation / fan-in-fan-out convention

quantizer:
    bit width and codebook
    scale, zero-point, block/group layout
    clipping and rounding rule
    backend/library version and dtype path

low-rank initializer:
    rank r and forward scale s
    alternating step count T
    SVD implementation, dtype, and tie behavior
    target-module set

artifact:
    final quantized base codes and metadata
    matching A_T and B_T
    adapter config and scale
    provenance linking base and adapter

After training begins, exact resume additionally requires optimizer moments, scheduler state, step counters, RNG/dropout state, precision/loss-scaling state, and the same base/adapter pairing. Re-running LoftQ is not resume.

The base and adapter are a coupled initialization artifact. They are not two independent files that may be regenerated or exchanged by shape alone.

A deployment admission check should fail when any of these differ:

def admit_loftq_artifact(base, adapter, manifest):
    assert base.quantizer_fingerprint == manifest.quantizer_fingerprint
    assert base.source_tensor_map == manifest.source_tensor_map
    assert adapter.rank == manifest.rank
    assert adapter.scale == manifest.scale
    assert adapter.target_modules == manifest.target_modules
    assert adapter.initialized_against_base_sha256 == base.sha256

0x07K Frobenius Error Is Not A Task Metric

For any input batch,

$$\|X(W-W_{eff})^\top\|_F \le \|X\|_2\|W-W_{eff}\|_F.$$

Reducing weight Frobenius error therefore tightens a generic output-error upper bound for fixed $X$. It does not rank task loss exactly. Real activation covariance can weight directions very differently, nonlinear layers can amplify or damp errors, and fine-tuning changes the adapter after initialization.

A data-weighted alternative would optimize something like

$$J_X = \|X(W-\widehat{Q}-sBA)^\top\|_F^2,$$

or use a curvature/Fisher proxy. That is a different design choice with a calibration-data contract. LQ-LoRA and later quantization-error methods belong in the comparison taxonomy of Loop 9; they are not silently substituted into LoftQ here.

0x07L Quantize-First And SVD-First Are Different Algorithms

The paper discusses swapping the two updates. Starting from zero states, the primary order is:

quantize W
fit rank-r residual of W - Qhat

The alternative begins:

fit rank-r approximation of W
quantize the remaining residual

In the exact toy fixture, the one-step objectives are:

$$J_{quantize\ first}=1.240953433611,$$ $$J_{SVD\ first}=3.650081878597.$$

This witness establishes non-equivalence, not universal dominance. The paper’s Table 6 reports that its quantize-first method outperformed the swapped order on the selected 2-bit DeBERTaV3 tasks. That comparison remains source-reported.

0x07M What The Paper Figures Establish

Source: LoftQ v4, Figure 2. The paper measures spectral- and Frobenius-norm initialization discrepancy for one randomly selected BART-large weight matrix under uniform and NormalFloat quantization. In this source-local example LoftQ reduces the discrepancy, especially at 2 bits. It does not prove an all-layer or task-quality theorem.

Source: LoftQ v4, Figure 3 left. This is source-reported uniform 2-bit DeBERTaV3-base MNLI accuracy. $T=0$ is QLoRA; $T=1,5,10$ are LoftQ settings. The selected run improves sharply at $T=1$, peaks at $T=5$, then decreases at $T=10$. It is evidence for sensitivity, not monotonicity.

Source: LoftQ v4, Figure 3 right. This source-reported NF4 BART-large result peaks at $T=1$ among the plotted settings and then declines. More alternating steps are neither free nor guaranteed to improve downstream quality.

0x07N PiSSA, QLoRA, And LoftQ Change Different Initial States

Question	QLoRA	PiSSA	LoftQ
base representation	quantized $W$	full-precision spectral residual	quantized residual state matched to adapter
initial branch	zero output	principal component of original $W$	top-r component of quantization residual
quantization required	yes	no	yes
full method changes base	quantization only	subtracts principal component	may re-quantize each alternating step
owned objective	training-memory-efficient adaptation	spectral initialization geometry	joint low-bit base plus low-rank reconstruction

PiSSA and LoftQ both use SVD, but SVD is a tool, not the method identity. PiSSA decomposes the original weight into principal and residual components. LoftQ decomposes the error left by a particular dequantized base. QPiSSA and other hybrids require their own state contract.

0x07O Failure And Claim Closure

Failure	Why it is wrong	Required check
call practical quantization an exact coordinate minimizer	the paper itself labels the update approximate	pin quantizer algorithm and avoid convergence theorem
assume every $T$ lowers error	dynamic discrete quantization can increase the full objective	record per-iteration objective and stop policy
load full-LoftQ factors on fresh $Q_N(W)$	$T>1$ factors are coupled to a modified final base	bind adapter to base fingerprint
ignore LoRA scale	paper factors reconstruct $D_t$, while runtime executes $sBA$	require $s=1$ or scale-aware factors
call PEFT replacement full LoftQ	it keeps the existing base and performs one residual SVD	name path and version explicitly
discard codebook/scales/layout	dequantized shape does not identify quantized state	persist quantizer ABI
claim weight error proves task quality	objective is input-agnostic Frobenius error	measure outputs and downstream tasks
assume dropout preserves initialization per sample	dropout changes branch inputs	separate eval and training-mode tests
call adapter branch free at serving	LoftQ initializes LoRA; it does not erase its matmuls	measure live or requantized-merge path

The Loop 7 publish gate is:

paper objective and repository orientation reconciled
quantizer and SVD guarantees separated
rank-r residual floor derived
exact alternating and nonmonotonic witnesses checked
T=0, T=1, T>1, and replacement paths separated
LoRA scale included in reconstruction
paper, reference repo, and pinned PEFT behavior distinguished
quantized base, adapter, and quantizer metadata bound as one artifact
dropout, merge, runtime, and resume boundaries explicit
paper figures labeled source-reported

0x08 DoRA: Magnitude-Direction Coordinates Change Forward, Backward, And Merge

LoRA’s additive rule is deliberately simple:

$$W_{LoRA}=W_0+sBA.$$

The same low-rank product must account simultaneously for changes in each weight vector’s length and orientation. The DoRA paper v6 asks whether those two jobs should have separate coordinates. Its motivation draws on weight normalization, but the adaptation contract is different: DoRA starts from a pretrained weight, freezes its dense directional base, makes a vector of magnitudes trainable, and applies LoRA to the directional update.

That sentence hides four contracts that cannot be compressed into “LoRA plus a norm”:

mathematical contract:
    which weight vectors are normalized, and along which tensor axis?

optimization contract:
    does backward differentiate through the normalization denominator?

runtime contract:
    is the adapter live, dropped out, cached, quantized, or merged?

artifact contract:
    which magnitude, factor, base, scale, cache, and resume states are required?

This section owns those contracts for a linear layer. It does not claim that DoRA is merely a larger-rank LoRA, that the learned object named magnitude remains nonnegative, or that a live DoRA adapter has the same serving cost as a merged dense weight.

Source: DoRA v6, Figure 1. The figure shows the paper’s column-vector parameterization and the low-rank directional branch. It does not specify a framework’s storage orientation, detached-norm backward, dropout path, cache policy, or unmerge preconditions.

0x08A Paper Columns And PyTorch Rows Are Transpose-Equivalent, Not Axis-Identical

The paper writes

$$W^{paper}\in\mathbb{R}^{d\times k}$$

and treats each column as one weight vector. With

$$c_j=\left\|v_j\right\|_2, \qquad q_j=\frac{v_j}{c_j},$$

the decomposition is

$$W^{paper} = Q\operatorname{diag}(m), \qquad Q= \left[ \frac{v_1}{\|v_1\|_2},\ldots, \frac{v_k}{\|v_k\|_2} \right].$$

The paper abbreviates this as

$$W^{paper}=m\odot\frac{V}{\|V\|_c},$$

where the broadcasting is over columns.

PyTorch stores an ordinary Linear.weight as

$$W_0\in\mathbb{R}^{d_{out}\times d_{in}}$$

and executes row-input batches as

$$Y=XW_0^\top+b, \qquad X\in\mathbb{R}^{T\times d_{in}}.$$

Each output neuron is therefore one row of weight. In that orientation, the transpose-equivalent DoRA vector norm is

$$c_i=\left\|u_i\right\|_2, \qquad i\in\{1,\ldots,d_{out}\},$$

which is torch.linalg.norm(U, dim=1), not dim=0. The magnitude state is

$$m\in\mathbb{R}^{d_{out}}.$$

The conversion ledger is:

Contract	Stored matrix	Vector being normalized	Norm axis	Magnitude length
DoRA paper	$W^{paper}[d,k]$	each column	paper column axis	$k$
PyTorch Linear	$W_0[d_{out},d_{in}]$	each output row	dim=1	$d_{out}$
relation	$W^{paper}=W_0^\top$	paper column = PyTorch row after transpose	transpose conversion	same logical vectors

Saying only “column norm” while showing a PyTorch-shaped tensor is an implementation bug waiting to happen. The Loop 8 oracle computes both orientations and obtains zero maximum difference after transposing back.

0x08B Derive The Scale-Aware Forward From The Executed Weight

Keep the repository’s vanilla-LoRA factor convention:

$$A\in\mathbb{R}^{r\times d_{in}}, \qquad B\in\mathbb{R}^{d_{out}\times r}, \qquad \Delta=BA.$$

The framework scale is part of the directional state:

$$U=W_0+s\Delta=W_0+sBA.$$

For every output row, define

$$c_i=\|u_i\|_2, \qquad q_i=\frac{u_i}{c_i}, \qquad f_i=\frac{m_i}{c_i}.$$

Stacking rows gives

$$Q=\operatorname{diag}(c)^{-1}U$$

and the effective DoRA weight is

$$W_{DoRA} = \operatorname{diag}(m)Q = \operatorname{diag}\!\left(\frac{m}{c}\right)U = \operatorname{diag}(f)(W_0+sBA).$$

The row-input linear layer therefore executes

$$Y = XW_{DoRA}^{\top}+b.$$

This derivation answers two common ambiguities:

1. $s$ must be inside the norm because the normalized object is the actual directional weight $W_0+sBA$.
2. $m$ is one scalar per normalized weight vector, not a rank-$r$ object.

The trainable-parameter ledger for one PyTorch linear is

$$P_{DoRA} = r(d_{in}+d_{out})+d_{out},$$

versus

$$P_{LoRA}=r(d_{in}+d_{out}).$$

The extra parameter count is small relative to $W_0$, but the forward is not just two thin adapter matmuls: it also needs the current dense directional norm and output-wise factor $f$ unless those values are safely cached or the adapter is merged.

def dora_linear_weight(W0, A, B, magnitude, scale):
    # PyTorch Linear.weight convention: [d_out, d_in].
    U = W0 + scale * (B @ A)
    c = torch.linalg.vector_norm(U, dim=1)  # [d_out]
    if torch.any(c == 0):
        raise ValueError("zero-norm DoRA direction row")
    f = magnitude / c                      # [d_out]
    return f[:, None] * U

The pinned PEFT v0.19.1 path does not clamp c with an epsilon. Adding an epsilon may be a defensible alternative design, but it changes forward values, gradients, merge parity, and the artifact contract. It must not be silently attributed to that implementation.

0x08C Initialization Preserves The Function But Opens Two First-Step Paths

At installation, DoRA takes the pretrained row norms:

$$m_0=\operatorname{rowNorm}(W_0),$$

and uses an ordinary no-op LoRA factor initialization, conventionally

$$B_0=0,$$

with $A_0$ nonzero. Then

$$U_0=W_0+sB_0A_0=W_0,$$ $$f_0=\frac{m_0}{\operatorname{rowNorm}(W_0)}=\mathbf{1},$$

so

$$W_{DoRA}(0)=W_0.$$

This is exact function preservation in the chosen dtype when every row norm is finite and nonzero. It does not imply zero gradients. As the next derivation shows, both the magnitude vector and $B$ can receive first-step gradients, while

$$\nabla_A\mathcal{L}=sB_0^\top\nabla_U\mathcal{L}=0$$

under the standard $B_0=0$ convention. DoRA therefore preserves vanilla LoRA’s first-step factor asymmetry while adding a separate first-step magnitude path.

0x08D The Full Normalization Jacobian Projects Direction Gradients Onto A Tangent Space

The matrix formula is easiest to audit one output vector at a time. Drop the row index and let

$$c=\|u\|_2, \qquad q=\frac{u}{c}, \qquad w=mq.$$

For an infinitesimal perturbation $du$,

$$dc = d\sqrt{u^\top u} = \frac{u^\top du}{\|u\|_2} = q^\top du.$$

Differentiate the normalized vector:

$$dq = d\left(\frac{u}{c}\right) = \frac{du}{c}-\frac{u}{c^2}dc.$$

Substitute $u=cq$ and $dc=q^\top du$:

$$dq = \frac{1}{c} \left(I-qq^\top\right)du.$$

Now let

$$g=\nabla_w\mathcal{L}.$$

Because

$$dw=q\,dm+m\,dq,$$

the loss differential is

$$d\mathcal{L} = g^\top q\,dm + g^\top \frac{m}{c}(I-qq^\top)du.$$

Therefore

$$\boxed{ \nabla_m\mathcal{L}=q^\top g }$$

and

$$\boxed{ \nabla_u\mathcal{L} = \frac{m}{c} \left(I-qq^\top\right)g }.$$

The projector removes the radial component:

$$q^\top\nabla_u\mathcal{L}=0.$$

Thus the directional update is tangent to the unit sphere under the full normalization Jacobian. This is the precise optimization geometry behind the paper’s discussion of projected and rescaled gradients. The paper’s statements about improved conditioning and stability are theoretical interpretation plus source-reported experiments; they are not a theorem that DoRA dominates LoRA on every objective.

The factor chain rule is still required. Since

$$U=W_0+sBA,$$

and $W_0$ is frozen,

$$\boxed{ \nabla_B\mathcal{L} = s\nabla_U\mathcal{L}A^\top }$$

and

$$\boxed{ \nabla_A\mathcal{L} = sB^\top\nabla_U\mathcal{L} }.$$

def dora_full_backward(U, magnitude, upstream_G):
    c = vector_norm(U, dim=1)             # [d_out]
    q = U / c[:, None]                    # [d_out, d_in]
    radial = (q * upstream_G).sum(dim=1)  # q_i dot g_i
    grad_m = radial
    grad_U = (magnitude / c)[:, None] * (
        upstream_G - q * radial[:, None]
    )
    assert allclose((q * grad_U).sum(dim=1), 0)
    return grad_m, grad_U

0x08E Detaching The Norm Preserves Forward Values, Not The Exact Gradient

The paper also proposes a lower-memory implementation that treats the current norm $c=\|u\|_2$ as a constant in backward. The forward still evaluates the same

$$w=m\frac{u}{c},$$

but the derivative no longer includes $dc/du$. Its gradients are

$$\nabla_m\mathcal{L}=q^\top g$$

and

$$\boxed{ \nabla_u^{detach}\mathcal{L} = \frac{m}{c}g }.$$

The difference is exactly the removed radial correction:

$$\nabla_u^{detach}\mathcal{L} - \nabla_u^{full}\mathcal{L} = \frac{m}{c}q(q^\top g).$$

Consequently,

$$q^\top\nabla_u^{detach}\mathcal{L} = \frac{m}{c}q^\top g,$$

which is generally nonzero. Calling this detached rule “the exact gradient of the normalized parameterization” would be wrong. It is an alternative backward contract with the same forward values.

The DoRA paper reports that this modification reduced training memory by about 24.4% in one LLaMA setup and 12.4% in one VL-BART setup with negligible or no reported accuracy change under those experiments. Those are source-reported measurements, not universal memory ratios. PEFT v0.19.1 implements the detached-norm path for its DoRA variants.

0x08F Exact Three-By-Three Forward And Backward Trace

The checked Loop 8 oracle uses

$$W_0= \begin{bmatrix} 3&4&0\\ 1&-2&2\\ 2&1&2 \end{bmatrix}, \qquad s=0.5,$$ $$A= \begin{bmatrix} 0.2&-0.1&0.3\\ -0.4&0.2&0.1 \end{bmatrix},$$ $$B= \begin{bmatrix} 0.1&-0.2\\ 0.3&-0.2\\ -0.1&0.4 \end{bmatrix}, \qquad m= \begin{bmatrix} 5.5&2.7&-3.2 \end{bmatrix}.$$

The negative third entry is intentional. The paper interprets $m$ as a magnitude, but PEFT stores an unconstrained trainable parameter initialized from positive norms. Without a positivity parameterization, optimizer updates do not mathematically guarantee $m_i\ge0$.

The low-rank product is

$$BA= \begin{bmatrix} 0.10&-0.05&0.01\\ 0.14&-0.07&0.07\\ -0.18&0.09&0.01 \end{bmatrix}.$$

Therefore

$$U=W_0+0.5BA = \begin{bmatrix} 3.05&3.975&0.005\\ 1.07&-2.035&2.035\\ 1.91&1.045&2.005 \end{bmatrix}.$$

Its row norms and DoRA factors are

$$c= \begin{bmatrix} 5.010304381971&3.070398996873&2.959755057433 \end{bmatrix},$$ $$f=\frac{m}{c} = \begin{bmatrix} 1.097737698291&0.879364539511&-1.081170548882 \end{bmatrix}.$$

The merged effective weight is

$$W_{DoRA}= \begin{bmatrix} 3.348099979786&4.363507350705&0.005488688491\\ 0.940920057277&-1.789506837905&1.789506837905\\ -2.065035748364&-1.129823223581&-2.167746950508 \end{bmatrix}.$$

For the oracle’s two-row input, decomposed live execution and dense merged execution both produce

$$Y= \begin{bmatrix} 1.177323681417&5.414687152038&-5.835618037589\\ 3.198533218394&-3.177803604657&-0.171635824635 \end{bmatrix},$$

with zero maximum difference at the checked precision.

For a fixed upstream weight gradient $G$, the exact full-normalization oracle obtains

$$\nabla_m\mathcal{L} = \begin{bmatrix} 0.188311912425&-0.404507688175&-0.201030148933 \end{bmatrix}$$

and

$$\nabla_U\mathcal{L} = \begin{bmatrix} 0.642578303506&-0.493323404260&0.219341247715\\ -0.227784924235&0.203924873172&0.323693850534\\ -0.356493839379&-0.725441291337&0.717700440229 \end{bmatrix}.$$

Central finite differences agree within

$$1.61\times10^{-10}$$

for $U$ and

$$1.50\times10^{-10}$$

for $m$. The checked tangent products are exactly zero after report rounding. The detached-norm gradient differs from the full gradient by Frobenius norm

$$0.465297169008,$$

and has nonzero radial components

$$\begin{bmatrix} 0.206717085306&-0.355709716941&0.217347876463 \end{bmatrix}.$$

At no-op initialization $B_0=0$ and

$$m_0=\begin{bmatrix}5&3&3\end{bmatrix},$$

the oracle verifies

effective-weight error: 0
grad(A):                exactly zero
grad(B):                nonzero
grad(m):                nonzero

This trace checks operator algebra and gradient contracts. It does not reproduce the paper’s training runs.

0x08G Pinned PEFT v0.19.1 Executes A Decomposed Correction

The official DoRA repository and modern PEFT are related implementations, not one timeless code path. For a versioned framework claim, this article pins PEFT v0.19.1 at commit ba6a19060d6ab54a87538a6e77e3e4d5a907375b.

Its DoraLinearLayer does the following for a linear layer:

1. dequantizes the base module weight when necessary;
2. obtains a dense LoRA weight, using identity-through-$A$/$B$ evaluation rather than direct B @ A in one path because of the pinned code’s FSDP compatibility concern;
3. computes norm(weight + scaling * lora_weight, dim=1);
4. detaches the LoRA weight for the norm and detaches the resulting norm;
5. forms $f=m/c$;
6. returns a correction to be added to the already-computed base result.

Ignoring bias for one line, the correction is

$$Y_{corr} = (f-1)\odot Y_{base} + f\odot sY_{LoRA}.$$

The wrapper then adds it to $Y_{base}$:

$$Y = Y_{base}+Y_{corr} = f\odot(Y_{base}+sY_{LoRA}).$$

With bias, the pinned path subtracts the base bias from the reused result inside the DoRA correction, so the outer linear bias remains exactly once.

def peft_v0191_dora_eval(x, base_layer, lora_A, lora_B, magnitude, scale):
    base_result = base_layer(x)
    lora_weight = materialize_lora_weight_for_norm(lora_A, lora_B)
    base_weight = dequantize_module_weight(base_layer)
    c = row_norm(base_weight + scale * stop_gradient(lora_weight))
    f = magnitude / stop_gradient(c)
    lora_result = lora_B(lora_A(x))
    base_no_bias = remove_bias_once(base_result, base_layer.bias)
    correction = (f - 1) * base_no_bias + f * scale * lora_result
    return base_result + correction

The implementation’s optional DoRA cache is disabled by default. In this version it is an ad hoc cache, active only under explicit enablement, eval mode, and an adapter name; entering training clears it. It is not state_dict checkpoint state. The code also uses ephemeral GPU offload in DoRA initialization; that flag should not be generalized into a claim that every live forward is automatically offloaded.

0x08H Adapter Dropout Changes The Executed Training Graph

When dropout is identity or the module is in eval mode, PEFT can reuse the already-computed base result. During training with nontrivial adapter dropout, the DoraLinearVariant forward instead applies dropout to $X$ and asks the DoRA layer to recompute the base linear map on that dropped input.

Let $D(X)$ be the sampled, scaled dropout input and omit bias. The total pinned training path is

$$\boxed{ Y_{train} = XW_0^\top + (f-1)\odot D(X)W_0^\top + f\odot sD(X)(BA)^\top }.$$

This is not, sample by sample,

$$XW_{DoRA}^\top.$$

At no-op initialization, however, $f_0=1$ and $B_0A_0=0$, so the dropout terms cancel exactly:

$$Y_{train}(0)=XW_0^\top.$$

The oracle checks both facts: trained dropout differs from merged eval, while initial dropout preserves the base output. A weight-space parity test that ignores training dropout is therefore insufficient.

0x08I Merge Is A State Transition; Exact Unmerge Needs Cached Pre-Merge Geometry

Define the already-scaled delta

$$\Delta=sBA,$$

the pre-merge directional norm

$$c=\operatorname{rowNorm}(W_0+\Delta),$$

and

$$f=\frac{m}{c}.$$

The merged weight is

$$\boxed{ W_{merged}=\operatorname{diag}(f)(W_0+\Delta) }.$$

This gives ordinary dense execution for one fixed adapter and is the condition under which a no-additional-inference-branch claim is meaningful.

But $c$ cannot generally be recomputed from $W_{merged}$ because normalization and magnitude scaling have changed the matrix. The pinned merge_safe, merge_unsafe, and unmerge paths cache the pre-merge directional norm and recover

$$\boxed{ W_0 = \operatorname{diag}(f)^{-1}W_{merged}-\Delta }.$$

Exact invertibility requires:

cached c corresponds to the same W0, delta, adapter, scale, and merge order
m and A/B remain unchanged while the adapter is merged
every f_i is finite and nonzero
merge and unmerge use compatible orientation and dtype rules
stacked adapter merges unwind in the framework's required order

If $m_i=0$, then $f_i=0$: merge can produce a finite zero row, while unmerge’s division by $f_i$ is undefined. PEFT’s safe merge checks whether the merged weight is finite, but that check alone does not prove later unmerge invertibility. A production wrapper should reject zero/nonfinite direction norms and zero/nonfinite DoRA factors before mutating the base.

def merge_dora(W0, delta_scaled, magnitude):
    c = row_norm(W0 + delta_scaled)
    require_all_finite_nonzero(c)
    f = magnitude / c
    require_all_finite_nonzero(f)
    cache = {"premerge_direction_norm": c}
    return f[:, None] * (W0 + delta_scaled), cache

def unmerge_dora(Wmerged, delta_scaled, magnitude, cache):
    c = cache["premerge_direction_norm"]
    f = magnitude / c
    require_all_finite_nonzero(f)
    return Wmerged / f[:, None] - delta_scaled

The Loop 8 oracle recovers $W_0$ with zero maximum error for its checked fixture. That is an operator test under explicit preconditions, not proof that every dtype, quantized base, or stacked-adapter path is lossless.

0x08J State, Checkpoint, And Runtime ABI

The minimum portable DoRA adapter is larger than a vanilla-LoRA factor pair:

portable adapter state:
    A[r, d_in]
    B[d_out, r]
    magnitude m[d_out]
    rank, alpha, and exact scale rule
    target module names and weight orientation
    base checkpoint identity and expected dtype/backend

exact training resume adds:
    optimizer moments for A, B, and m
    scheduler state and global step
    gradient-scaler / precision state
    RNG and data-order state
    distributed sharding and ownership metadata

runtime-only state:
    optional eval norm / dense-delta cache
    merge-unmerge cache keyed by adapter and merge order

The runtime cache is neither sufficient nor appropriate as a portable training checkpoint. Conversely, omitting $m$ from an adapter export silently turns the artifact into a different operator even if $A$ and $B$ load without shape errors.

For $N$ adapted linear modules, trainable adapter storage is

$$M_{trainable} = b_{param} \sum_{\ell=1}^{N} \left[ r_\ell(d_{in,\ell}+d_{out,\ell})+d_{out,\ell} \right].$$

Optimizer state multiplies that trainable ledger according to optimizer and precision. Live PEFT may additionally dequantize the base for norm evaluation, materialize a dense $BA$-equivalent tensor for the norm path, retain $d_{out}$-length norm and factor vectors, and execute the ordinary LoRA branch. Those temporary and compute costs do not appear in the adapter file size.

M6.5 owns adapter registries, request batching, hotswap, and admission policy. The pinned PEFT linear path rejects per-sample adapter_names when a DoRA adapter is enabled; this is a versioned integration fact, not a universal DoRA limitation.

0x08K Design Alternatives Must Name Which Contract Changes

DoRA admits several implementation choices. They are not interchangeable optimizations:

Choice	Forward effect	Backward effect	Runtime/artifact consequence
paper column vs PyTorch row orientation	transpose-equivalent if converted correctly	corresponding axis conversion	store orientation explicitly
full norm Jacobian	exact normalized parameterization	tangent projection present	larger backward graph
detached norm	same forward values	radial projection removal absent	paper/PEFT memory-saving path
epsilon/clamped norm	changes values near small norms	changes gradients near clamp	epsilon becomes ABI/config state
unconstrained $m$	permits signed scale	ordinary scalar gradient	unmerge must reject $f=0$
positive parameterization of $m$	enforces magnitude semantics	chain rule through softplus/exp	different checkpoint and optimizer geometry
live adapter	adapter remains selectable	training possible	norm/factor/LoRA runtime work remains
cached live eval	same values only if invalidation is correct	eval-only	cache identity and invalidation state
merged adapter	one dense weight	not a training-dropout path	no live hotswap; unmerge cache required
quantized base / QDoRA	dequantization and quantizer state enter norm	backend-specific	crosses Loop 2 and M8 contracts

The paper’s QDoRA experiments motivate composition with quantized backbones, but generic quantized-number representation remains M8 work and QLoRA’s frozen low-bit training state remains Loop 2. This section only states the magnitude-direction compatibility boundary.

0x08L Paper, Reference Repository, And PEFT v0.19.1 Are Three Contracts

Surface	Paper v6	Official NVlabs reference repository	PEFT v0.19.1
orientation in explanation/code	columns in $W[d,k]$	PyTorch rows in implementation	PyTorch rows, with fan_in_fan_out transpose helper
magnitude initialization	pretrained vector norms	weight_m_wdecomp copied from row norms	trainable DoraLinearLayer.weight initialized from current row norms
norm backward	derives full projector; proposes detach alternative	dora_simple selects detached versus full-style path	detached norm in the integrated DoRA forward
scale inside norm	paper notation omits framework-specific detail	implementation-specific LoRA scaling	explicitly includes current adapter scaling
quantized initialization	QDoRA evaluated	repository-specific paths	dequantizes module weight; pinned init special-cases Linear4bit copy
live cache	not an algorithm theorem	implementation-specific	optional ad hoc cache, disabled by default
dropout	experiment/training configuration	reference code behavior	explicit dropped-input recomputation path
merge/unmerge	merged inference claim	reference integration	cached pre-merge norm required for exact unmerge
mixed adapter names	not a paper contract	not generalized here	rejected for DoRA by pinned linear wrapper
supported module classes	method-level description	repository task integrations	code has version-specific linear, embedding, and convolution variants; oracle is linear-only

The official DoRA repository at the pinned commit uses a distinct PEFT fork with weight_m_wdecomp and a dora_simple switch. The modern PEFT implementation uses integrated LoRA variant hooks. Neither code surface should be quoted as if it were the other, and neither should be generalized beyond its pinned revision.

0x08M What The Source-Reported Figures Establish

Source: DoRA v6, weight-decomposition analysis figure for query projections in the paper’s VL-BART case study. Each panel reports layer/checkpoint pairs under that protocol. The differing fitted slopes motivate the separate-coordinate hypothesis; they do not prove that a negative slope is universally optimal or that the three methods used equal budgets.

The paper defines magnitude change as an average absolute difference of vector norms and direction change from cosine distance. Those are diagnostics over merged weights, not training objectives. A method can move to a different point in this plot because of target modules, rank, scale, optimizer, data, or checkpoint timing as well as parameterization.

Source: DoRA v6, layer-wise query projection comparison in the paper’s LLaMA2-7B commonsense-reasoning case study. The plotted gaps are source-reported observations for that setup. They do not establish a universal update-size ordering or replace task, runtime, and ablation measurements.

0x08N Why This Is Not Merely Higher-Rank LoRA

Increasing $r$ expands the set of additive matrices $BA$. DoRA changes the map from trainable state to effective weight:

LoRA:
    effective W = W0 + s B A
    factor gradient sees the ordinary additive weight gradient

DoRA:
    effective W = diag(m / row_norm(W0 + s B A)) (W0 + s B A)
    trainable m owns vector scale
    factors act through a normalized directional coordinate
    backward may use full tangent projection or detached-norm approximation

Rank and parameterization are independent controls. A higher-rank LoRA may be more expressive as an additive delta; that does not make its gradients, magnitude state, dropout graph, merge cache, or runtime path equal to DoRA.

0x08O Failure And Claim Closure

Failure	Plausible symptom	Broken contract
normalize dim=0 on PyTorch Linear.weight	shapes broadcast, output differs	paper columns confused with repository rows
omit $s$ inside $U$ and its norm	parity holds only for $s=1$	executed directional weight changed
add an undocumented epsilon	rare small-norm drift	operator and merge ABI changed
call detached norm the exact Jacobian	forward tests pass	tangent projection silently removed
initialize $m$ from the wrong axis	no-op test fails	magnitude does not reconstruct $W_0$
save $A/B$ but omit $m$	checkpoint loads partially	adapter operator changed
assume learned $m$ stays positive	training seems normal	unconstrained parameter can cross zero
merge with $f_i=0$	finite zero row	later unmerge divides by zero
unmerge without cached pre-merge norm	approximate-looking recovery	normalization is not algebraically inverted
mutate $A/B/m$ while merged	unmerge returns wrong base	cache and adapter generation diverged
treat training dropout as one merged weight	eval parity passes	sampled training graph omitted
call live DoRA zero-overhead	adapter file is small	dense norm and branch work omitted
use runtime norm cache as resume state	eval is fast after warmup	optimizer/RNG/training state missing
enable per-sample adapter names in pinned PEFT	request batch fails	versioned mixed-batch restriction ignored
generalize paper plots	one setup resembles FT	source-reported evidence made universal
call QDoRA generic quantization ownership	methods compose	M8 representation boundary collapsed

The precise conclusion is broader than “DoRA learns magnitude separately.” DoRA reparameterizes each weight vector as a learned scalar times a normalized direction, sends the low-rank factors through that direction, and thereby changes both the factor gradients and the runtime state machine. Full and detached normalization have identical forward values but different backward operators. A fixed merge can recover ordinary dense inference, while exact unmerge requires cached pre-merge geometry and nonzero factors.

0x09 Composition Is A Contract Vector, Not A Compatibility Checkbox

The preceding loops deliberately changed one object at a time. Real training recipes rarely remain that clean. A run may use a 4-bit base, rsLoRA scaling, PiSSA initialization, DoRA coordinates, and a factor-specific optimizer in one configuration. The tempting conclusion is that independently valid methods are automatically valid when their switches are enabled together.

That conclusion is false.

Two methods can have compatible tensor shapes while disagreeing about which base weight exists, which scale is inside the update, which parameters receive gradients, which optimizer owns them, or what must be saved. A framework may accept both flags yet ignore one. It may execute a forward path for which no paper derives the backward or export contract. It may also reject a combination before constructing a layer.

The owner question for this loop is therefore:

When several LoRA variants are stacked, which exact algebra, configuration, execution, checkpoint, merge, runtime, and evidence contracts remain valid?

This section answers for the operators frozen in Loops 2 through 8 and for Hugging Face PEFT v0.19.1. It does not claim that another PEFT version, another backend, or every layer type behaves identically.

0x09A A Stack Must Name Every Object It Changes

Represent one variant stack as

$$\mathcal{S} = (R_{\text{base}}, A_{\text{rank}}, s, I, P, O, C_{\text{artifact}}, N_{\text{adapters}}),$$

where:

Axis	Question	Examples in this article
$R_{\text{base}}$	what represents the frozen base?	dense floating tensor, NF4 payload plus metadata
$A_{\text{rank}}$	who owns rank and its schedule?	fixed rank, AdaLoRA budget and ranknum
$s$	how is the low-rank branch scaled?	$\alpha/r$, $\alpha/\sqrt r$
$I$	what operator creates the initial base and factors?	zero-$B$, PiSSA, LoftQ
$P$	what coordinates define the effective weight?	additive LoRA, normalized DoRA
$O$	who owns factor gradients and optimizer groups?	AdamW, LoRA+, LoRA-FA
$C_{\text{artifact}}$	native pair or converted/exported state?	coupled residual base, portable LoRA conversion
$N_{\text{adapters}}$	how many initialized adapters mutate or share a base?	one, multiple

This tuple prevents an acronym such as “QDoRA” from hiding two distinct operators. The quantized base controls how $W_q$ becomes a compute tensor; DoRA controls how that compute tensor and $BA$ become a normalized effective weight. Neither name alone defines the optimizer, conversion path, or serving artifact.

0x09B Compatibility Has Eight Independent Coordinates

For one stack, record the compatibility vector

$$\mathcal{C}(\mathcal{S}) = (C_{\text{alg}}, C_{\text{cfg}}, C_{\text{fwd}}, C_{\text{bwd}}, C_{\text{art}}, C_{\text{merge}}, C_{\text{run}}, C_{\text{evd}}).$$

The coordinates answer different questions:

1. Algebra: does one unambiguous effective-weight operator exist?
2. Configuration: does the pinned constructor accept the flags, warn, or reject?
3. Forward: is there an executable layer path for the representation and parameterization?
4. Backward: are all trainable coordinates and their gradients defined?
5. Artifact: can the exact state be saved, identified, and reconstructed?
6. Merge: is a supported dense or representation-specific export defined?
7. Runtime: can the pinned backend execute the resulting state?
8. Evidence: is the composition supported by a primary method source, official code/test, component-only reasoning, or no evidence beyond shape?

“Compatible” is a lossy projection of this vector. For example, the pinned AdaLoRA configuration inherits use_rslora and accepts it, so $C_{\text{cfg}}$ passes. Its SVD layer still computes lora_alpha / ranknum, not $\alpha/\sqrt{r}$, so the intended forward and backward rsLoRA coordinates are shadowed. A green configuration cell would hide the central fact.

The evidence ladder used below is:

primary composition source + pinned implementation/test
    stronger than
pinned implementation path only
    stronger than
component-level derivation with an explicit missing contract
    stronger than
same shape or accepted flags

No lower rung is promoted to a higher one by prose.

0x09C Composition Order Is Part Of The Operator

The generic additive branch is

$$U = \operatorname{Dequant}(Q,\theta_q) + sBA,$$

but that equation is not yet a stack. Initialization may replace the base by a residual, quantization may round that residual, DoRA may normalize $U$, and an optimizer may freeze one factor. A useful execution order is

1. resolve base checkpoint and target-module orientation
2. construct or load the base representation
3. run any base-mutating initializer (PiSSA or LoftQ contract)
4. install rank/allocation and the declared scale rule
5. wrap the resulting direction with additive LoRA or DoRA coordinates
6. construct exactly one optimizer/backward owner per trainable parameter
7. derive the union artifact and its conversion/merge path
8. admit only a runtime that implements that final operator

These operations generally do not commute. The Loop 9 oracle uses the same $3\times3$ matrix and a deterministic $0.5$ quantization step in two orders:

$$W_a = Q_{0.5}(W_0 - \Pi_1(W_0)) + \Pi_1(W_0),$$ $$W_b = Q_{0.5}(W_0) + \Pi_1\!\left(W_0-Q_{0.5}(W_0)\right),$$

where $\Pi_1$ is a best rank-one approximation. The exact fixture gives

$$\lVert W_a-W_b\rVert_{\infty}=0.356653083918.$$

Both paths contain a quantizer and a rank-one correction. They do not define the same initial model. A manifest that records only rank=1 and bits=4 cannot distinguish them.

0x09D QDoRA Has A Defensible Two-Layer Contract

QDoRA is the clearest sourced composition in this design space. DoRA v6 explicitly discusses replacing the LoRA component inside a QLoRA-style frozen low-bit backbone, and the official QDoRA reproduction directory pins a bitsandbytes/FSDP execution path. PEFT v0.19.1 dispatches a DoRA variant for supported quantized linear layers.

Let

$$W_q = \operatorname{Dequant}(Q,\theta_q), \qquad U_q = W_q + sBA,$$

and let $m$ contain one learned magnitude per PyTorch output row. The effective weight is

$$W_{\text{QDoRA}} = \operatorname{diag}\!\left( \frac{m}{\operatorname{row\_norm}(U_q)} \right)U_q.$$

The quantized payload is not normalized directly as packed integers. The runtime obtains a compute value $W_q$, adds the scaled low-rank direction, and applies DoRA’s row geometry. In the deterministic fixture,

Wq row 1              = [1.0, -0.5, 0.5]
s = alpha / r         = 2.0
(Wq + sBA) row 1      = [1.22, -0.68, 0.50]
initial magnitude     = ||Wq row 1|| = 1.224744871392
effective row 1       = [1.007198734693, -0.561389458681, 0.412786366677]

That state requires the union

quantized base payload + quantizer ABI + scale metadata + compute dtype
LoRA A/B + rank + alpha + scale rule
DoRA magnitude + normalization axis + runtime variant
base checkpoint + target-module identity

It is source-supported, but its accuracy bars remain source-reported rather than reproduced here.

Source: the official NVlabs/DoRA QDoRA materials, also discussed in DoRA v6 Figure 4. The bars report the authors’ LLaMA2-7B and LLaMA3-8B Orca-Math setup. They do not prove that QDoRA universally beats QLoRA or full fine-tuning, and this article does not reproduce the experiment.

0x09E Explicit Or Bounded Framework Compositions

Several other stacks have a narrow, defensible contract when their exact scope is named.

QLoRA Plus rsLoRA

QLoRA changes $R_{\text{base}}$; rsLoRA changes only

$$s_r=\frac{\alpha}{\sqrt r}.$$

For a fixed-rank ordinary LoRA layer, the forward composition is therefore

$$Y=XW_q^\top+\frac{\alpha}{\sqrt r}(XA^\top)B^\top.$$

The runtime path is straightforward, but the artifact is still coupled to the quantizer ABI. It is not a license to merge directly into the packed NF4 payload without a defined dequantize/merge/requantize export.

LoRA-FA Plus rsLoRA

The pinned create_lorafa_optimizer helper accepts use_rslora and computes $\alpha/\sqrt r$; its test suite exercises that option. The Loop 5 backward remains valid because rsLoRA changes the scalar multiplying the frozen-$A$, trainable-$B$ branch. The artifact must retain frozen $A$ even though it has no optimizer state.

PiSSA Plus Fixed-Rank rsLoRA Conversion

PiSSA’s principal/residual split must use the actual scale. With fixed rank, PEFT can convert the mutated-base PiSSA artifact into a portable LoRA delta while preserving rsLoRA scaling. If rank_pattern or alpha_pattern is also present, configuration emits a warning and save_pretrained later rejects the conversion. Runtime training may have worked; portable export still fails.

Quantized-Base AdaLoRA

The pinned PEFT tree includes SVDLinear4bit. That establishes a framework path for AdaLoRA factors and allocation state over a bitsandbytes 4-bit base. It does not make the method LoftQ: no alternating quantization-error initializer follows merely from using a quantized AdaLoRA layer. The pinned path also does not expose an ordinary dense merge contract.

0x09F Executable Does Not Mean Paper-Verified

Some combinations are constructible and differentiable in the pinned code yet remain conditional research stacks.

LoRA+ Plus DoRA

The pinned LoRA+ helper groups parameters using names and dimensionality:

if "lora_A" in name:
    group = groupA
elif "lora_B" in name or parameter.ndim == 1:
    group = groupB_no_decay

DoRA’s magnitude vector is one-dimensional. It therefore receives the same high learning-rate, no-decay group as $B$ in that helper. This is observable framework behavior, not a DoRA or LoRA+ paper result. A different classifier could assign $m$ its own learning rate and weight decay, producing a different optimizer.

PiSSA Plus DoRA

The pinned construction order can run PiSSA initialization before installing the DoRA magnitude variant. Algebraically, if PiSSA creates

$$W_0 = W_{\text{res}} + sBA$$

and DoRA initializes $m=\operatorname{row\_norm}(W_0)$, then

$$\operatorname{diag}\!\left( \frac{m}{\operatorname{row\_norm}(W_{\text{res}}+sBA)} \right)(W_{\text{res}}+sBA)=W_0.$$

The oracle checks zero reconstruction error. That proves the no-op initialization for the declared operator. It does not prove that PEFT’s PiSSA-to-portable-LoRA conversion also preserves DoRA magnitude and runtime identity; no primary composition contract was found for that export.

LoftQ Plus DoRA

The same distinction applies after LoftQ. Let

$$U_{\text{LoftQ}}=W_q+sBA$$

be the coupled initialized direction. Setting $m=\operatorname{row\_norm}(U_{\text{LoftQ}})$ makes DoRA initially reproduce that direction exactly. The fixture verifies zero error. It does not turn the result into a standalone adapter: the quantized base, quantizer ABI, LoftQ factors, and DoRA magnitude remain coupled.

0x09G Rejected, Conflicting, And Shadowed Stacks

The following failures are qualitatively different and must not share one “unsupported” label.

Stack	Pinned result	Why
AdaLoRA + DoRA	hard configuration rejection	AdaLoraConfig rejects use_dora
AdaLoRA + LoftQ	hard configuration rejection	AdaLoraConfig rejects loftq_config
AdaLoRA + rsLoRA	accepted but shadowed	SVD layer executes $\alpha/\text{ranknum}$ rather than $\alpha/\sqrt r$
LoRA+ + LoRA-FA	optimizer-constructor conflict	LoRA+ trains $A/B$ with separate rates; LoRA-FA freezes $A$ and owns a custom $B$ update
LoRA-FA + DoRA	missing backward derivation	the frozen-$A$ helper does not derive DoRA normalization and magnitude gradients
PiSSA + patterned rsLoRA conversion	training/config warning, export rejection	portable conversion cannot reconcile the patterned scaling contract
multiple native PiSSA adapters on one base	unsafe shared-base mutation	each initialization can claim a different residual-base identity

The AdaLoRA/rsLoRA case is especially important because no exception protects the reader. AdaLoraConfig inherits the flag, but the pinned SVD layer’s executed scale remains

$$s_{\text{Ada}}=\frac{\alpha}{\text{ranknum}}.$$

The exact oracle reports accepted_but_shadowed at configuration and shadowed for forward/backward. This is a same-shape counterexample to the claim that accepted flags prove composition.

LoRA-FA plus DoRA is a mathematical, not merely administrative, gap. Loop 5 derived a $B$-only additive update. Loop 8 showed that DoRA’s full gradient passes through

$$\frac{m_i}{\lVert u_i\rVert_2} (I-q_iq_i^\top).$$

Freezing $A$ does not remove the tangent projection or the gradient of $m$. Until an optimizer derives and tests those terms, reusing the additive LoRA-FA helper would be an unverified new method.

0x09H Artifact Compatibility Is The Union Of Changed State

For a stack $\mathcal S$, define

$$\mathcal A(\mathcal S) = \bigcup_{v\in\mathcal S}\mathcal A_v \cup \mathcal A_{\text{interactions}}.$$

The union is necessary but not always sufficient. Interaction fields can be new. A QDoRA artifact needs both quantizer and magnitude identities. A PiSSA/rsLoRA conversion needs the initial adapter and exact scale convention. LoRA+ plus DoRA needs an optimizer-group policy for $m$, which neither method’s short name specifies.

Changed mechanism	Additional portable or resume state
NF4 base	packed payload, quantizer type, block size, scales/double-quant metadata, compute dtype
AdaLoRA	rank pattern, ranknum, schedule; importance and optimizer state for exact resume
rsLoRA	explicit scale rule and rank/alpha used to reconstruct it
PiSSA	residual-base identity, initial adapter identity, conversion rule
LoftQ	quantizer ABI and coupled quantized-base identity
DoRA	magnitude vector, normalization axis, DoRA runtime variant
LoRA+	parameter-group classifier and learning-rate ratio
LoRA-FA	frozen-$A$ payload/identity and FA scale rule

A load test that checks only tensor keys can still accept the wrong semantics. The loader must validate base identity, orientation, target modules, rank and scale convention, initializer/conversion lineage, quantizer ABI, coordinate parameterization, and optimizer schema when resuming.

0x09I Paper, Repository, And Framework Answer Different Questions

Evidence surface	What it can establish	What it cannot establish alone
method paper	equations, intended algorithm, source-reported experiments	behavior of a later PEFT release
official method repository	one concrete integration and reproduction path	all frameworks, layers, and pairwise variants
pinned PEFT source	accepted/rejected flags and executed code path	scientific validity or equivalence to paper pseudocode
pinned PEFT tests	behavior actually asserted by maintainers	untested export and cross-method guarantees
Loop 9 oracle	exact tiny-matrix identities and counterexamples	paper benchmark reproduction or large-model quality

DoRA v6 says DoRA is compatible with LoRA variants and presents QDoRA. That is positive evidence for the compositions actually described. It is not a proof over the Cartesian product of AdaLoRA, LoRA+, LoRA-FA, PiSSA, LoftQ, every quantizer, and every runtime.

Likewise, official code can prove a hard rejection or a shadowed flag without proving that the rejected mathematical combination is impossible. It means the combination is not the pinned framework contract and would require a new owner, derivation, implementation, and test.

0x09J A Decision Procedure That Preserves Uncertainty

def audit_variant_stack(stack, pinned_framework, target_layer):
    axes = normalize_changed_object_tuple(stack)

    algebra = derive_effective_weight_in_execution_order(axes)
    config = inspect_constructor_accept_warn_reject(axes, pinned_framework)
    forward = trace_exact_layer_dispatch(axes, target_layer, pinned_framework)
    backward = derive_all_trainable_gradients_and_optimizer_owners(axes)

    artifact = union_component_state_and_interaction_state(axes)
    merge = derive_export_and_unmerge_preconditions(algebra, artifact)
    runtime = trace_supported_representation_and_layer_path(axes)
    evidence = grade_primary_repo_framework_test_or_component_only(axes)

    contracts = [algebra, config, forward, backward,
                 artifact, merge, runtime, evidence]

    if any(contract.status in {"rejected", "conflict", "missing_derivation"}
           for contract in contracts):
        return reject_or_open_new_method_owner(contracts)

    if any(contract.status in {"conditional", "shadowed", "unverified"}
           for contract in contracts):
        return require_explicit_manifest_and_targeted_oracle(contracts)

    return admit_for_end_to_end_reload_testing(contracts)

The final line hands the stack to Loop 10. Loop 9 establishes whether a coherent candidate exists; it does not yet prove that training, checkpoint, export, reload, and resumed optimization preserve it end to end.

0x09K Executable Oracle And Negative Fixtures

The repository evidence bundle contains a deterministic composition oracle:

docs/numinens-evidence/m6-4-adaptation-design-space/
    variant_composition_fixture.json
    invalid_variant_composition_fixtures.json
    variant_composition_oracle.py
    variant_composition_report.json

It evaluates sixteen named stacks over all eight compatibility coordinates. It also checks:

• classic and rsLoRA scale values for the same $r$ and $\alpha$;
• QDoRA execution over a dequantized base value;
• exact PiSSA-plus-DoRA and LoftQ-plus-DoRA initialization identities;
• non-commutation of quantization and spectral correction;
• the pinned LoRA+ name/dimensionality classifier, including DoRA magnitude;
• artifact-field unions for every stack;
• hard-rejected, shadowed, conflicting, and unverified cases separately;
• 66 one-field-invalid fixtures, all of which must fail validation.

The oracle intentionally reports PiSSA-plus-DoRA and LoftQ-plus-DoRA initialization identities as exact while leaving their portable artifact evidence false. A numerical identity must not erase a missing source or runtime contract.

0x09L Composition Failure Closure

Failure	Plausible symptom	Broken coordinate
treat accepted config as proof	run starts, intended rsLoRA behavior absent	configuration confused with forward
infer compatibility from equal shapes	tensors load, gradients follow wrong operator	algebra/backward
apply DoRA to packed integers	finite output with wrong geometry	representation/forward
reuse LoRA+ grouping for $m$ without declaration	magnitude trains at accidental high LR	optimizer/evidence
reuse LoRA-FA helper under DoRA	memory improves, gradient is incomplete	backward
convert patterned PiSSA/rsLoRA as ordinary LoRA	export fails or scale drifts	artifact
call quantized AdaLoRA LoftQ	4-bit layer runs, intended initialization absent	initializer/evidence
save factors without quantizer ABI	adapter loads against a different dequantized base	artifact/runtime
save DoRA factors without $m$	shapes partially match, effective weight changes	parameterization/artifact
initialize multiple PiSSA adapters against one mutated base	adapter order changes function	base identity
generalize QDoRA bars	one paper setup becomes a universal ranking	evidence
claim version-independent support	upgrade changes validation or dispatch	runtime/evidence

The decision rule is deliberately conservative: a stack advances only when its changed-object tuple is complete, every compatibility coordinate has an owner, and uncertainty remains visible. More acronyms do not imply a better method; they imply a larger proof and artifact surface.

0x09M Boundary To End-To-End Execution And Multi-Adapter Serving

Loop 10 owns one complete training/checkpoint/export/reload/resume path after a stack passes this decision surface. It must validate state-dict keys, optimizer ownership, conversion output, and numerical reload parity.

M6.5 owns multiple independently named adapters at serving time: registry, residency, request routing, heterogeneous batching, hotswap, eviction, and KV compatibility. M6.9 owns learned routing, shared factors, composition policies, and continual adapter systems. M8 owns generic low-bit deployment formats and quantization release decisions. This loop uses those mechanisms only to audit one training-stack contract.

0x0A End-To-End Variant Lifecycle: Train, Resume, Export, Reload, Release, Roll Back

Loop 9 answered a composition question: is a proposed stack coherent across algebra, configuration, forward, backward, optimizer, artifact, runtime, and evidence coordinates? That is necessary, but it is not a lifecycle proof. A stack can be mathematically coherent and still fail when a training job resumes, when adapter namespaces are rewritten, when a native initialization is converted, or when a release is rolled back.

This section therefore follows four admitted stacks through one bounded but complete transaction:

Stack	Why it is included	Changed lifecycle surface
QLoRA + rsLoRA	quantized/dequantized base plus rank-stabilized additive branch	quantizer ABI, scaling metadata, dense export boundary
QDoRA	low-bit base plus normalized magnitude/direction parameterization	detached-normalization backward, magnitude state, normalized export
LoRA-FA + rsLoRA	frozen $A$, trainable $B$	optimizer ownership differs from portable artifact ownership
PiSSA + fixed-rank rsLoRA	training mutates the base/factor decomposition	native residual-base checkpoint must be converted for the original base

They are not a claim that these are the four best methods. They are selected because each exposes a different failure class. The executable oracle uses small deterministic float64 tensors, not a paper training reproduction or a replacement for the pinned PEFT runtime.

0x0AA Seven Checkpoints, Not One save_pretrained

The transaction is divided into seven checkpoints:

Checkpoint	Required state	Question it answers
$P_0$	immutable manifest	what exact base, quantizer, scale, initializer, namespace, and optimizer does this run mean?
$P_1$	exact-resume snapshot after update one	can optimization restart rather than merely inference?
$P_2$	uninterrupted state after update two	what is the reference continuation?
$P_3$	reload $P_1$, then apply the same update two	is resumed state identical to uninterrupted state?
$P_4$	slot-neutral portable adapter, reloaded under another slot name	does namespace translation preserve the function?
$P_5$	representation-specific release	what payload actually executes after export?
$P_6$	immutable-base rollback with no adapter	can the release be removed without subtracting rounded deltas?

The first distinction is between three payload classes:

exact resume:
    model/base identity + trainable parameters + required frozen adapter state
    + optimizer moments + optimizer step + scheduler/RNG state in a real run

portable adapter:
    inference parameters + base/scale/quantizer metadata
    - optimizer moments
    - training slot name

release:
    the representation consumed by the serving runtime
    (dense weight, live adapter, or a separately validated low-bit package)

The reduced oracle has no stochastic layers or scheduler, so its exact-resume payload contains the base tensors, adapter tensors, AdamW moments, and step counter. A real manifest must additionally own RNG streams, data position, gradient scaler, scheduler, and distributed-shard metadata.

0x0AB The Common Two-Step Optimization Trace

All four transactions use the same shape contract:

W0, Wq, W_eff : [d_out=3, d_in=3]
A              : [r=2, d_in=3]
B              : [d_out=3, r=2]
magnitude      : [d_out=3]       # QDoRA only
X, G           : [tokens=2, 3]
Y = X W_eff^T  : [tokens=2, 3]

For the scalar oracle loss

$$\mathcal L=\langle G,Y\rangle_F, \qquad Y=XW_{\mathrm{eff}}^\top,$$

the effective-weight adjoint is

$$\frac{\partial\mathcal L}{\partial W_{\mathrm{eff}}} =G^\top X.$$

Each trainable coordinate $\theta$ owns independent AdamW moments. With $g_t=\partial\mathcal L/\partial\theta_t$,

$$m_t=\beta_1m_{t-1}+(1-\beta_1)g_t, \qquad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,$$ $$\widehat m_t=\frac{m_t}{1-\beta_1^t}, \qquad \widehat v_t=\frac{v_t}{1-\beta_2^t},$$ $$\theta_{t+1} =\theta_t-\eta_\theta \left( \frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} +\lambda_\theta\theta_t \right).$$

The subscript on $\eta_\theta$ and $\lambda_\theta$ matters. QDoRA’s magnitude vector can use a different optimizer group; LoRA-FA has no optimizer slot for $A$ at all. Saving only tensors named lora_A and lora_B cannot reproduce either optimizer contract.

The exact-resume test is intentionally stronger than an output tolerance:

state_1 = adamw_step(initial_state, batch_1)
resume_payload = json_round_trip(all_resume_state(state_1))

state_2_direct = adamw_step(state_1, batch_2)
state_2_reload = adamw_step(restore(resume_payload), batch_2)

assert state_2_direct == state_2_reload
assert sha256(state_2_direct) == sha256(state_2_reload)

The JSON encoding retains Python float round-trip precision; it is not rounded for the resume path. All four $P_2/P_3$ hashes match exactly. That proves the oracle’s serialized float64 state machine, not bitwise parity across GPU kernels, distributed reductions, or framework releases.

0x0AC QLoRA Plus rsLoRA: Coupled Base Identity, Separate Release Identity

The QLoRA transaction begins from a stored low-bit payload whose execution value is represented by $W_q$. Its additive path is

$$W_{\mathrm{eff}}=W_q+sBA, \qquad s=\frac{\alpha}{\sqrt r}.$$

For $r=2$ and $\alpha=4$, the oracle records $s=2\sqrt2\approx2.828427$. The persistent training state includes:

immutable base identity       nf4_payload_v1
quantizer ABI                 nf4_block64_double_quant_v1
adapter                       A[2,3], B[3,2]
optimizer                     m_A, v_A, m_B, v_B, step
scale contract                alpha=4, r=2, alpha/sqrt(r)

The quantizer ABI is not decorative metadata. The portable factors reconstruct the same function only against the same dequantized base contract. A loader that silently changes block size, codebook, double-quant metadata, or base revision changes $W_q$ before the adapter executes.

The oracle’s $P_4$ payload removes the training slot:

local:
    base_model.model.proj.lora_A.default.weight
    base_model.model.proj.lora_B.default.weight

portable:
    proj.lora_A.weight
    proj.lora_B.weight

reloaded local namespace:
    base_model.model.proj.lora_A.task_blue.weight
    base_model.model.proj.lora_B.task_blue.weight

It then reconstructs $A$ and $B$ from that portable payload, binds them to the immutable $W_q$ identity, and obtains zero weight and output error relative to the trained state.

The $P_5$ oracle release is deliberately a dequantized dense weight,

$$W_{\mathrm{dense}}=W_q+sBA.$$

That is not an in-place low-bit merge. Producing a low-bit serving artifact would require a new quantization pass, calibration and error checks, kernel compatibility, and release acceptance owned by M8. Calling the dense material- ization “merging into NF4” would erase an entire representation transition.

At $P_6$, rollback restores the immutable quantized runtime base plus no adapter. It does not restore the pre-quantized $W_0$, and it does not compute $W_{\mathrm{dense}}-sBA$.

0x0AD QDoRA: The Forward Geometry And Training Adjoint Are Separate Contracts

QDoRA adds a learned row magnitude $m$ around the low-bit direction:

$$U=W_q+sBA, \qquad q_i=\frac{U_i}{\lVert U_i\rVert_2}, \qquad (W_{\mathrm{eff}})_i=m_iq_i.$$

The declared initializer uses $B=0$ and $m_i=\lVert(W_q)_i\rVert_2$. Therefore

$$(W_{\mathrm{eff}})_i =\lVert(W_q)_i\rVert_2 \frac{(W_q)_i}{\lVert(W_q)_i\rVert_2} =(W_q)_i.$$

This identity is tested before optimization. Reusing the nonzero illustrative $B$ from another stack would violate the initializer even though all tensor shapes still match.

For one output row, let $g_i=\partial\mathcal L/\partial (W_{\mathrm{eff}})_i$. The full normalization adjoint is

$$\frac{\partial\mathcal L}{\partial U_i} =\frac{m_i}{\lVert U_i\rVert_2} \left[g_i-(g_i^\top q_i)q_i\right], \qquad \frac{\partial\mathcal L}{\partial m_i}=g_i^\top q_i.$$

The bracket projects the gradient onto the tangent space of the unit sphere. The pinned PEFT DoRA path uses a detached weight norm in the low-rank branch; the corresponding pedagogical training adjoint omits that radial projection:

$$\left.\frac{\partial\mathcal L}{\partial U_i}\right|_{\text{detached norm}} =\frac{m_i}{\operatorname{stopgrad}(\lVert U_i\rVert_2)}g_i.$$

Those are intentionally not described as the same gradient. The oracle finite-difference checks the full forward derivative and obtains maximum error below $3.04\times10^{-10}$; separately, it executes the detached-normalization training rule. Their largest elementwise difference in the toy state is $1.3475$, making the implementation choice visible instead of hiding it behind the name DoRA.

The QDoRA exact-resume payload must contain moments for $A$, $B$, and $m$. The portable inference payload excludes those moments but includes all three parameter tensors, base identity, quantizer ABI, normalization parameterization, and scale. Omitting $m$ produces a shape-plausible LoRA artifact with the wrong function.

Its $P_5$ release materializes

$$W_{\mathrm{dense},i} =m_i\frac{(W_q+sBA)_i}{\lVert(W_q+sBA)_i\rVert_2}.$$

As with QLoRA, this is a dense normalized release, not proof of a supported low-bit in-place merge.

0x0AE LoRA-FA Plus rsLoRA: Frozen Is Not Absent

LoRA-FA keeps $A$ fixed and updates only $B$:

$$W_{\mathrm{eff}}=W_0+sBA, \qquad s=\frac{\alpha}{\sqrt r}, \qquad dA=0.$$

The optimizer owns only

B, m_B, v_B, step

but both training continuation and portable inference still depend on $A$. The resulting ownership table is deliberately asymmetric:

State	Trainable?	Optimizer moments?	Saved for exact resume?	Saved for portable adapter?
$A$	no	no	yes	yes
$B$	yes	yes	yes	yes
base $W_0$ identity	no	no	yes	referenced
scale rule	no	no	yes	yes

This is why “save all trainable parameters” is not a sufficient artifact rule. Dropping frozen $A$ destroys $BA$ even though no optimizer ever updated it. Conversely, placing fake AdamW moments beside $A$ would misrepresent the optimization state and complicate exact resume.

The oracle confirms that the $P_1$ optimizer key set is only B, while the $P_4$ portable key set contains both lora_A and lora_B. Its dense release $W_0+sBA$ reproduces the live branch exactly, and rollback restores immutable $W_0$ rather than subtracting the learned update.

0x0AF PiSSA Plus Fixed-rank rsLoRA: Native And Portable Coordinates Differ

PiSSA begins with a scale-aware rank-$r$ factorization:

$$W_0=W_{\mathrm{res}}+sB_0A_0.$$

Training keeps $W_{\mathrm{res}}$ fixed and changes the factors to $B_t,A_t$:

$$W_{\mathrm{native},t}=W_{\mathrm{res}}+sB_tA_t.$$

That native artifact cannot be interpreted as an ordinary adapter on the original $W_0$, because $W_0$ already contains $sB_0A_0$. Substitute the initial decomposition:

$$W_{\mathrm{native},t} =W_0+s(B_tA_t-B_0A_0).$$

The difference of two rank-$r$ products has rank at most $2r$. A portable ordinary-LoRA representation is therefore

$$B_\Delta=[\,B_t\;B_0\,], \qquad A_\Delta= \begin{bmatrix} A_t\\ -A_0 \end{bmatrix},$$ $$B_\Delta A_\Delta=B_tA_t-B_0A_0.$$

The conversion doubles the stored rank. Under rsLoRA, simply changing the metadata from $r$ to $2r$ would change the scale. To preserve $s=\alpha/\sqrt r$, the converted alpha must satisfy

$$\frac{\alpha'}{\sqrt{2r}}=\frac{\alpha}{\sqrt r} \quad\Longrightarrow\quad \alpha'=\sqrt2\,\alpha.$$

For the fixture, $r=2$ becomes $r'=4$ and $\alpha=4$ becomes $\alpha'=4\sqrt2\approx5.656854$, while $s$ remains $2\sqrt2$. The converted payload reloads under task_blue against immutable $W_0$ with zero weight and output error.

This stack produces two distinct $P_5$ components:

1. a rank-doubled portable adapter on $W_0$;
2. a dense materialization of the same effective weight.

The dense component contains no live adapter keys. The portable component does. Combining those statements into “the release has no adapter” would be false. Rollback uses the immutable original $W_0$ plus no adapter; the training-time residual $W_{\mathrm{res}}$ is not an archival rollback checkpoint by itself.

0x0AG Namespace Translation Is A State-Dict Operation, Not Algebra

Adapter slot names are runtime registry coordinates. They should not be baked into a portable artifact as if default were part of the learned function. The loader performs an explicit translation:

def portable_to_local(portable_state, local_prefix, reload_slot):
    local = {}
    for portable_key, tensor in portable_state.items():
        stem = strip_portable_prefix(portable_key)
        local_key = f"{local_prefix}.{stem}.{reload_slot}.weight"
        local[local_key] = tensor
    return local

Translation is admitted only after validating base identity, quantizer ABI, rank, alpha, scale rule, initializer/conversion status, and parameterization. Renaming keys does not authorize rebasing an adapter, changing its quantizer, or dropping DoRA magnitude. The oracle proves default is absent from portable keys and task_blue is present after reload; it also reconstructs the output from the translated tensors rather than reusing the in-memory trained model.

0x0AH Export Is A Derived Artifact, Not The Resume Checkpoint

The four export rules are:

Stack	Native training execution	Portable/runtime artifact	Dense release rule
QLoRA + rsLoRA	$W_q+sBA$	$A,B$ + quantizer/base/scale ABI	materialize $W_q+sBA$ in declared dense dtype
QDoRA	$m\odot\operatorname{normalize}(W_q+sBA)$	$A,B,m$ + quantizer/base/normalization ABI	materialize normalized effective weight
LoRA-FA + rsLoRA	$W_0+sBA$, $A$ frozen	$A,B$ + base/scale ABI	materialize $W_0+sBA$
PiSSA + rsLoRA	$W_{\mathrm{res}}+sB_tA_t$	rank-$2r$ converted $B_\Delta,A_\Delta$ on $W_0$	materialize either equivalent expression

None of these portable artifacts contains optimizer moments. None is therefore an exact-resume checkpoint. Conversely, an exact-resume checkpoint can be a poor serving artifact because it contains optimizer state, training namespaces, native residual coordinates, or a training-oriented quantizer layout.

Release acceptance checks the function at the boundary:

portable = export_slot_neutral_adapter(trained_state)
reloaded = load_against_immutable_base(portable, slot="task_blue")
dense = materialize_representation_specific_release(reloaded)

assert max_abs(reloaded.weight - trained.weight) <= tolerance
assert max_abs(X @ dense.T - X @ trained.weight.T) <= tolerance
assert dense_contains_no_live_adapter_keys(dense)

The last assertion applies to the dense component, not to a bundle that also ships a portable adapter.

0x0AI Rollback Restores Identity; It Does Not Invert Floating-point History

Arithmetic unmerge looks attractive:

$$\widehat W_{\mathrm{rollback}} =\operatorname{fl}(W_{\mathrm{release}}-sBA).$$

But it depends on retaining the exact factors and scale, matching dtype and rounding order, and avoiding any intervening quantization or normalization. QDoRA is not additive in the released coordinate at all. A requantized QLoRA release is also not invertible by subtracting a floating-point delta from its packed payload.

The lifecycle rule is therefore:

rollback = load(immutable_base_identity) + activate(no_adapter)

For low-bit stacks, that identity is the immutable quantized runtime base. For dense LoRA-FA and converted PiSSA, it is the immutable dense $W_0$. All four oracle transactions report zero rollback error and explicitly report uses_arithmetic_unmerge = false.

0x0AJ Rejected And Bounded Lifecycle Cases Stay Rejected

Loop 10 does not turn Loop 9’s unknown or conflicting stacks into supported methods merely by writing generic save code:

Stack	Stop point	Why no end-to-end transaction is claimed
AdaLoRA + DoRA	configuration	pinned configuration rejects the composition
AdaLoRA + LoftQ	configuration	pinned initialization path does not establish the combination
LoRA+ + LoRA-FA	optimizer construction	separate-factor rates conflict with frozen-$A$ ownership
LoRA-FA + DoRA	backward derivation	no admitted $B$-only normalized-direction adjoint
patterned PiSSA + rsLoRA	portable export	conversion cannot preserve an undeclared per-pattern scale transformation
quantized AdaLoRA	merge	supported merge/release path is not established

“Rejected” means the proof surface stops at the named stage. It does not mean the combination is mathematically impossible forever. Implementing one would require a new owner contract, backward derivation, artifact schema, runtime path, and evidence bundle.

0x0AK Executable Oracle And Measured Closure

The repository evidence bundle is executable:

docs/numinens-evidence/m6-4-adaptation-design-space/
    variant_lifecycle_fixture.json
    invalid_variant_lifecycle_fixtures.json
    variant_lifecycle_oracle.py
    variant_lifecycle_report.json

It checks:

• four selected stack transactions over $P_0\ldots P_6$;
• full-forward finite-difference gradients, separately from QDoRA’s detached training adjoint;
• exact direct-versus-resumed state hashes after the second AdamW update;
• portable state with optimizer exclusion and slot-neutral keys;
• reload under a different adapter name from the portable payload itself;
• PiSSA rank doubling and rsLoRA alpha conversion;
• dense release output parity and component-specific adapter-key accounting;
• immutable-base rollback without arithmetic unmerge;
• six rejected or bounded stack cases;
• 95 one-field-invalid manifests, all of which must fail validation.

The maximum full-forward finite-difference error is $6.54\times10^{-10}$. Every direct/resume state error, portable reload weight error, portable reload output error, dense export output error, and immutable rollback error is zero in the deterministic fixture. These are oracle results, not model-quality, throughput, convergence, low-bit kernel, or cross-release compatibility measurements.

0x0AL Training, Artifact, And Runtime Comparison

Method	Extra training state	Forward-path change	Standard dense merge possible?	Serving caveat
LoRA	$A,B$ and optimizer state	two low-rank matmuls	yes	adapter identity/rank affects batching
QLoRA	quant metadata and possible paged optimizer behavior	quantized base kernel plus LoRA	only through declared materialize/requantize path	training format may not be serving format
AdaLoRA	importance statistics, schedule, SVD-like factors	low-rank branch with changing active components	after final structure is fixed	heterogeneous ranks complicate kernels
rsLoRA	no new tensor family	different scalar scale	yes	scaling convention must travel with checkpoint
LoRA+	optimizer groups and learning-rate ratio	unchanged	yes	optimizer state changes; inference algebra does not
LoRA-FA	optimizer state for $B$ only; portable artifact retains $A$	additive branch with frozen $A$	yes	frozen state is still required state
PiSSA	SVD initialization and residual base	low-rank branch over residual base	yes after correct conversion	native and portable coordinates differ
LoftQ	quantizer state and alternating initialization	quantized base plus LoRA	implementation-dependent export	preserve quantizer ABI and coupled base identity
DoRA	magnitude vector and normalization	magnitude/direction reconstruction	yes after materializing normalized weight	live runtime needs the matching DoRA path

0x0AM Boundary To Multi-Adapter Serving And Generic Deployment

This loop follows one selected adapter at a time. M6.5 owns adapter registries, multiple resident adapters, heterogeneous-request batching, hotswap, eviction, request routing, and KV compatibility. M6.9 owns learned routing, shared factors, adapter composition, and continual systems. M8 owns generic dense-to- low-bit release formats, calibration, target-hardware evaluation, packaging, release promotion, and rollback across deployment representations.

0x0B Failure Closure, Complete Sources, Figures, Links, And Reproducibility

A list of plausible mistakes is not yet a failure contract. A publishable method comparison must say which invariant failed, which executable observation detects it, what state becomes untrustworthy, and which component owns recovery. That is especially important here because many failures preserve tensor shape: a wrong scale, stale quantizer ABI, missing frozen factor, or incompatible base can all load successfully and still execute the wrong function.

0x0BA A Failure Record Needs Five Coordinates

Every failure in the machine-readable closure manifest has five coordinates:

failure id and category:
    a stable identity and the alarm family that owns it

invariant:
    the algebraic, representation, optimizer, artifact, or lifecycle fact
    that must remain true

detector:
    an executable equality, hash, conservation law, state-schema check,
    finite-difference comparison, or bounded discrepancy

consequence:
    the first state or claim that can no longer be trusted

recovery owner:
    reject configuration, rebuild initialization, restore immutable base,
    reload exact resume state, regenerate export, or narrow the claim

The recovery owner matters. A quantizer-ABI mismatch is not repaired by rescaling the adapter. A missing optimizer moment is not repaired by reloading a portable inference artifact. A source-reported curve is not made reproducible by rerunning the tiny algebra oracle.

0x0BB Same Shape Does Not Mean Same Semantics

Shape checks are necessary, but they cannot establish method identity. The following counterexamples preserve the expected matrix dimensions while violating a different coordinate:

Same-shape state	Hidden semantic mismatch	Required detector	Recovery
NF4 payload with the expected packed byte count	codebook order or nested-scale metadata changed	quantizer ABI plus exact dequantized-value hash	reject the artifact and restore the declared quantized base
AdaLoRA triplets with the expected maximum rank	importance EMA, uncertainty, or budget step omitted	replay the schedule and global threshold against the recorded step	reload exact allocator state or restart allocation from a declared boundary
rsLoRA factors with unchanged $A,B$ shapes	rank changed while alpha or scale exponent stayed stale	recompute $s(r)$ and the branch-output checksum	reject or rescale under an explicit conversion contract
LoRA+ optimizer with all expected parameter names	$A$ and $B$ groups use the same learning rate	inspect group identity and scheduler-adjusted rates	rebuild optimizer groups before the next update
LoRA-FA checkpoint with a valid trainable $B$	frozen $A$ absent or paired with another base	hash frozen $A$ and compare the induced projector	restore the exact frozen factor; do not synthesize one
PiSSA factors of rank $r$	residual base paired with the original pretrained base or portable rank-$2r$ payload mislabeled as native	reconstruct the initial effective weight and inspect artifact schema	reload the matching residual base or run the declared conversion
LoftQ adapter with valid low-rank factors	quantized base came from another alternating step or quantizer	evaluate the coupled reconstruction objective and base hash	restore the coupled base/adapter pair
DoRA magnitude vector with the expected length	magnitude belongs to another normalized direction or row/column convention	reconstruct executed rows and compare live/merge output	reject the pair and reload matching magnitude-direction state
portable adapter whose keys parse correctly	base revision, target slice, scale, or representation identity differs	slot-neutral reload plus effective-weight checksum	fail admission before graph mutation
dense release with the expected tensor shapes	release was formed from stale live state or arithmetic unmerge	compare release components and output against the immutable manifest	regenerate release from the admitted checkpoint

Output equality at one probe is also insufficient. A zero-$B$ LoRA branch and a missing branch both produce the base output at initialization, but only the former owns trainable state and the first-step gradient asymmetry. QDoRA’s full normalization derivative and detached-norm training adjoint share the same forward tensor yet implement different backward contracts. Detection must therefore inspect the coordinate that defines the method, not merely one output.

0x0BC Failures Belong To Different Alarms

The closure manifest groups thirty-six records across Loops 2 through 10:

Alarm family	Representative invariant	Detector	Canonical recovery owner
representation and precision	packed base, codebook, scale hierarchy, compute dtype, and quantizer ABI agree	exact dequantization trace and representation hash	artifact admission or immutable-base restore
allocation and schedule	rank budget, importance state, mask interval, and optimizer step are synchronized	schedule replay, global projection count, and exact-resume state hash	allocator/checkpoint owner
scaling and optimization	branch scale, factor groups, frozen state, and backward operator match the declared variant	analytical/finite-difference gradient and optimizer-group trace	configuration or optimizer reconstruction
spectral and initialization	principal/residual or quantization-error coordinates reconstruct the declared initial function	SVD residual, objective, and initial-output equality	initialization/conversion owner
composition and lifecycle	union artifact, namespace translation, release, and rollback preserve representation identity	slot-neutral reload, component inventory, output equality, and immutable hash	export/release controller
evidence and scope	a paper plot, teaching oracle, framework trace, and production claim keep distinct evidence classes	provenance record and claim firewall	editorial/source owner

No alarm substitutes for another. A finite-difference gradient can validate a local backward formula while saying nothing about a missing RNG state at resume. A successful portable reload can validate inference state while saying nothing about optimizer continuity. A paper figure can motivate a method while proving neither this article’s oracle nor a target deployment.

0x0BD Complete Source, Link, Visual, And Negative-Test Surface

Loop 11 closes the static publication surface rather than sampling it:

28 unique source records
    3 digest-pinned local owners
    11 version-pinned primary papers
    1 unversioned QLoRA arXiv landing link kept by a figure caption
    13 pinned official implementation or documentation surfaces

25 unique external URLs published by this article
5 normalized local post routes
53 A22 figure dispositions, including all 18 M6.4-owned source figures
31 rendered visual elements and 41 static desktop/mobile asset routes
36 typed failure records across Loops 2-10
9 prior invalid-fixture families with 484 one-field mutation cases
18 new one-field publication-surface mutations

The source manifest and published URL set are exact peers: an external URL in the article without a record is uncovered evidence, while a record absent from the article is stale publication metadata. The A22 disposition remains complete even when a source-local image is replaced, deferred, or superseded by a primary paper figure. Every article image route must resolve, carry nonempty alternative text, have cumulative provenance, and, for SVG, expose role="img", a viewBox, <title>, and <desc>.

These are static guarantees. They do not prove that an external host is live, that MathJax completed in a browser, that a mobile table does not overflow, or that dark-mode contrast is acceptable. Loop 12 owns normalized desktop/mobile, light/dark rendering and live-route acceptance.

0x0BE Reproducibility Is A Chain Of Named Artifacts

A serious variant comparison records:

base checkpoint and immutable revision
target module path, object identity, packed slice, and storage alias
rank or rank budget per module
alpha, scale rule, and rank-pattern overrides
initialization method and any residual-base conversion
adapter, accumulation, normalization, and compute dtype
quantizer type, codebook, group size, scale hierarchy, and metadata dtype
optimizer class, factor groups, factor-specific learning rates, and scheduler
which factors are frozen and which activations are retained
rank-allocation schedule, importance smoothing, and projection step
dropout, clipping, sequence length, and activation checkpointing
framework, kernel, and quantizer ABI revision
base, quantizer, adapter, frozen-factor, and optimizer-state hashes
local, resume, portable, and release state-dict schemas
optimizer moments, step, scheduler, scaler, RNG, sampler, and counter ownership
native-to-portable conversion rule and output metadata
release component inventory and target runtime capability
rollback base identity and generation transition
evidence class for every reported number or figure

The executable chain is equally explicit:

Loop 1 manifest and owner firewalls
    -> Loops 2-8 method-specific tensor/state oracles
    -> Loop 9 composition classifier
    -> Loop 10 train/resume/export/reload lifecycle oracle
    -> Loop 11 static source, figure, failure, and publication audit
    -> Loop 12 normalized browser and cross-article acceptance

Without that chain, two runs both labeled “QLoRA rank 16” can be materially different systems even when their adapter tensors have identical shapes.

0x0BF Final Formula, Rendering, Accessibility, And Boundary Closure

The final audit reconciles five surfaces that cannot substitute for one another:

Surface	Exact closure	What it does not prove
source	298 source-ordered display formulas, 52 tables, 89 fenced code blocks, 31 rendered visual elements backed by 41 desktop/mobile asset routes, 25 external URLs, and five normalized local post routes	formal correctness of every derivation or future availability of an external host
generated route	every source display formula becomes one display MathJax source, while every inline and display source completes without a MathJax error	compatibility with every renderer, font override, browser, or assistive technology
responsive images	ten picture elements select their mobile source below the declared breakpoint and their desktop source above it; the remaining 21 images retain one source	perceptual quality under every zoom level, color-vision condition, or user stylesheet
four browser profiles	desktop/mobile and light/dark profiles load every image and local owner route, preserve unique heading targets, contain article-owned overflow, and produce no article console exception	framework execution, GPU kernels, model quality, memory, latency, throughput, or live third-party endpoint health
boundaries	concrete M6.3, M6.5, M6.7, M6.9, and M8.1 owners resolve without long exact prose duplication	that those neighboring owners are already publication-complete

Browser acceptance does not promote a paper figure, source trace, or teaching oracle into production evidence. It proves only that this built M6.4 snapshot exposes the declared formulas, links, tables, alternatives, and responsive figures without article-local publication defects under the normalized profiles. The source manifest still owns claim provenance; Loops 2 through 10 still own their narrow tensor, optimizer, composition, and lifecycle contracts; and M6.5, M6.7, M6.9, and M8 retain their deferred runtime, theory, continual, and deployment responsibilities.

0x0D Boundary To M6.5 And M8

M6.4 ends after training and export state are defined.

M6.5 owns:

adapter registry and residency
live branch kernels
heterogeneous ranks in one batch
merge, unmerge, and hotswap
adapter paging and eviction
request/KV compatibility
latency and memory telemetry

M6.9 owns learned adapter routing, shared factors, composition, and continual systems.

M6.10 owns distillation and model transfer; M6.11 owns pruning and structural compression.

M8.1 begins M8, the quantization track, which carries low-bit representations through packaging, hardware-specific formats, release promotion, and rollback.

0x0E Publish Gate

Check	Status
Vanilla LoRA state, quantization, rank, initialization, and runtime axes precede the variant map	pass
Every variant is organized by the mechanism it changes	pass
QLoRA base, adapter, gradient, and optimizer state are separated	pass
NF4, double quantization, and paging are explained mechanistically	pass
NF4’s asymmetric exact-zero construction is distinguished from naive normal quantiles	pass
Double-quantization metadata savings are derived with their source-specific block assumptions	pass
Frozen base gradients are separated from input and adapter gradients	pass
AdaLoRA rank budget, schedule, importance state, and runtime consequence are present	pass
rsLoRA scaling is derived and compared numerically	pass
LoRA+ is identified as an optimizer change	pass
LoRA-FA trainable state, backward equations, and activation-saving boundary are explicit	pass
PiSSA’s principal/residual split is scale-aware and preserves the initial function under the declared dtype tolerance	pass
PiSSA factor gradients, tangent rotation, exact/fast SVD distinction, and dropout boundary are derived	pass
Native residual-base PiSSA and portable rank-doubled conversion artifacts are separated	pass
LoftQ objective derives the executed low-bit base plus scaled low-rank branch	pass
LoftQ’s approximate quantizer update and exact fixed-base SVD update are separated	pass
T=0, T=1, full alternation, and PEFT one-step replacement are not collapsed	pass
LoftQ scale, quantizer ABI, coupled base/adapter artifact, dropout, merge, and resume contracts are explicit	pass
Alternating-step and downstream monotonicity are not overclaimed	pass
DoRA magnitude/direction tensors and reconstruction are explicit	pass
Variant composition is audited over algebra, configuration, forward, backward, artifact, merge, runtime, and evidence independently	pass
QDoRA is derived over the dequantized base value and its source-reported evidence remains scoped	pass
Hard rejection, optimizer conflict, missing derivation, and accepted-but-shadowed flags are not collapsed	pass
Composition order and the union artifact are explicit, including a numerical non-commutation witness	pass
The executable oracle classifies sixteen stacks and rejects all 66 one-field-invalid fixtures	pass
End-to-end execution separates immutable manifest, exact resume, uninterrupted reference, resumed continuation, portable renamed reload, release, and rollback	pass
QLoRA and QDoRA bind portable state to the declared base identity and quantizer ABI	pass
QDoRA full normalization gradient and detached-norm training adjoint remain separate contracts	pass
LoRA-FA saves frozen $A$ while assigning optimizer moments only to trainable $B$	pass
PiSSA fixed-rank rsLoRA conversion doubles rank and adjusts alpha to preserve scale	pass
Portable renamed reload reconstructs effective weights from the portable payload rather than the live training model	pass
Exact-resume direct and reloaded second-update hashes match for all four selected stacks	pass
Dense release components contain no live adapter keys and preserve oracle outputs	pass
Immutable rollback is representation-specific and never uses arithmetic unmerge	pass
The lifecycle oracle closes four transactions and rejects all 95 one-field-invalid manifests	pass
Training, artifact, release, and runtime consequences are compared	pass
Source-reported results are not generalized	pass
Thirty-six typed failures bind invariants to detectors, consequences, and recovery owners	pass
All 28 source records, 25 external URLs, five local routes, 53 A22 figure dispositions, and 41 static asset routes reconcile	pass
Nine prior invalid-fixture families contain 484 one-field failures across Loops 2-10	pass
Eighteen one-field publication-surface mutations fail closed	pass
Static Loop 11 closure remains distinct from browser and live-link evidence	pass
Multi-adapter runtime is deferred to M6.5	pass

0xFF References

• Transformer Series A22 - LoRA
• M6.3 - LoRA Foundations
• LoRA v2
• QLoRA v1
• AdaLoRA v2
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA v1
• LoRA+ v2
• LoRA-FA v3
• PiSSA v4
• PiSSA official repository
• MiLoRA v3
• LoftQ v4
• LoftQ reference repository
• Hugging Face PEFT v0.19.1
• DoRA v6
• DoRA official repository
• Official QDoRA reproduction instructions
• Hugging Face PEFT LoRA documentation