M6.9 - Routed, Shared, and Continual Adapters

0x00 Overview: One Adapter Is A Tensor Pair; Many Adapters Are A Control System

M6.3 derived the LoRA operator and its gradients. M6.4 compared changes to rank, scaling, initialization, precision, and weight parameterization. M6.5 treated independently trained adapters as serving objects that must be loaded, batched, pinned, and associated with request K/V state. M6.7 then separated factor rank from objective intrinsic dimension and showed why factor coordinates are not unique.

Those articles intentionally stop before a different family of questions:

If several low-rank updates exist at the same layer, should the model select one, mix several, share one factor, preserve each task as immutable state, or merge the updates into a new checkpoint?

The common shorthand is “LoRA plus MoE.” It is too imprecise. Two systems can compute a weighted sum of adapter outputs and still differ in every important lifecycle decision:

when the specialists were trained
whether specialist tensors remain frozen
whether the router is supervised by task labels
whether one route applies to a task, request, sequence, token, or layer
whether all experts run or only top-k experts run
whether experts share A, share B, or share nothing
whether old task state is preserved, merged, or discarded
whether the serving kernel sees one adapter id or route metadata per row

These choices determine interference, compute, memory growth, mergeability, cache provenance, and rollback. The router equation alone does not define the method.

0x00A The Problem Did Not Begin With A Router Equation

The historical path to routed adapters did not begin by replacing an FFN expert with a LoRA expert. It began with an artifact-management problem. Vanilla LoRA made one task update cheap to train and store, but a useful system soon accumulated several updates for the same frozen base. The first design decision was therefore not which router to use. It was what the adapter collection means.

The design space widened in five steps:

1. One task, one adapter. Each low-rank delta is trained and evaluated independently. Task identity is external, and serving resolves one adapter ID. This is the baseline owned by M6.3 and M6.5.
2. Static fusion. Several trained deltas receive manually chosen or learned constants and are added into one effective update. The resulting artifact has no input-conditioned route, even when its coefficients were tuned on data.
3. Staged or post-hoc composition. Domain specialists are trained first; a later stage learns how to select or mix them. Mixture-of-LoRAs uses explicit domain supervision in a staged multi-task procedure, whereas MoLE treats trained LoRAs as capabilities to be composed by learned layer-local gates.
4. Joint routed experts and factor sharing. LoRAMoE initializes several adapter experts and a router for one heterogeneous tuning run. HydraLoRA changes the hypothesis class again by sharing one factor while routing several task-sensitive factors.
5. Continual adapter state. O-LoRA and AM-LoRA assume tasks arrive sequentially. Their central object is not merely a gate vector; it is a growing, versioned history of old and new low-rank state plus a rule for limiting interference or composing prior features.

These families can end with a weighted sum of low-rank outputs, but that algebra does not make their training histories equivalent.

Family	Specialist origin	Controller or constraint	State updated in the composition stage	Route lifetime	Resulting artifact
independent LoRA	one adapter trained per task	external adapter ID	selected adapter only	request	separate adapter checkpoints
static fusion	previously trained adapters	fixed coefficients	coefficients and/or merged delta	deployment artifact	one fixed effective update
Mixture-of-LoRAs	domain adapters trained first	explicit domain labels and routing	paper-specific multi-task stage	sequence/domain	specialists plus routing state
MoLE	previously trained LoRAs	learned layer-local gates	composition mechanism; specialist provenance remains distinct	input and layer	adapter pool plus gates
LoRAMoE	experts initialized for one joint run	learned soft router plus localized balancing	router and adapter experts	token and layer	one coupled routed-adapter checkpoint
HydraLoRA	asymmetric experts initialized jointly	learned router over multiple $B$ heads	shared $A$, routed $B_i$, and router	paper-specific input scope	shared factor plus specialist heads
O-LoRA	one new low-rank subspace per arriving task	orthogonality regularization	new task factors; prior factors stay frozen	task sequence	ordered adapter history
AM-LoRA	one LoRA per arriving task	attentional selector with $L_1$ pressure	current adapter and selector; prior adapters stay frozen	input and task time	adapter history plus feature selector

The table is a lifecycle taxonomy, not a claim that every implementation freezes or updates exactly the same tensors. Loop 2 freezes those identities method by method; Loops 4, 5, and 7 then trace the paper-specific training paths.

The continual branch also changes the geometry of the motivating problem. A parameter region that works for one task may not overlap the low-error region of the next task, so following the new-task gradient can leave the old solution basin. The source-local A22 figure below visualizes that motivation; it does not prove that an orthogonal factor update or an attentional mixture prevents functional forgetting.

0x00B Why Routed Adapters Appeared

A single LoRA trained on heterogeneous data must encode several potentially conflicting update directions through the same rank bottleneck. Increasing rank expands the bottleneck but does not tell the optimizer which task should own which directions. Training one adapter per domain avoids direct parameter overwrite, but it requires task identification and creates a growing collection of artifacts. Arithmetic adapter merging removes the routing decision, but it can destroy specialist behavior because the sum is applied to every input.

Routed-adapter methods explore a middle ground:

specialize:
    preserve separate low-rank updates

share:
    reuse some factors or a common base representation

select:
    choose useful updates from the current input

compose:
    combine capabilities without duplicating the base model

retain:
    keep old task state while learning new tasks sequentially

The original Transformer Series A29 introduces this family through LoRAMoE, HydraLoRA, Mixture-of-LoRAs, MoLE, and continual variants. This article reconstructs that material around the tensor, training, and runtime contracts found in the primary papers.

The source-local LoRAMoE diagram is useful because it exposes the architectural transition directly: the backbone and ordinary Transformer path remain frozen while trainable low-rank branches and a router are inserted beside the frozen FFN path. It is retained as historical evidence, not as a universal diagram for every routed-adapter method.

0x00C What This Article Owns

task-, sequence-, and token-level adapter routing
soft mixtures, top-k selection, and router-gradient behavior
independent versus shared low-rank factors
LoRAMoE localized expert roles
Mixture-of-LoRAs two-stage specialist training
MoLE post-hoc composition of trained adapters
HydraLoRA shared-A asymmetric experts
static arithmetic merging versus input-conditioned composition
continual adapter sequences, O-LoRA, and AM-LoRA
adapter provenance, storage growth, runtime kernels, and KV compatibility

It does not repeat LoRA fundamentals, variant mechanics, generic MoE balancing, expert-parallel all-to-all, or independent-adapter paging. Those remain in M6.3, M6.4, M6.2/M6.6, and M6.5. M6.7 owns the distinction between factor coordinates, matrix rank, function-space interference, and empirical retention. M6.9 may consume those contracts, but it must not turn a routed-adapter result into a new proof of low-rank theory.

The owner question for this article is therefore:

How should a frozen base model represent, train, select, compose, and preserve several low-rank capabilities when one independent adapter ID is no longer sufficient?

0x00D Evidence And Claim-Status Contract

Four evidence classes remain separate throughout the article:

Evidence class	What it can establish	What it cannot establish
source-local A22/A29 material	the handwritten lineage, terminology, and figure provenance	independent correctness or benchmark reproduction
primary paper	the method and results reported under that paper’s protocol	behavior outside the stated model, data, objective, and evaluation contract
pinned official code	what one repository revision stores and executes	a universal implementation contract or correspondence to every paper equation
local derivation or oracle	algebraic consequences and tiny deterministic checks under explicit assumptions	reproduction of paper training runs or source-reported quality gains

This distinction matters immediately. The pinned LoRAMoE code computes a dense soft mixture in its traced linear layer; that does not imply every method named “LoRA-MoE” is dense. The pinned MoLE PEFT fork constructs every listed expert result before gate weighting in the traced path; $L_1$ or gate sparsity therefore does not, by itself, prove sparse execution. The HydraLoRA paper is the canonical source for the shared-$A$ architecture; a repository path is supporting implementation evidence, not permission to infer untraced tensor sharing. O-LoRA’s orthogonality penalty appears in its trainer path and is not part of ordinary LoRA inference.

A citation proves provenance; it does not reproduce the run. Every benchmark number remains source-reported until a later loop supplies an exact model, data, seed, optimizer, hardware, and evaluation manifest.

0x00E Loop Program And Claim Ceiling

Loop 1 freezes the historical family map, paper and code revisions, source figures, consumed owners, and claims that are still forbidden. It does not certify router gradients, factor-sharing equivalence, continual retention, runtime sparsity, or paper benchmark reproduction.

The remaining loops proceed in dependency order:

Loop 2: immutable experts, factors, routers, route scope, training provenance, versions
Loop 3: common operator, router gradients, collapse telemetry
Loop 4: LoRAMoE, Mixture-of-LoRAs, and MoLE training contracts
Loop 5: HydraLoRA shared-factor derivation and coupled gradients
Loop 6: static/dynamic composition, quantization order, artifact provenance
Loop 7: O-LoRA, AM-LoRA, and continual task-state lifecycle
Loop 8: task/request/sequence/token/layer route lifetimes
Loop 9: independent/shared-factor runtime execution and M6.5 handoff
Loop 10: executable operator, gradient, merge, and resume oracles
Loop 11: complete failure, source, visual, and evaluation closure
Loop 12: final formula, source, render, accessibility, and boundary audit

At this handoff, no model has been trained or served, no paper benchmark has been reproduced, no router is proved collapse-free, and no continual method is proved to preserve every old task.

0x00F Immutable Adapter-State Manifest: Identities Before Operators

The common weighted sum in 0x01 is useful only after its operands have identities. A tensor with shape $[r,d_{in}]$ is not yet “the $A$ factor”: it belongs to one base revision, target matrix, adapter artifact, factor convention, training stage, and parent state. The same rule applies to a router. A vector of three coefficients is meaningless until its candidate pool, route scope, activation rule, tie policy, and router revision are known.

Loop 2 therefore freezes a method state before Loop 3 differentiates any operator. The minimum semantic record is:

base execution identity:
    checkpoint revision, config revision, tokenizer revision

target identity:
    module FQN, matrix role, exact input/output dimensions

adapter identity:
    artifact revision, expert-pool revision, factor IDs and convention

route identity:
    router revision, candidate pool, scope, activation, k, tie policy

training provenance:
    ordered stages, parent artifacts, frozen/trainable sets,
    supervision role, task order, source-paper and code revisions

runtime handoff:
    stored artifact IDs and semantic route state, but not a GPU slot,
    page address, kernel choice, or cache-compatibility decision

The manifest deliberately excludes physical residency. expert_2 is a semantic member of a candidate pool; slot 17 is one transient placement chosen by the M6.5 runtime. Replacing one with the other makes checkpoint replay depend on whichever process happened to load the adapter.

0x00F.1 Canonical IDs Are Derived, Not Assigned By Filename

For one low-rank factor, define a canonical semantic digest:

$$H_{factor} = H( H_{base}\,\|\,m\,\|\,q\,\|\,\operatorname{shape}\,\|\,\operatorname{dtype}\,\|\,r\,\|\,s\,\|\,H_{parent}\,\|\,H_{source})$$

where $m$ is the exact module name, $q$ is the factor role and multiplication convention, $s$ is the scale, and $H_{parent}$ identifies the state against which the factor was trained. For a route:

$$H_{route} = H(H_{router}\,\|\,H_{pool}\,\|\,\rho\,\|\,a\,\|\,k\,\|\,\pi\,\|\,\ell)$$

with route scope $\rho$, activation rule $a$, selected degree $k$, tie policy $\pi$, and supervision-role declaration $\ell$. These are teaching hash projections, not a claim that the cited projects publish this exact serialization. Their purpose is to make silent substitution fail.

Two consequences follow.

1. Shape equality is necessary for some compositions but never sufficient for artifact compatibility. An adapter trained against base@v1 is not silently re-parented to base@v2 because both expose a $4096\times4096$ matrix.
2. Factor letters are not stable identities. O-LoRA writes its update as $A_tB_t$ with $A_t\in\mathbb{R}^{d_{out}\times r}$, while this article usually writes $B_tA_t$. The manifest stores shape, role, and multiplication order rather than assuming every paper uses the same letters.

0x00F.2 The Same Forward Envelope Contains Different Method States

The six primary families and two controls freeze as follows:

Record	Factor ownership	Route contract	Training lifecycle	Supervision and inference contract
independent LoRA control	one independent $A/B$ pair per artifact	fixed adapter ID, no learned router	each specialist trained against an identified parent	request metadata selects an artifact; M6.5 owns residency
static-fusion control	compatible trained deltas plus fixed coefficients	no input-conditioned route	adapters exist before coefficient selection or merge	one materialized child revision; dynamic behavior is lost
LoRAMoE	independent $A_i/B_i$ experts at traced FFN linear layers	dense token-conditioned softmax in the paper and pinned path	base frozen; experts and router jointly trained	sample/expert type enters the localized balance objective, not the inference router input
Mixture-of-LoRAs	separately initialized domain LoRAs, then a parallel pool	sequence-level, layer-specific router with inference voting/selection	Stage 1 trains specialists separately; Stage 2 trains routers and LoRA parameters on balanced multi-domain data	domain labels supervise $L_{cls}$; they are targets, not proof of a task-ID router input
MoLE	previously trained adapter outputs remain distinct experts	input-conditioned hierarchical soft gate	base and trained LoRAs frozen; gate optimized post hoc	downstream objective plus gate-balance loss; all candidate outputs are formed in the traced path
HydraLoRA	one shared input factor $A$ and several output factors $B_i$	input-conditioned soft router over $B_i$	base frozen; shared $A$, $B_i$, and router trained jointly	claimed intrinsic components need no task labels; component count still has initialization provenance
O-LoRA	one ordered $A_t/B_t$ pair per task	no learned mixture in the reported core method	old factors frozen; new factors trained sequentially with orthogonality to prior $A_i$ subspaces	task order and cumulative parent state are part of the artifact; no task ID is required in the paper’s comparison table
AM-LoRA	one frozen LoRA per completed task, one current LoRA, one zero adapter, and per-adapter selectors	feature-conditioned softmax over task LoRA outputs	base and prior LoRAs frozen; current LoRA and attentional selector state trained at task $t$	$L_1$ pressure acts on selector parameters; it does not certify hard sparse execution

This table prevents three common historical errors. Mixture-of-LoRAs is not post-hoc gate-only composition because its second stage jointly optimizes the routing objective and LoRA parameters. MoLE is not joint routed training because its trained LoRAs are inputs to a new frozen-expert composition stage. HydraLoRA is not merely a smaller independent expert pool because all $B_i$ updates share one learned $A$ and therefore one input-side subspace.

0x00F.3 Training Labels, Router Inputs, And Expert Roles Are Different Fields

LoRAMoE defines a training sample type and a predefined expert-group type inside its localized balancing coefficient. The paper then states that inference routes automatically without prespecified data types. The manifest therefore records:

training label role: localized-balance target
router input: current hidden representation
inference task label: absent

Mixture-of-LoRAs uses domain labels to supervise a router-classification loss. That supports domain label -> auxiliary target; it does not support domain ID -> direct router input. HydraLoRA claims automatic discovery of intrinsic components, but the number and initialization of those components still come from a specified initialization procedure. A label, a loss target, a clustering seed, and a runtime route are four separate pieces of state.

0x00F.4 Ordered Continual State Cannot Be Treated As An Unordered Pool

For continual task $t$, let:

$$\mathcal{P}_t=(H_{base},H_{1},\ldots,H_{t-1})$$

be the ordered parent lineage. The next artifact identity is:

$$H_t=H(\mathcal{P}_t\,\|\,D_t\,\|\,\Theta^{train}_t\,\|\,\Theta^{frozen}_t\,\|\,H_{objective})$$

O-LoRA’s penalty refers to all previous task subspaces. AM-LoRA’s selector at task $t$ sees a zero branch, prior task features, and the current branch. Reordering task IDs after training changes the meaning of those references even if every factor file remains byte-identical. The registry must preserve task position, parent digest, and whether old state was kept separate or merged into a new base revision.

0x00F.5 A Tiny Factor Ledger Exposes The Structural Choices

Use a teaching layer with $d_{in}=6$, $d_{out}=5$, rank $r=2$, and $J=3$ experts. Three independent adapters contain:

$$P_{independent}=Jr(d_{in}+d_{out})=3\times2\times(6+5)=66$$

trainable factor scalars. Sharing $A$ gives:

$$P_{sharedA}=rd_{in}+Jrd_{out}=2\times6+3\times2\times5=42$$

whereas the shared-$B$ counterfactual gives:

$$P_{sharedB}=Jrd_{in}+rd_{out}=3\times2\times6+2\times5=46$$

The numbers are not interchangeable savings claims. Shared $A$ constrains all experts to one row space; shared $B$ constrains them to one output column space. A continual independent pool grows by $r(d_{in}+d_{out})=22$ factor scalars per task before selector, optimizer, registry, or merge-history state.

0x00F.6 Saved Route State Is Not A Runtime Or Cache Proof

A replayable routed forward needs, at minimum, the router revision, candidate-pool revision, scope, selected IDs or full soft weights, normalization state, and the mapping from route slots to semantic experts. Token- or layer-level routes may also need logits or probabilities for backward. This state can reproduce the operator decision under the declared method; it cannot establish that every expert is resident, that unselected branches were skipped, or that K/V and prefix caches are compatible.

The Loop 2 fixture therefore permits runtime fields such as execution_evidence = dense_all_candidates_in_pinned_path or sparse_execution = not_verified, but forbids physical rank, slot, page address, kernel name, latency, and throughput from entering method identity. M6.5 receives semantic artifact and route IDs, then owns placement and cache-safe serving.

Fields not fixed by a paper or inspected code remain the literal value not_verified; they are not filled with an attractive implementation guess. Loop 2 validates identities, shapes, lifecycle order, and source ceilings only. It executes no model, router, GPU kernel, training run, or paper benchmark.

0x01 A Common Operator Before The Variants Diverge

Let one frozen linear layer be:

$$Y_0=XW^{\top}, \qquad X\in\mathbb{R}^{T\times d_{in}}, \qquad W\in\mathbb{R}^{d_{out}\times d_{in}}$$

and let adapter expert $j$ own:

$$A_j\in\mathbb{R}^{r_j\times d_{in}}, \qquad B_j\in\mathbb{R}^{d_{out}\times r_j}$$

with scale $s_j$. Its output update is:

$$E_j(X)=s_jXA_j^{\top}B_j^{\top}$$

For $J$ adapters and row-dependent gates $G\in\mathbb{R}^{T\times J}$:

$$Y = XW^{\top} + \sum_{j=1}^{J} \operatorname{Diag}(G_{:,j})E_j(X)$$

or row by row:

$$y_t = Wx_t + \sum_{j=1}^{J}g_{t,j}s_jB_jA_jx_t$$

This equation is only a mathematical envelope. It leaves at least five design axes unspecified.

0x01A Routing Scope

Scope	Gate is constant over	Typical source	Runtime implication
task	every request with a known task id	metadata or external classifier	ordinary adapter selection is possible
request/sequence	all rows in one sequence	prompt representation or domain router	compute route once and persist it
token	one hidden row	per-layer hidden state	row-to-expert metadata changes at every layer
layer	selected target layer or matrix	layer-specific router	route identity is part of module state

The axes can compose. A sequence-level decision can be different at every layer. A token-level router can select two experts at each target matrix. “Sequence routing” therefore does not imply one global adapter for the whole model unless the method says so.

0x01B Activation Rule

A dense soft mixture uses:

$$g_t=\operatorname{softmax}(z_t)$$

and normally evaluates every expert with nonzero weight. A sparse mixture selects:

$$\mathcal{E}_t=\operatorname{TopK}(z_t,k)$$

then normalizes over the selected set. Sparse routing can reduce adapter-branch work, but only if the implementation avoids evaluating unselected factors. Applying a top-k mask after all $J$ low-rank products have already run saves no matmul work.

0x01C Training Provenance

Three families must remain separate:

1. Joint routed training. Router and low-rank experts are initialized for one heterogeneous run and optimized together. LoRAMoE and HydraLoRA fit this broad lifecycle.
2. Specialists first, routing second. Domain adapters are trained separately, then a router and possibly the specialists receive a balanced multi-domain pass. Mixture-of-LoRAs uses this staged structure.
3. Post-hoc composition. Existing trained adapters are inputs to a new composition mechanism. MoLE trains gates over those frozen or separately produced capabilities rather than pretending they were one jointly coordinated factorization.

The final forward pass can look similar while the learned coordinate systems and interference history are very different.

0x01D Forward Tensor Trace: Route, Shrink, Expand, Mix

The compact weighted sum hides four distinct tensor operations. Freeze one teaching layer with $T$ rows, $J$ independent adapter experts, common rank $r$, router matrix $R\in\mathbb{R}^{J\times d_{in}}$, and router bias $b\in\mathbb{R}^{J}$. The row-batch graph is:

$$Z=XR^{\top}+\mathbf{1}b^{\top} \in\mathbb{R}^{T\times J}, \qquad G=\operatorname{Route}(Z) \in\mathbb{R}^{T\times J},$$ $$U_j=XA_j^{\top} \in\mathbb{R}^{T\times r}, \qquad E_j=s_jU_jB_j^{\top} \in\mathbb{R}^{T\times d_{out}},$$ $$\bar E = \sum_{j=1}^{J}\operatorname{Diag}(G_{:,j})E_j, \qquad Y=XW^{\top}+\bar E.$$

Object	Storage axes	Semantic role	Must be saved for backward?
$X$	$[T,d_{in}]$	layer input and router input in this teaching graph	yes, unless recomputed
$Z$	$[T,J]$	pre-activation router scores	logits or sufficient activation state
$G$	$[T,J]$	dense probabilities or selected-set probabilities	yes, with selected IDs for top-$k$
$U_j$	$[T,r]$	rank-$r$ shrink result	save or recompute from $X,A_j$
$E_j$	$[T,d_{out}]$	scaled expert update	needed by the router VJP; save or recompute
$\bar E$	$[T,d_{out}]$	mixed adapter update	not a substitute for the individual $E_j$ values

For selected-softmax routing with support $\mathcal S_t=\operatorname{TopK}(z_t,k)$, the normalization domain is the selected axis:

$$g_{t,j} = \begin{cases} \dfrac{\exp z_{t,j}} {\sum_{i\in\mathcal S_t}\exp z_{t,i}}, &j\in\mathcal S_t,\\[8pt] 0,&j\notin\mathcal S_t. \end{cases}$$

This is not the same as taking a full softmax and zeroing the losers without renormalization. It is also not yet an execution plan. The Loop 3 oracle distinguishes selected_top2, which logically materializes six selected expert rows for $T=3,k=2$, from selected_top2_postmask, which produces the same numbers after all nine candidate rows have already run.

0x01E Similar Weighted Sums Still Have Different Graphs

The equations above are a local independent-factor teaching envelope, not a claim that every source method has the same router graph.

Method	What the envelope captures	What it must not replace
LoRAMoE	the paper’s dense weighted low-rank experts and the pinned layer’s hidden-state softmax	localized balancing, sample/expert role targets, and the full training contract in Loop 4
Mixture-of-LoRAs	a weighted independent-specialist output after a sequence route is broadcast	its two-stage lifecycle, domain-classification supervision, voting, and layer-specific route state
MoLE	the final weighted sum of candidate outputs	its gate input, which depends on concatenated normalized candidate outputs; the pinned path builds all candidate branches first
HydraLoRA	a weighted expert-output envelope after an expert has been defined	independent $A_j$ factors; the paper owns one shared $A$ and several $B_j$, whose coupled gradients belong to Loop 5

This graph firewall matters for both implementation and differentiation. Replacing MoLE’s gate input with $XR^{\top}$ changes the model. Applying the independent-$A_j$ gradient below to HydraLoRA duplicates a shared parameter and changes its hypothesis class.

0x02 Router Gradients Explain Collapse Better Than The Word “Balance”

For a soft router:

$$g_i=\frac{e^{z_i}}{\sum_j e^{z_j}}$$

the Jacobian is:

$$\frac{\partial g_i}{\partial z_j} = g_i(\mathbf{1}[i=j]-g_j)$$

Let the scaled adapter output for one row be:

$$e_i=s_iB_iA_ix, \qquad \bar e=\sum_i g_ie_i$$

and let upstream gradient be $\delta=\partial\mathcal{L}/\partial\bar e$. Then:

$$\frac{\partial\mathcal{L}}{\partial z_j} = g_j\,\delta^{\top}(e_j-\bar e)$$

This form exposes two failure modes.

First, if expert outputs are nearly identical, $e_j-\bar e$ is small. The router receives little evidence for specialization. Second, if one gate saturates near one, the other $g_j$ values become small and their router gradients shrink. A slightly better early expert can receive more data, improve faster, and reinforce its advantage.

Router collapse is therefore not only a histogram problem. It changes which adapters receive useful task gradients:

$$\frac{\partial\mathcal{L}}{\partial e_j}=g_j\delta$$

A nearly unused expert receives nearly no direct expert update in a dense weighted mixture. Hard top-k adds discontinuous selection and can starve unselected experts completely without auxiliary routing mechanisms.

0x02A Derive Router Credit From The Softmax VJP

The centered form should be derived, not memorized. For the whole row-batch, let

$$D=\frac{\partial\mathcal L}{\partial Y} \in\mathbb{R}^{T\times d_{out}}, \qquad q_{t,j}=\langle D_t,E_{j,t}\rangle,$$

where $q_{t,j}$ is the scalar task-loss credit that row $t$ would assign to expert $j$. Because

$$\frac{\partial\mathcal L}{\partial g_{t,j}}=q_{t,j},$$

the softmax vector-Jacobian product gives

$$\begin{aligned} \frac{\partial\mathcal L}{\partial z_{t,j}} &= \sum_i q_{t,i} \frac{\partial g_{t,i}}{\partial z_{t,j}}\\ &= \sum_i q_{t,i}g_{t,i}(\mathbf 1[i=j]-g_{t,j})\\ &= g_{t,j} \left( q_{t,j}-\sum_i g_{t,i}q_{t,i} \right). \end{aligned}$$

Define the gate-weighted mean credit

$$\bar q_t=\sum_i g_{t,i}q_{t,i} =\langle D_t,\bar E_t\rangle.$$

Then the compact VJP is

$$dZ_{t,j}=g_{t,j}(q_{t,j}-\bar q_t).$$

Only relative expert credit moves the router. Adding the same constant to every $q_{t,j}$ changes no router gradient, and every row obeys

$$\sum_j dZ_{t,j}=0.$$

This invariance is why equal expert outputs erase the task-loss routing signal. If $E_{1,t}=\cdots=E_{J,t}$, then every $q_{t,j}=\bar q_t$ and $dZ_t=0$, even when the gate distribution itself is highly nonuniform.

0x02B Expert, Factor, Router, And Input Gradients

For expert $j$, gate the upstream rows before differentiating its low-rank path:

$$\widetilde D_j = \operatorname{Diag}(G_{:,j})D \in\mathbb{R}^{T\times d_{out}}.$$

Starting from $U_j=XA_j^{\top}$ and $E_j=s_jU_jB_j^{\top}$ gives

$$dB_j = s_j\widetilde D_j^{\top}U_j \in\mathbb{R}^{d_{out}\times r},$$ $$dU_j = s_j\widetilde D_jB_j \in\mathbb{R}^{T\times r},$$ $$dA_j = dU_j^{\top}X \in\mathbb{R}^{r\times d_{in}}, \qquad dX_{\text{experts}} = \sum_j dU_jA_j.$$

The router is a second differentiable consumer of $X$. For $Z=XR^{\top}+\mathbf1b^{\top}$:

$$dR=dZ^{\top}X, \qquad db=\mathbf1^{\top}dZ, \qquad dX_{\text{router}}=dZR.$$

The frozen base still has a mathematical local derivative:

$$dW=D^{\top}X, \qquad dX_{\text{base}}=DW.$$

Freezing $W$ means that $dW$ is not retained as trainable optimizer state. It does not remove the base contribution to the input adjoint. The complete input gradient is:

dX = dX_base + dX_experts + dX_router

Dropping dX_router is a common hand-derived-backward bug: it treats route weights as constants even though the router consumed $X$. Detaching the router is a possible training design, but it is a different graph and must be declared explicitly.

0x02C Top-k Has A Piecewise Gradient Contract

Top-$k$ contains a discrete support map and a differentiable selected-softmax map. Loop 3 differentiates only the latter while holding

$$\mathcal S_t=\operatorname{TopK}(z_t,k)$$

fixed. Away from a selection boundary, the same centered-credit formula applies over the selected set:

$$dZ_{t,j} = \begin{cases} g_{t,j} \left(q_{t,j}-\sum_{i\in\mathcal S_t}g_{t,i}q_{t,i}\right), &j\in\mathcal S_t,\\[6pt] 0,&j\notin\mathcal S_t. \end{cases}$$

At a boundary where the $k$-th and $(k+1)$-th logits exchange order, this local derivative is not defined. The executable fixture uses a deterministic tie policy, descending logit followed by semantic candidate order, and verifies that every finite-difference perturbation preserves the selected IDs. That makes the test a local derivative check, not a claim that TopK itself has a conventional derivative.

The $k=1$ case exposes a stronger result. With one selected expert, $g=1$ on that support and $\bar q=q$, so

$$dZ=1\cdot(q-q)=0$$

for the task loss inside a fixed support region. A hard top-1 router therefore needs some other learning path if the selection operation itself is not relaxed: a straight-through estimator, noisy or differentiable routing during training, an auxiliary router objective, reinforcement-style credit, or soft training followed by hard inference. These are alternative estimators and training contracts, not algebraically equivalent implementations.

0x02D Executable Oracle And Collapse Witnesses

The exact Loop 3 fixture uses

$$T=3, \quad d_{in}=6, \quad d_{out}=5, \quad r=2, \quad J=3.$$

It checks 135 scalar coordinates per policy across $X,W,R,b,A_1\ldots A_3,B_1\ldots B_3$. Dense, selected top-2, and selected top-1 therefore contribute 405 centered finite-difference checks.

Policy	Selected IDs by row	Loss	Mean gate entropy	Materialized expert rows	Maximum gradient error
dense_soft	all three	1.319343617	0.786831	9	$4.02\times10^{-10}$
selected_top2	$[2,0],[1,0],[0,2]$	1.307927831	0.572245	6	$2.25\times10^{-10}$
selected_top2_postmask	same selected weights	1.307927831	0.572245	9	same analytic graph; work differs
selected_top1	$[2],[1],[0]$	1.232500000	0	3	$1.88\times10^{-10}$
selected_topJ	all three, score order	1.319343617	0.786831	9	exactly matches dense output

The selected top-2 boundary margin is at least $0.85$ in the fixture, so the perturbations do not cross a route boundary. selected_top2 and selected_top2_postmask have identical gates, outputs, losses, and local gradients. Only the declared materialization ledger differs. This is the smallest counterexample to the claim that a top-$k$ mask proves sparse execution.

Three additional probes isolate collapse mechanisms:

1. Identical experts. Copying one expert output into all three slots leaves the maximum router-logit gradient below $4.9\times10^{-19}$. The task loss has no routing evidence.
2. Saturated router. Logits $[16,0,-16]$ produce probabilities approximately $[0.999999887,1.13\times10^{-7},1.27\times10^{-14}]$. The weakest coordinate’s absolute gradient falls below $5\times10^{-16}$ in the probe.
3. Top-1 starvation. A fixed route selecting expert 0 for all three rows gives $dZ=0$ and exactly zero $dA_j,dB_j$ for experts 1 and 2.

These witnesses do not prove that a paper checkpoint collapses. They prove that the common operator contains mechanisms through which collapse or starvation can occur.

0x02E Mechanism Pseudocode

def routed_adapter_forward(X, W, R, b, factors, policy):
    # X [T, d_in], R [J, d_in], Z/G [T, J]
    Z = X @ R.T + b
    support = route_support(Z, policy, tie_break="semantic_candidate_order")
    G = softmax_over_selected_logits(Z, support)

    Y = X @ W.T
    saved = {"X": X, "Z": Z, "G": G, "support": support, "experts": []}
    for j in materialized_experts_by_row(support, policy):
        A_j, B_j, scale_j = factors[j]
        U_j = X_for_rows(j, support) @ A_j.T
        E_j = scale_j * (U_j @ B_j.T)
        rows = rows_for(j, support)
        scatter_add(Y, rows, G[rows, j, None] * E_j)
        saved["experts"].append((j, U_j, E_j, rows))
    return Y, saved

def routed_adapter_backward(D, W, R, factors, saved):
    X, G, support = saved["X"], saved["G"], saved["support"]
    q = expert_credit(D, saved["experts"])       # q[t,j] = dot(D[t], E_j[t])
    q_bar = (G * q).sum(axis=-1, keepdims=True)
    dZ = where_support(G * (q - q_bar), support, 0)

    dR = dZ.T @ X
    db = dZ.sum(axis=0)
    dX_router = dZ @ R

    dX_experts = zeros_like(X)
    for j, U_j, E_j, rows in saved["experts"]:
        A_j, B_j, scale_j = factors[j]
        D_j = G[rows, j, None] * D[rows]
        dB_j = scale_j * D_j.T @ U_j
        dU_j = scale_j * D_j @ B_j
        dA_j = dU_j.T @ X[rows]
        scatter_add(dX_experts, rows, dU_j @ A_j)

    dX_base = D @ W
    dX = dX_base + dX_experts + dX_router
    return dX, dR, db

For a dense paper path, materialized_experts_by_row returns all candidates. For a selected execution plan it returns only selected row/expert pairs. A framework implementation may recompute $E_j$ during backward instead of saving it, but it must still recover the individual expert credits needed by $dZ$.

0x02F What To Measure

One utilization chart is insufficient. A routed-adapter training run should record:

mean gate probability per expert
top-k selection frequency per expert
gate entropy by layer and data domain
expert-output norm and pairwise similarity
gradient norm per adapter factor
task-to-expert mutual information
router classification accuracy when labels exist
quality with router frozen, uniform, shuffled, or oracle-routed
minimum top-k boundary margin and route-flip rate
router-logit gradient norm per expert
zero-gradient expert count by layer and step
expert rows materialized before and after selection

Uniform utilization is not automatically desirable. If domains are imbalanced or one shared expert is intentionally general, a uniform target can oppose specialization. Balance losses encode a prior about expert roles and should be evaluated as such.

0x03 Independent Adapter Experts: Similar Shape, Different Objectives

0x03A LoRAMoE: Joint Experts With Localized Roles

LoRAMoE inserts multiple low-rank experts beside frozen FFN linear layers and learns a soft router. In the paper’s notation, the layer is:

$$o = W_0x + \frac{\alpha}{r} \sum_{i=1}^{N}\omega_iB_iA_ix$$

where all $N$ expert outputs participate in the written formulation. This is adapter conditional composition, but it is not automatically sparse top-k execution.

The paper targets a specific failure: large instruction-tuning mixtures can improve downstream instruction following while damaging performance on tasks that depend on pretrained world knowledge. Freezing the backbone prevents direct overwrite of $W_0$, but a large learned adapter update can still alter the model’s function enough to hide or disrupt that knowledge. LoRAMoE therefore divides experts into two semantic groups:

knowledge-oriented experts:
    trained to leverage world-knowledge-related data

downstream-task experts:
    trained to specialize on other instruction tasks

For sample $m$ with $T_m$ tokens, define router importance for expert $n$:

$$Q_{n,m} = \sum_{j=1}^{T_m}G(x_j)_n$$

The method defines a coefficient matrix $I$ using known training sample and expert group types:

$$I_{n,m} = \begin{cases} 1+\delta,&\operatorname{Type}_e(n)=\operatorname{Type}_s(m)\\ 1-\delta,&\text{otherwise} \end{cases}$$

Then:

$$Z=I\odot Q, \qquad \mathcal{L}_{lbc}=\frac{\operatorname{Var}(Z)}{\operatorname{Mean}(Z)}$$

and:

$$\mathcal{L}_{total} = \mathcal{L}_{LM} + \beta\mathcal{L}_{lbc}$$

The localized balancing constraint is not unsupervised expert discovery: it uses sample-type and expert-group assignments during training. At inference, the learned router operates without those labels. The soft constraint also tries to balance experts within a role while biasing the two role groups differently.

This design should be compared against at least four controls:

one mixed LoRA
multiple experts without localized balancing
ordinary global balance loss
oracle route from known sample type

Without these controls, retention improvements cannot be attributed specifically to localized roles.

0x03B Mixture-of-LoRAs: Specialists Before The Router

Mixture-of-LoRAs starts from $N$ task/domain adapters trained separately. The second stage installs those adapters in parallel and learns sequence-level routing using balanced multi-domain data and domain labels. Its training objective combines language modeling and router classification.

Using the factor convention of this series:

$$y = W_0x + \sum_{i=1}^{N}g_i(s)B_iA_ix$$

where $g_i(s)$ is associated with a sequence or sequence representation rather than independently changing for every token. The paper discusses inference voting or selecting a final expert decision to improve contextual consistency and avoid repeating an unstable token-level decision throughout generation.

The staged training has a useful property: every specialist begins from a competent domain solution instead of waiting for a joint router to give it enough traffic. It also has a cost: the initial specialists were optimized independently, so their scales and functional effects may differ. The second stage must learn both routing and cross-domain compatibility.

0x03C MoLE: Compose Existing LoRAs Without Arithmetic Destruction

MoLE addresses a different problem: several already trained LoRAs must be combined while preserving their characteristics. The paper contrasts this with direct arithmetic merging, which produces one fixed update whether or not every input needs every capability.

At a transformer-block level, MoLE forms trained adapter outputs $E_{\Delta\theta_i}(x)$, concatenates and normalizes them, and trains a gate. The final composition is:

$$\widetilde E_{\Omega}(x) = \sum_{i=1}^{N}G_i(x)E_{\Delta\theta_i}(x)$$ $$O(x)=F_{\theta}(x)+\widetilde E_{\Omega}(x)$$

Its gate-balancing term uses average gate probability $q(i)$ across the gated blocks:

$$\mathcal{L}_{balance} = -\log\prod_{i=1}^{N}q(i) = -\sum_{i=1}^{N}\log q(i)$$

This objective strongly penalizes an expert whose average probability approaches zero. The overall objective is domain loss plus a weighted balance term.

MoLE’s paper explores hierarchical control at block, layer, or matrix granularity. That flexibility changes cost substantially. Gating complete block-level adapter outputs can require running every candidate path before the gate has the features it consumes. A matrix-level implementation can instead gate low-rank branches inside a shared base forward. The method name alone does not identify the runtime.

0x03D Mechanism Comparison

Method	Adapter origin	Router supervision	Route scope in the paper	What remains trainable in composition stage	Primary concern
LoRAMoE	joint initialization	sample-role signal enters localized balance	token/layer soft weights	router and LoRA experts	world-knowledge retention under broad SFT
Mixture-of-LoRAs	separately trained domain LoRAs	explicit domain classification	sequence-level, layer-specific routers	router plus multi-task adaptation stage	heterogeneous task interference
MoLE	already trained LoRAs	downstream objective plus gate balance	hierarchical, input-conditioned composition	composition gates; adapter lifecycle is separate	post-hoc capability composition
HydraLoRA	joint asymmetric factors	task labels not required for claimed intrinsic components	input-conditioned soft mixture	shared A, B experts, router	heterogeneous corpus interference with fewer factors

The benchmark numbers in these papers are source-reported under different bases, tasks, budgets, and training procedures. They are not a ranking table.

0x03E One Phrase, Six Training States

“A mixture of LoRAs” is not a sufficient training specification. The three methods above require at least six distinct states once the paper graph, stage order, and pinned implementations are kept separate:

State	Parent artifact	Trainable state	Privileged training signal	Result
LoRAMoE paper graph	pretrained base	all routed LoRA experts and routers	sample role and predefined expert role inside localized balance	one coupled router-expert checkpoint
LoRAMoE pinned graph	pretrained base	all routed LoRA experts and routers	task_types inside each traced balance term	coupled checkpoint under a code-local objective
Mixture Stage 1	same base reloaded for each domain	one domain LoRA per run	domain-specific language-model data	ordered pool of independent specialists
Mixture Stage 2	ordered Stage 1 pool	every domain LoRA and every layer router	balanced domain samples and domain-classification targets	jointly reconciled pool and routers
MoLE paper graph	trained adapter pool	candidate-output gate and paper temperature	domain loss and gate-balance loss	composition state plus immutable pool identity
MoLE pinned V&L graph	trained adapter pool	parameters admitted by the pinned wg name filter	task loss plus an optional recorded statistic	implementation-specific gate state

These states are not interchangeable. Loading the same adapter tensors with a different trainable set changes the optimization problem. Moving a domain label from a loss target into the router input changes the information available at inference. Moving a gate before candidate evaluation changes both the function and the execution ledger. Even two expressions called “balance loss” can produce different gradients.

The immutable transition graph is:

LoRAMoE:
    pretrained base
        -> initialize router + all low-rank experts
        -> one joint optimizer state
        -> coupled routed-adapter checkpoint

Mixture-of-LoRAs:
    pretrained base --domain 0 run--> adapter 0
    pretrained base --domain 1 run--> adapter 1
    ...
    ordered adapter pool
        -> balanced Stage 2, adapters + routers trainable
        -> reconciled adapters + routers

MoLE:
    pretrained base + ordered trained adapter pool
        -> freeze both capability sources
        -> optimize composition state
        -> gate state + exact adapter-pool identity

The arrows transfer parameter values and artifact identities. They do not prove that Adam moments, learning-rate schedules, dataloader cursors, random-number streams, or gradient-scaler state cross a stage boundary. Unless a source explicitly serializes and restores those objects, optimizer continuity must remain unspecified.

0x03F LoRAMoE: Paper Objective Versus Pinned Code

The paper and the pinned repository agree on the main graph: a frozen base linear map, independently parameterized low-rank experts, a hidden-state softmax router, and joint training of experts and router. They do not implement the localized balancing expression identically.

Let a minibatch contain $B$ samples, sample $b$ contain $T_b$ valid tokens, and let there be $J$ experts. For router probability $g_{btj}$, aggregate each sample’s traffic:

$$Q_{bj} = \sum_{t=1}^{T_b}g_{btj}.$$

Let $s_b$ be the known training role of sample $b$ and $e_j$ the predefined role of expert $j$. The paper constructs:

$$I_{bj} = \begin{cases} 1+\delta,&s_b=e_j,\\ 1-\delta,&s_b\ne e_j, \end{cases} \qquad Z_{bj}=I_{bj}Q_{bj}.$$

The printed paper equation is:

$$\mathcal L_{lbc}^{paper} = \frac{\operatorname{Var}(Z)}{\operatorname{Mean}(Z)}.$$

Var does not, by itself, specify a population or sample correction. The executable teaching oracle declares population variance so that every denominator is explicit. Flatten $Z$ into $N=BJ$ values, define

$$\mu=\frac{1}{N}\sum_{n=1}^{N}z_n, \qquad v=\frac{1}{N}\sum_{n=1}^{N}(z_n-\mu)^2, \qquad \mathcal L_{paper}^{teach}=\frac{v}{\mu}.$$

Since $\sum_n(z_n-\mu)=0$,

$$\frac{\partial v}{\partial z_n} = \frac{2(z_n-\mu)}{N},$$

and the quotient rule gives:

$$\frac{\partial \mathcal L_{paper}^{teach}}{\partial z_n} = \frac{2(z_n-\mu)}{N\mu} - \frac{v}{N\mu^2}.$$

The pinned cv_squared path instead evaluates PyTorch’s default sample variance and divides by squared mean plus epsilon:

sample variance / (mean^2 + epsilon)

In symbols,

$$v_s = \frac{1}{N-1}\sum_{n=1}^{N}(z_n-\mu)^2, \qquad D=\mu^2+\epsilon, \qquad \mathcal L_{lbc}^{code}=\frac{v_s}{D}.$$

Therefore:

$$\frac{\partial \mathcal L_{lbc}^{code}}{\partial z_n} = \frac{ \frac{2(z_n-\mu)}{N-1}D - v_s\frac{2\mu}{N} }{D^2}.$$

The two gradients are not generally scalar multiples: one denominator is $\mu$, the other is $\mu^2+\epsilon$, and the variance normalization also differs. In the checked fixture, the same gates produce:

Quantity	Paper-form teaching convention	Pinned-code convention
$\mu$	0.517849	0.517849
variance	0.071864 population	0.082130 sample
localized objective	0.138774	0.306265

That is a mechanism discrepancy, not a benchmark accusation. The paper states the intended method; the pinned revision states what that code path computes.

The remaining chain rule is common. Let

$$h_{bj} = I_{bj}\frac{\partial \mathcal L_{lbc}}{\partial Z_{bj}}.$$

Because $Q_{bj}=\sum_t g_{btj}$, every valid token in sample $b$ receives the same upstream gate credit $h_{bj}$. For router logits $\ell_{btj}$:

$$\frac{\partial \mathcal L_{lbc}}{\partial \ell_{btj}} = g_{btj} \left( h_{bj} - \sum_{k=1}^{J}g_{btk}h_{bk} \right).$$

This localized gradient is added to the task-loss router credit derived in 0x02. In the pinned LLaMA path, the local scalar is returned by each routed gate_proj, up_proj, and down_proj, accumulated across layers, and finally added to causal-language-model cross entropy. Consequently, blc_weight is effectively applied many times through local terms; it is not one single model-level statistic formed after pooling every route.

def loramoe_joint_step(batch, model, beta):
    # sample_role is training-only metadata, not a router input.
    token_loss, local_balance_terms = model(
        input_ids=batch.input_ids,
        labels=batch.labels,
        task_types=batch.sample_role,
    )

    # Paper graph: one declared model-level localized objective.
    # Pinned graph: layer/projection-local CV-squared terms are returned.
    loss = token_loss + beta * aggregate_under_declared_contract(local_balance_terms)
    loss.backward()
    optimizer.step()  # router and all A_i/B_i; frozen base has no optimizer update

One final provenance trap matters. The paper’s reported main configuration uses six experts and nonzero localized-balance coefficients. The pinned repository’s inspected shell script defaults to eight experts and sets both balance coefficients to zero. That script is a code example at a pinned revision, not an automatic reproduction command for the paper’s main result.

0x03G Mixture-of-LoRAs: Two Optimizer State Machines

Mixture-of-LoRAs is not “load several adapters and train a router.” Its paper contract has two optimization stages.

Stage 1: independent specialists. For domain $j$, reload the same base checkpoint, attach one LoRA $\phi_j=(A_j,B_j)$, freeze the base, and optimize only $\phi_j$ on domain $j$. The parent of adapter $j$ is the base checkpoint, not adapter $j-1$:

specialists = []
for domain in ordered_domains:
    model = load_same_frozen_base(base_checkpoint)
    adapter = initialize_lora()
    optimizer = new_optimizer(adapter.parameters())
    train_language_model(model, adapter, domain.dataset, optimizer)
    specialists.append(save_adapter(adapter, domain_id=domain.id))

This produces an ordered adapter pool. Ordering matters because domain-classification targets and router outputs must refer to the same expert indices.

Stage 2: balanced joint reconciliation. Load all Stage 1 adapters in parallel, add a distinct sequence router to every traced Transformer layer, freeze the base, and optimize both the routers and LoRA parameters. The paper evenly samples domains for this stage. For batch item $b$, layer $\ell$, and domain label $c_b$, let router logits be $z_{b\ell j}$ and probabilities $g_{b\ell j}$.

Using conventional minimized negative log likelihood, the classification term is:

$$\mathcal L_{cls} = -\frac{1}{BL} \sum_{b=1}^{B}\sum_{\ell=1}^{L} \log g_{b\ell,c_b}.$$

The paper gives no display for $\mathcal L_{cls}$; it describes it in prose as the cross-entropy loss of expert classification. Its one displayed sum of log probabilities is the language-modeling loss, $\sum_i\log P(x_i\mid x_{<i})$, printed without a minus sign while called a loss. The oracle uses the conventional negative sign required by minimization for the classification term above, writes the language-model term as a negative log likelihood $\mathcal L_{NLL}$, and records that notation choice rather than silently treating the printed sign as unambiguous. The combined objective is:

$$\mathcal L_{stage2} = \mathcal L_{NLL} + \eta\mathcal L_{cls}.$$

The classification gradient is exact:

$$\eta\frac{\partial \mathcal L_{cls}}{\partial z_{b\ell j}} = \frac{\eta}{BL} \left(g_{b\ell j}-\mathbf 1[j=c_b]\right).$$

That is only the classification part of the Stage-2 router gradient. The language-model loss also depends on the router, through the gate-weighted adapter sum of 0x03B, so the full gradient adds the softmax-VJP credit of 0x02A. Because one sequence gate serves every token row of sample $b$ at layer $\ell$, that credit sums over the sample’s rows $t$:

$$\frac{\partial \mathcal L_{stage2}}{\partial z_{b\ell j}} = g_{b\ell j} \left(q_{b\ell j}-\sum_i g_{b\ell i}q_{b\ell i}\right) + \frac{\eta}{BL} \left(g_{b\ell j}-\mathbf 1[j=c_b]\right), \qquad q_{b\ell j}=\sum_t\langle D_{b\ell t},E_{j,b\ell t}\rangle,$$

where $E_{j,b\ell t}$ is expert $j$‘s scaled adapter output for row $t$ at layer $\ell$ and $D_{b\ell t}$ is the gradient of $\mathcal L_{stage2}$ at that layer’s routed output. $D$ carries the language-model loss and, wherever a later layer’s router reads a hidden state downstream of layer $\ell$, that layer’s classification term as well. The checked fixture in 0x03J holds the NLL value fixed, so it finite-differences the classification part alone.

Domain labels are targets, not router inputs. At inference, the model receives the sequence representation available to the router, not $c_b$. This distinction prevents an evaluation leak in which an oracle domain ID is accidentally supplied to a model advertised as automatic routing.

def mixture_stage2_step(batch, base, adapter_pool, layer_routers):
    assert balanced_domain_histogram(batch.domain_label)
    freeze(base)
    unfreeze(adapter_pool)   # paper Stage 2 updates LoRA parameters too
    unfreeze(layer_routers)

    logits, router_logits = forward_with_sequence_routes(
        base, adapter_pool, layer_routers, batch.tokens
    )
    nll = causal_lm_loss(logits, batch.labels)
    cls = mean_cross_entropy_over_layers(router_logits, batch.domain_label)
    (nll + eta * cls).backward()
    stage2_optimizer.step()

The paper describes layer routers through both a compact weight notation and a later two-layer-MLP description. The invariant is the interface — one sequence-level domain distribution per layer — not an unsupported claim that every implementation must use one exact router architecture.

Generation adds another policy choice. Layer decisions can vote for one adapter, or the last router can provide the final selection. These are not equivalent. In the oracle, layer choices [2, 2, 0] yield expert 2 by majority vote and expert 0 by last-router selection.

def choose_generation_adapter(layer_choices, policy):
    if policy == "router_vote":
        return deterministic_majority_vote(layer_choices)
    if policy == "last_router":
        return layer_choices[-1]
    raise ValueError("policy is part of the deployment contract")

The selection policy, tie break, adapter ordering, and router version must therefore be saved with the Stage 2 artifact. Saving only the LoRA factors loses part of the model.

0x03H MoLE: Paper Gate Graph Versus Pinned V&L Graph

MoLE begins after specialist training. The base and candidate LoRAs are capability inputs to a new composition stage; they are not reinterpreted as jointly initialized experts.

For the paper graph, every candidate first produces $E_i(x)$. The candidates are concatenated and normalized:

$$E_{\Omega}(x) = \operatorname{Norm} \left(E_1(x)\oplus\cdots\oplus E_J(x)\right).$$

After flattening, a learned map produces candidate logits $\varepsilon_i$. With paper temperature $\tau$:

$$g_i = \frac{\exp(\varepsilon_i/\tau)} {\sum_j\exp(\varepsilon_j/\tau)}, \qquad \widetilde E(x)=\sum_i g_iE_i(x).$$

This dependency fixes the execution order:

all candidate outputs
    -> concatenate and normalize
    -> gate logits and temperature
    -> softmax weights
    -> weighted candidate output

It is impossible to skip an unselected candidate before computing a gate whose input contains that candidate. A different hidden-state router may permit another execution plan, but it is a different graph.

Across $M$ gated blocks, define:

$$q_i=\frac{1}{M}\sum_{m=1}^{M}g_{mi}, \qquad \mathcal L_{balance}=-\sum_{i=1}^{J}\log q_i.$$

The direct probability gradient is:

$$\frac{\partial \mathcal L_{balance}}{\partial g_{mi}} = -\frac{1}{Mq_i}.$$

For total gate-probability credit $u_{mi}$ from the domain objective plus $\alpha\mathcal L_{balance}$, the logit gradient is again the softmax VJP:

$$\frac{\partial \mathcal L}{\partial a_{mi}} = g_{mi} \left(u_{mi}-\sum_j g_{mj}u_{mj}\right), \qquad a_{mi}=\frac{\varepsilon_{mi}}{\tau}.$$

If $\tau$ is trainable in the paper graph, its local derivative is:

$$\frac{\partial \mathcal L}{\partial \tau} = -\frac{1}{\tau^2} \sum_{m,i} \frac{\partial \mathcal L}{\partial a_{mi}}\varepsilon_{mi}.$$

The pinned V&L paths inspected for this loop are different in three ways:

1. Their traced gate is formed from normalized current hidden states rather than concatenated candidate outputs.
2. The PEFT fork still materializes all listed candidate LoRA outputs before weighting and reduction.
3. The inspected trainer admits parameters whose names contain wg; the observed temperature state is therefore outside that trainable set even though the paper calls $\tau$ learnable.

The optional entropy path also requires a differentiability audit. In the pinned trace, gate statistics are detached, accessed through .data, converted to Python lists, and reconstructed as a new tensor in the trainer. An arithmetic value can still be added to the scalar loss, but no autograd edge leads from that reconstructed value back to the gate logits. It is a disconnected diagnostic, not a differentiable reproduction of the paper’s gate-balance objective in that path.

def mole_paper_gate_step(x, frozen_base, frozen_candidates, gate, tau):
    candidate_outputs = [candidate(x) for candidate in frozen_candidates]
    gate_input = normalize(concat(candidate_outputs))
    weights = softmax(gate(flatten(gate_input)) / tau)
    output = frozen_base(x) + weighted_sum(weights, candidate_outputs)
    loss = domain_loss(output) + alpha * balance_loss(weights)
    loss.backward()
    gate_optimizer.step()  # gate and paper tau only


def pinned_vl_path(x, frozen_candidates, wg):
    weights = softmax(wg(normalize(x)))
    candidate_outputs = [candidate(x) for candidate in frozen_candidates]
    output = weighted_sum(weights, candidate_outputs)

    recorded = detach(weights)
    entropy = tensor(product(to_python_list(recorded)))
    assert not entropy.requires_grad  # diagnostic cannot train wg in this path
    return output, entropy

This comparison is revision-specific. It does not claim that every MoLE implementation has the same hidden-state gate or detached statistic.

0x03I Objective Gradients And Frozen-State Semantics

“Frozen” constrains parameter updates; it does not erase a module from the computational graph.

Suppose a frozen candidate computes $E_i(x;\phi_i)$ and a trainable gate consumes that output. Setting phi_i.requires_grad = False means:

no dL/dphi_i accumulation
no optimizer slot for phi_i
no optimizer update to phi_i

It does not imply:

detach E_i from x
discard dL/dE_i
remove the candidate from the gate input
skip dL/dx through the frozen function

Those operations would change how earlier trainable composition state receives credit. In a hierarchy with trainable gates at several blocks, later frozen base and adapter maps may still carry gradients to earlier gates.

Method state	Parameter gradients	Input-gradient path	Optimizer state
LoRAMoE joint	router and every $A_i/B_i$	base, experts, and router dependence	one coupled optimizer state
Mixture Stage 1	one specialist $A_j/B_j$	through frozen base and active adapter	new independent optimizer per specialist run
Mixture Stage 2	all specialists and layer routers	through weighted multi-adapter graph	new or explicitly restored Stage 2 state; paper weight handoff alone does not prove moment continuity
MoLE paper stage	composition gate and $\tau$	through frozen candidates where needed	gate-only optimizer state
pinned MoLE trace	wg-admitted parameters	hidden-state gate and dense candidate reduction	revision-specific; detached diagnostic has no gate gradient

The training artifact must therefore record at least:

base checkpoint identity
ordered adapter identities and factor shapes
router/gate architecture and scope
trainable parameter-name manifest
frozen parameter-name manifest
objective formula and reduction axes
auxiliary-label schema and availability phase
stage-parent artifact digest
optimizer-state restore policy
inference selection policy

0x03J Executable Stage Oracle

The checked teaching fixture uses small float64 logits, not paper checkpoints. It answers four local questions exactly:

Contract	Coordinates finite-differenced	Maximum absolute error
LoRAMoE printed-form balance under declared population variance	16	$3.70\times10^{-11}$
LoRAMoE pinned CV-squared form under sample variance	16	$1.18\times10^{-10}$
Mixture domain-classification term	27	$4.17\times10^{-11}$
MoLE linearized domain credit plus balance	9	$1.30\times10^{-10}$

For the Mixture fixture, one sample from each of three domains gives the immutable histogram [1, 1, 1]. The mean layer/domain classification loss is 0.694150; with $\eta=0.4$, it contributes 0.277660 to the teaching total. For MoLE, mean gate probabilities are approximately [0.326606, 0.340376, 0.333018], so the balance term is 3.296264 before weighting.

These checks establish algebra and state ownership only. They do not execute framework autograd, a GPU kernel, a paper checkpoint, an optimizer schedule, or a benchmark. They also do not prove that localized balance prevents forgetting, that balanced routing improves quality, or that dense soft composition skips any physical work.

The replay sequence is:

python3 docs/numinens-evidence/m6-9-routed-shared-continual-adapters/build_loop4_artifacts.py
python3 -m unittest docs/numinens-evidence/m6-9-routed-shared-continual-adapters/test_loop4_method_training_contract.py

0x03K Failure Lab And Measurements

Broken contract	Observable symptom	Root cause	Required measurement or rejection
paper and code LoRAMoE losses collapsed into one formula	reproduced values disagree despite matching gates	variance correction or mean denominator changed	log $\mu$, variance convention, denominator, and every local term
LoRAMoE role supplied at inference	unrealistically clean routes	privileged training metadata leaked into runtime	reject any inference schema containing sample role
Mixture Stage 2 freezes specialists	router learns around incompatible adapter scales but specialists cannot reconcile	wrong trainable set	parameter-delta and optimizer-slot audit by adapter
Mixture Stage 2 uses naturally imbalanced data	router classifier learns corpus frequency	balanced paper contract lost	per-domain sample and token histogram per optimizer window
Mixture domain label fed to router	routing accuracy appears perfect	target became an input feature	ablate labels and inspect model input signature
vote and last-router policy treated as identical	adapter changes after deployment despite same weights	unversioned inference policy	disagreement rate and deterministic tie-policy test
MoLE paper gate moved before candidate outputs	lower compute but different function	gate dependency rewritten	source-to-tensor trace and candidate materialization count
frozen MoLE candidates detached wholesale	earlier gates receive incomplete gradients	parameter freezing confused with graph detachment	input-gradient norm before and after candidate blocks
pinned entropy statistic called a training loss	scalar changes but gate gradients do not	detached/reconstructed tensor	requires_grad, grad_fn, and gate-gradient ablation
paper temperature called trainable in pinned run	$\tau$ never changes	parameter-name filter excludes it	trainable-name manifest and before/after value digest

For real runs, record method-specific telemetry rather than one generic utilization plot:

LoRAMoE:
    per-layer and per-projection LM loss, local balance term, mu, variance,
    sample/expert role traffic, router entropy, role-label ablation,
    every expert A/B gradient norm, and base-weight immutability

Mixture-of-LoRAs:
    Stage 1 parent digest per specialist, domain competence before Stage 2,
    balanced sample/token counts, per-layer classification accuracy,
    adapter and router parameter deltas, vote/last disagreement,
    and cross-domain regression after joint reconciliation

MoLE:
    exact ordered adapter pool, candidate outputs materialized,
    gate input source, tau trainability, q_i and balance gradient,
    gate-only parameter delta, detached-statistic audit,
    and quality against static arithmetic merge and single-adapter controls

The common evaluation rule is causal: compare against a single adapter, an ungated uniform mixture, a router with its auxiliary objective removed, and an oracle route only when the oracle’s privileged information is clearly labeled. Otherwise a gain cannot be attributed to routing, balancing, or composition separately.

0x04 HydraLoRA: Sharing One Factor Changes The Hypothesis Class

HydraLoRA begins from a concrete adaptation problem rather than from the MoE analogy. A heterogeneous corpus can contain tasks or subdomains whose updates interfere, yet allocating one complete LoRA pair to every component multiplies adapter state. The paper asks whether the two LoRA factors play sufficiently different roles that only one of them needs to be replicated.

Its answer is an asymmetric ownership contract: retain one input-side factor $A$, allocate several output-side factors $B_i$, and learn an input-conditioned router over those $B_i$ factors. This is not ordinary independent-expert LoRA with fewer labels. Sharing one parameter object changes which subspaces can be represented, which gradients are coupled, which optimizer moments exist, and which computations can in principle be reused.

Source: HydraLoRA, Figure 1. The progression is structural: one LoRA pair, several independent pairs under the same parameter budget, then one shared input factor with several output factors. The letters are paper-local; tensor shapes and multiplication order determine what is actually shared.

0x04A From An Empirical Plot To A Sharing Hypothesis

The paper reaches the asymmetric design in two observational steps. First, under its reported setups, splitting a fixed LoRA parameter budget into several smaller task-specific heads can outperform one monolithic head. The paper interprets this as evidence that heterogeneous data can create update interference. Second, it separately trains LoRA heads for Dolly-15K subsets and visualizes flattened $BA$, $A$, and $B$ module parameters with t-SNE. The plotted $A$ points overlap more across tasks while the $B$ points are more distinguishable.

Source: HydraLoRA, Figure 3, recovered from the official repository at commit afb693b. The reported setup flattens 32 layers times four q/v LoRA submodules for LLaMA2-7B and compares task-specific points. This is empirical motivation for shared $A$, not a gauge-invariant proof that $A$ universally stores common knowledge.

That qualification matters because the LoRA factorization is not unique. For any invertible $S\in\mathbb{R}^{r\times r}$,

$$(BS^{-1})(SA)=BA.$$

The function is unchanged while the raw coordinates of both factors can move substantially. In the Loop 5 witness, one such transform changes $A$ by $0.2717$ in Frobenius norm while changing the output by only $1.11\times10^{-16}$. A coordinate plot, cosine similarity, or Euclidean distance between unaligned factors is therefore gauge-dependent. A stronger follow-up would compare row- and column-space projectors or principal angles, align factors before comparing them, repeat across seeds and model families, and verify the architectural choice with equal-budget ablations.

The paper itself is appropriately cautious: it attributes part of the observed asymmetry to the usual nonzero-$A$/zero-$B$ initialization and notes that behavior can vary across architectures and datasets. Transformer System consequently treats the plot as a design hypothesis with source-reported evidence, not as an algebraic theorem.

0x04B Exact Ownership And Parameter Ledgers

Let the frozen target matrix be $W_0\in\mathbb{R}^{d_{out}\times d_{in}}$, the shared input factor be $A\in\mathbb{R}^{r\times d_{in}}$, and each expert readout be $B_i\in\mathbb{R}^{d_{out}\times r}$. For one token column $x$, HydraLoRA’s adapter path is

$$\Delta y(x) = s\sum_{i=1}^{J}g_i(x)B_iAx, \qquad g(x)=\operatorname{softmax}(Rx),$$

where $R\in\mathbb{R}^{J\times d_{in}}$ in the pinned main implementation and $s=\alpha/r$. Three superficially similar designs have different trainable object graphs:

Design	Factor objects	Factor parameters	Structural assumption
Independent	$A_i,B_i$ for every expert	$Jr(d_{in}+d_{out})$	each expert chooses input and output subspaces
Shared $A$	one $A$, distinct $B_i$	$rd_{in}+Jrd_{out}$	all experts share one input row space
Shared $B$	distinct $A_i$, one $B$	$Jrd_{in}+rd_{out}$	all experts share one output column space

The shared-$A$ saving relative to independent experts follows by cancellation:

$$\begin{aligned} P_{independent}-P_{sharedA} &=Jr(d_{in}+d_{out})-(rd_{in}+Jrd_{out})\\ &=(J-1)rd_{in}. \end{aligned}$$

The router is a separate ledger. The pinned bias-free projection adds $Jd_{in}$ parameters per targeted linear. For the exact teaching dimensions $d_{in}=4$, $d_{out}=3$, $r=2$, and $J=3$:

Design	Factors	Router	Factors + router	Adam-like factor state: weight + two moments
Independent	42	12	54	126 elements
Shared $A$	26	12	38	78 elements
Shared $B$	30	12	42	90 elements

Those persistent-state counts intentionally omit gradients, mixed-precision master copies, allocator padding, and the frozen base. They isolate the ownership change. They also show why an equal-rank comparison is not an equal-parameter comparison. Under a factor-only budget of 120 elements, the largest integer ranks in this toy geometry are 5 for independent experts, 9 for shared $A$, and 8 for shared $B$. A fair paper comparison must state which budget is held fixed.

0x04C Hypothesis-Class Restrictions And Counterexample

Parameter saving is purchased with a precise representational restriction. Each shared-$A$ expert update is

$$\Delta W_i=sB_iA.$$

For any matrices $U,V$,

$$\operatorname{row}(UV)\subseteq\operatorname{row}(V), \qquad \operatorname{rank}(UV)\leq\min(\operatorname{rank}(U),\operatorname{rank}(V)).$$

Therefore

$$\operatorname{row}(\Delta W_i)\subseteq\operatorname{row}(A), \qquad \operatorname{rank}(\Delta W_i)\leq r.$$

For one token, define the routed readout

$$B_g(x)=\sum_i g_i(x)B_i.$$

Its token-specific effective update is

$$\Delta W_{eff}(x)=sB_g(x)A,$$

which still has rank at most $r$ and the same row-space ceiling. Increasing $J$ adds more possible readouts from the shared latent $Ax$; it does not add new rows to the input subspace.

Independent experts do not have that ceiling on a fixed mixture. With $d_{out}=3$, $d_{in}=4$, $r=2$, and three independent experts, choose updates that sum to

$$\sum_{i=1}^{3}B_iA_i = \begin{bmatrix} 1&0&0&0\\ 0&1&0&0\\ 0&0&1&0 \end{bmatrix}.$$

The sum has rank 3, so no shared rank-2 $A$ or shared rank-2 $B$ can represent it. This is a counterexample to hypothesis-class equivalence, not a claim that the larger class always generalizes better. Sharing encodes an inductive bias that may help when the common-subspace hypothesis is true and bottleneck adaptation when it is false.

There is a second subtlety. If $g(x)$ changes with $x$, then $\Delta W_{eff}(x)$ changes too. The adapter mapping is generally nonlinear in $x$ even though every $A$ and $B_i$ is linear. In the exact witness, all three token-specific matrices have rank 2, but the largest pairwise matrix difference is $0.3228$. One static merged update does not represent that route witness. Loop 6 owns the deployment consequences.

0x04D Forward Tensor Trace And Coupled Backward

For a row batch $X\in\mathbb{R}^{T\times d_{in}}$, compute router logits and weights

$$L=XR^{\top}\in\mathbb{R}^{T\times J}, \qquad G=\operatorname{softmax}_{J}(L).$$

An optimized shared-$A$ schedule computes the common latent once:

$$Z=XA^{\top}\in\mathbb{R}^{T\times r}, \qquad E_i=ZB_i^{\top}\in\mathbb{R}^{T\times d_{out}}.$$

The adapter output is

$$\Delta Y = s\sum_i\operatorname{Diag}(G_{:,i})E_i.$$

Now let $D=\partial\mathcal{L}/\partial Y\in\mathbb{R}^{T\times d_{out}}$ and define the gate-weighted upstream for expert $i$:

$$U_i=s\operatorname{Diag}(G_{:,i})D.$$

Each distinct readout receives

$$\frac{\partial\mathcal{L}}{\partial B_i} = U_i^{\top}Z \in\mathbb{R}^{d_{out}\times r}.$$

All readout paths accumulate into one latent gradient and one shared factor:

$$\frac{\partial\mathcal{L}}{\partial Z} = \sum_i U_iB_i, \qquad \frac{\partial\mathcal{L}}{\partial A} = \left(\frac{\partial\mathcal{L}}{\partial Z}\right)^{\top}X.$$

The router receives only relative expert credit. For token $t$, define

$$q_{t,i}=s\langle D_t,E_{i,t}\rangle, \qquad \bar q_t=\sum_jG_{t,j}q_{t,j}.$$

Then

$$\frac{\partial\mathcal{L}}{\partial L_{t,i}} = G_{t,i}(q_{t,i}-\bar q_t), \qquad \frac{\partial\mathcal{L}}{\partial R} = \left(\frac{\partial\mathcal{L}}{\partial L}\right)^{\top}X.$$

The input gradient has three separately auditable paths:

$$\frac{\partial\mathcal{L}}{\partial X} = DW_0 + \left(\frac{\partial\mathcal{L}}{\partial Z}\right)A + \left(\frac{\partial\mathcal{L}}{\partial L}\right)R.$$

The frozen base receives no optimizer update, but it remains in the input-gradient path. Removing that path because $W_0$ is frozen is a backward bug.

Shared storage also changes optimizer dynamics even when initial values do not. Suppose independent experts begin with value-equal copies $A_i=A$. Their forward output exactly matches shared $A$, and at that point

$$\left.\frac{\partial\mathcal{L}}{\partial A}\right|_{shared} = \sum_i \left.\frac{\partial\mathcal{L}}{\partial A_i}\right|_{independent}.$$

After one SGD step, however,

$$A^{+}=A-\eta\sum_i\nabla_{A_i}\mathcal{L}, \qquad A_i^{+}=A-\eta\nabla_{A_i}\mathcal{L}.$$

The independent copies diverge unless every expert gradient is identical. In the exact witness, the initial forward difference is zero and the shared gradient matches the sum to $5.55\times10^{-17}$, yet the largest pairwise $A_i$ difference after one step is $0.03383$. A shared parameter is not shorthand for synchronized copies; it is one optimizer state receiving accumulated credit.

0x04E Zero-B Initialization Is A Training Schedule

The standard LoRA initialization sets $A$ nonzero and every $B_i$ to zero so that the initial adapter update is exactly zero. HydraLoRA’s paper states this baseline convention, and the pinned main code uses Kaiming-uniform $A$, zero $B_i$, and a separately initialized router. That creates a specific first-step gradient schedule.

At $B_i=0$ for all $i$:

$$E_i=0, \qquad \Delta Y=0, \qquad q_{t,i}=0.$$

Therefore

$$\nabla_L\mathcal{L}=0, \qquad \nabla_R\mathcal{L}=0, \qquad \nabla_A\mathcal{L}=0,$$

while

$$\nabla_{B_i}\mathcal{L} = s\left(\operatorname{Diag}(G_{:,i})D\right)^{\top}XA^{\top}$$

can be nonzero. Under task loss alone, the first update goes into the $B_i$ readouts. Shared $A$ and the router begin receiving task gradient only after the $B_i$ paths become nonzero. The pinned main path returns a scalar-zero balance placeholder, so the inspected snapshot does not add a separate gate-balancing gradient at initialization.

The exact fixture makes this visible: the initial $A$ and router gradient norms are both zero, while the three $B_i$ gradient norms are $0.3563$, $0.6715$, and $0.4439$. After one teaching update to $B_i$, the $A$ gradient norm becomes $0.03941$ and the router gradient norm becomes $0.007667$.

This schedule is not automatically harmful, but it is part of the method. Router symmetry, early expert allocation, warmup, auxiliary objectives, nonzero-$B$ initialization, and staged training are alternative design choices, not implementation trivia. If both the router and experts remain symmetry-equivalent, specialization can be delayed or collapse into redundant branches.

0x04F Paper Algebra Versus Pinned Main-Code Schedule

The paper-level workflow is:

Source: HydraLoRA, Figure 4. The left side separates component-count initialization from joint routed fine-tuning. The right side describes input-conditioned weighted composition of $B_i$ around shared $A$. “Routing LoRA merge” is dynamic composition here, not one permanent weight update shared by every input.

The pinned repository at commit afb693b contains several LoRA/MoE paths that must not be conflated:

Pinned path	Observed ownership and route	What it establishes
HydraLoRA/peft/tuners/lora.py	one lora_A object, lora_B0...B{J-1}, bias-free token router	main shared-$A$ code path used by fine-tuning.py
HydraLoRA/peft/tuners/mmoelora.py	independent $A_i/B_i$ lists, task-embedding router	symmetric/task-routed baseline, not HydraLoRA sharing evidence
MLLM-HydraLoRA/moe.py	projection-specific shared-$A$ parameters, dense training and sparse inference branches	later multimodal extension, not reproduction of the main paper experiments

The main path’s forward loop is equivalent to the following source trace:

route_weight = softmax(
    lora_route(x), dim=-1, dtype=float32
).to(result.dtype)

for i in range(J):
    result += (
        route_weight[:, :, i, None]
        * lora_B[i](lora_A(dropout(x)))
        * scaling
    )

return result, scalar_zero_balance_placeholder

This proves shared parameter storage but not optimized shared computation. The A projection is recomputed inside the expert loop, and all $J$ branches are evaluated. There is no top-k branch-skipping path in this inspected class. An optimized dense implementation could hoist

Z = lora_A(dropout(x))
expert_outputs = stack([lora_B[i](Z) for i in range(J)], dim=expert)
result += scaling * sum(route_weight[..., i, None] * expert_outputs[..., i, :])

only where every branch sees the same dropout output: at inference, where dropout is the identity, or with dropout probability zero. In training, the pinned loop calls dropout(x) once per expert, so each $B_i$ branch receives its own random mask (the training entrypoint defaults to dropout 0.1, as noted below). Hoisting Z replaces those $J$ independent masks with one shared mask and changes the training computation; 0x08C lists the same mask among the reuse preconditions. Even where the hoist is exact, algebraic permission is not evidence that the pinned runtime performs the optimization. In the tiny FLOP ledger, the optimized shared-$A$ schedule uses 48 multiply-add FLOPs for $XA^{\top}$, while the pinned loop performs 144 because it recomputes that projection three times. Both then spend 108 on the three $B_i$ readouts and 72 on the router under the stated convention.

The source also exposes a dimensional ambiguity in the paper text. The printed router description gives $W_g\in\mathbb{R}^{r\times N}$ while saying it consumes intermediate token representation $x$. The pinned code unambiguously maps the target linear’s in_features to $J$ logits before applying $A$. This article follows the executable tensor contract and labels the printed shape as unresolved rather than silently repairing it.

The training entrypoint defaults to q_proj,v_proj, rank 8, $\alpha=32$, dropout 0.1, and two $B$ factors. Those are snapshot defaults, not universal HydraLoRA requirements. The paper’s Table 2 separately reports a HydraLoRA rank-8 configuration with one $A$, three average $B$ matrices, and the same reported 0.124% parameter fraction as rank-16 LoRA. Those results are source-reported; this loop does not execute the checkpoint or reproduce quality, latency, or energy measurements.

0x04G Independent, Shared-A, And Shared-B Design Choices

Sharing $B$ instead gives

$$\Delta Y = s\sum_i\operatorname{Diag}(G_{:,i})XA_i^{\top}B^{\top}.$$

Because $B$ is common and linear, an optimized schedule may first combine latent rows,

$$Z_g = \sum_i\operatorname{Diag}(G_{:,i})XA_i^{\top},$$

then apply $B$ once. This saves output-side readout work but forces

$$\operatorname{col}(BA_i)\subseteq\operatorname{col}(B).$$

The alternatives therefore encode different beliefs:

Choice	Representation	Gradient coupling	Optimized dense schedule	Main risk
Independent $A_i/B_i$	separate input and output subspaces	no shared factor; router still couples selection	$J$ input projections and $J$ readouts	parameter and optimizer-state growth
Shared $A$	common input row space, distinct output readouts	all experts accumulate into one $A$	one $A$ projection and $J$ readouts	incompatible input subspaces bottleneck $A$
Shared $B$	distinct input projections, common output column space	all experts accumulate into one $B$	$J$ input projections and one readout after latent mixing	incompatible output directions bottleneck $B$

Router/component choice	Benefit	Cost or caveat
developer-specified components	reproducible semantic partition	requires prior structure and may encode wrong boundaries
k-means-selected $J$	data-driven component-count heuristic	clustering representation and criterion become provenance
dense soft routing	differentiable weighted composition	every branch runs unless a specialized kernel changes the schedule
sparse top-k routing	lower active branch count in principle	selection, capacity, load balance, and dispatch become new contracts

HydraLoRA’s paper uses MoE as a composition mechanism and explicitly notes that other fusion methods are possible. The factor-sharing assumption and the routing mechanism should therefore be evaluated separately: one can test shared $A$ with a fixed mixture, or a learned router with independent factors, before attributing a gain to the combined design.

0x04H Executable Shared-Factor Oracle

The Loop 5 evidence package freezes the paper, code, object, tensor, and claim contracts in hydralora_shared_factor_fixture.json. Its pure-Python oracle derives all three variants under one row-batch loss:

shared A:
    X -> one Z = X A^T -> J B_i readouts -> tokenwise soft mixture
    gradients: X, A, every B_i, and router R

independent:
    X -> J Z_i = X A_i^T -> J B_i readouts -> same router mixture
    gradients: X, every A_i/B_i, and R

shared B:
    X -> J Z_i = X A_i^T -> one shared B readout family
    gradients: X, every A_i, shared B, and R

It checks 170 scalar coordinates by centered finite differences in float64. The largest observed absolute error is $2.71\times10^{-10}$ against an $8\times10^{-8}$ tolerance. It also verifies:

factor parameters:         independent 42 · shared-A 26 · shared-B 30
cloned/shared forward:     max difference 0
shared dA vs sum dA_i:     max difference 5.55e-17
one-step clone divergence: 0.03383
independent rank witness:  3 > shared rank 2
gauge-equivalent output:   max difference 1.11e-16
token effective matrices:  ranks [2,2,2], not one static matrix

The checked report is hydralora_shared_factor_report.json. Eighty-eight one-field negative fixtures reject source substitution, factor alias changes, wrong shapes, changed route or initialization contracts, and any upgrade of an unsupported claim.

0x04I Failure Lab And Measurements

Failure	Why a shape check can miss it	Required detection
$A$ duplicated once per expert but called shared	values may begin equal and outputs initially match	object identity, checkpoint-key count, optimizer-slot count
shared $A$ gradient not summed across experts	every local gradient has shape $[r,d_{in}]$	compare against $\sum_i\nabla_{A_i}$ oracle
raw-factor similarity called invariant	$BA$ may be identical after a gauge transform	align factors or compare subspaces/functions
shared $A$ treated as same hypothesis class as independent factors	every individual update still has rank $r$	rank-3 sum counterexample under $r=2$
zero-$B$ router expected to specialize on the first task step	softmax runs and returns valid probabilities	log task-loss router gradient norm at initialization
pinned dense loop described as sparse MoE	gate values can be near zero while all branches execute	branch invocation counts and kernel trace
shared projection assumed reused	one parameter object exists	count $A(x)$ calls or inspect the compiled graph
mmoelora.py assumed to implement shared $A$	filename sounds authoritative	trace allocations: that file owns independent $A_i/B_i$
shared $B$ presented as equivalent to shared $A$	parameter counts are both smaller than independent	row-space versus column-space containment tests
dynamic token route exported as one permanent merge	each effective matrix has rank at most $r$	compare effective matrices across tokens and requests

For a real training run, record the following before interpreting specialization:

identity and storage:
    exact A/B/router checkpoint keys, aliases, shapes, dtypes,
    parameter count, optimizer slot count, parent/base digest

gradient schedule:
    per-step ||dA||, ||dB_i||, ||dR||,
    expert pairwise gradient cosine after gauge-aware alignment,
    time until router and shared A receive nonzero task credit

route behavior:
    entropy, per-expert probability mass, branch invocation count,
    input scope, layer/projection scope, component-count provenance

hypothesis diagnostics:
    row-space projector drift for A, column-space drift for B_i,
    effective-matrix rank, independent-versus-shared equal-budget controls

runtime:
    actual A-projection call count, B_i readout count,
    materialized temporary bytes, launch count, and latency by phase

The minimum causal comparison is independent factors versus shared $A$ versus shared $B$ under both equal rank and equal parameter budget, each with fixed and learned routing controls. Without that matrix, a quality or efficiency change cannot be attributed specifically to asymmetry, routing, budget, or implementation reuse.

Loop 5 closes factor ownership and training algebra only. Static merge, dynamic composition packaging, quantized state, and provenance move to Loop 6; continual adapter state remains Loop 7; sparse runtime execution remains Loop 9.

0x05 Static Merge, Dynamic Composition, Quantization, And Provenance

The word merge is overloaded. It can mean adding full update matrices, constructing a higher-rank LoRA whose product equals that sum, approximating the sum at a smaller rank, combining the two factor coordinates separately, resolving conflicts between full task vectors, or evaluating an input-conditioned mixture at runtime. Those operations do not share one algebra or one artifact contract.

A22’s source-local fusion survey already separates manually chosen weights, learned blending, and LoRA-expert hybrids. Loop 6 turns that taxonomy into an executable boundary: first name the coordinate space, then state whether the result is exact, approximate, stochastic, or input-dependent, and only then decide whether it can be packaged as one static artifact.

0x05A Composition Starts With The Coordinate Space

For adapter $i$, define the deployed update matrix rather than only its stored factors:

$$D_i = s_iB_iA_i \in \mathbb{R}^{d_{out}\times d_{in}}, \qquad s_i=\frac{\alpha_i}{r_i}.$$

For request-static coefficients $\lambda_i$, the intended update is

$$D = \sum_{i=1}^{J}\lambda_iD_i = \sum_{i=1}^{J}c_iB_iA_i, \qquad c_i=\lambda_is_i.$$

That equation is in delta-matrix coordinates. It is gauge-invariant: replacing $A_i$ by $S_iA_i$ and $B_i$ by $B_iS_i^{-1}$ leaves $D_i$ unchanged. A method that instead averages $A_i$ tensors and $B_i$ tensors separately is operating in factor coordinates. The factorization is non-identifiable, so equality in one coordinate system does not imply equality in the other.

Operation	Input coordinates	Output	Exactness contract
direct dense sum	$D_i$	one dense $D$ or $W+D$	exact before finite-precision rounding
block-concatenated LoRA	$A_i,B_i$ with weights absorbed into one side	one rank-$\sum_i r_i$ factor pair	exactly reconstructs $\sum_i c_iB_iA_i$
delta then truncated SVD	full $D$	one rank-$r'$ factor pair	exact only when $r'\geq\operatorname{rank}(D)$
factor-linear or factor-wise TIES/DARE	$A_i$ and $B_i$ separately	one rank-compatible factor pair	generally differs from delta arithmetic
input-conditioned composition	$D_i$ plus $g_i(x)$	runtime function $D_{eff}(x)x$	not one global static matrix in general

The first compatibility check is therefore not “do these files have the same rank?” It is:

same parent base and target tensor?
same factor convention and scale interpretation?
same intended coordinate space?
same static coefficients for every request and token?
same dtype and rounding boundary?
same stochastic mask, seed, and reduction policy if used?

Matching shapes are not a compatibility proof.

0x05B Exact Static Sum By Block Concatenation

Assume each adapter targets the same $W\in\mathbb{R}^{d_{out}\times d_{in}}$, but allow heterogeneous ranks $r_i$. Stack the weighted input factors by rows and the output factors by columns:

$$A_{cat} = \begin{bmatrix} c_1A_1\\ c_2A_2\\ \vdots\\ c_JA_J \end{bmatrix} \in \mathbb{R}^{(\sum_i r_i)\times d_{in}},$$ $$B_{cat} = \begin{bmatrix} B_1&B_2&\cdots&B_J \end{bmatrix} \in \mathbb{R}^{d_{out}\times(\sum_i r_i)}.$$

Block multiplication contains only matching block products:

$$\begin{aligned} B_{cat}A_{cat} &= \begin{bmatrix}B_1&\cdots&B_J\end{bmatrix} \begin{bmatrix}c_1A_1\\\vdots\\c_JA_J\end{bmatrix}\\ &= \sum_{i=1}^{J}c_iB_iA_i =D. \end{aligned}$$

There are no $B_iA_j$ cross terms because the blocks meet along disjoint latent coordinates. The rank bound follows from the inner dimension:

$$\operatorname{rank}(D) \leq \min\left(d_{out},d_{in},\sum_i r_i\right).$$

The pinned PEFT v0.19.1 implementation uses this construction for combination_type="cat": it multiplies each $A_i$ by the requested weight and adapter scaling, concatenates those rows, concatenates $B_i$ columns, and allocates output rank $\sum_i r_i$. That code path supports different input ranks, but the resulting adapter can become large enough to erase the storage and runtime benefit of low rank.

The Loop 6 rank-one witness is deliberately small:

$$A_1=\begin{bmatrix}1&0\end{bmatrix}, \quad B_1=\begin{bmatrix}1\\0\end{bmatrix}, \quad A_2=\begin{bmatrix}0&1\end{bmatrix}, \quad B_2=\begin{bmatrix}0\\1\end{bmatrix},$$

with $c_1=1$ and $c_2=-\tfrac12$. Therefore

$$D = \begin{bmatrix}1&0\\0&0\end{bmatrix} -\frac12 \begin{bmatrix}0&0\\0&1\end{bmatrix} = \begin{bmatrix}1&0\\0&-0.5\end{bmatrix}.$$

Each input adapter has rank 1, while the composed update has rank 2. The block factors have shapes $[2,2]$ and $[2,2]$ and reconstruct $D$ with maximum error exactly zero in the float64 teaching oracle.

def exact_cat(adapters, coefficients):
    A_cat = concat_rows([
        coefficient * adapter.scale * adapter.A
        for adapter, coefficient in zip(adapters, coefficients)
    ])
    B_cat = concat_columns([adapter.B for adapter in adapters])
    return A_cat, B_cat  # B_cat @ A_cat is the exact weighted delta sum

This is a representation of the composed adapter update. Folding it into the base as $W' = W+D$ is a separate lifecycle step owned at the single-adapter level by M6.3 and at deployment-package level by M8.

0x05C Rank-Constrained Delta SVD And Its Error

If rank $\sum_i r_i$ is too expensive, one can first construct $D$ in delta coordinates and then refactor it. Let

$$D=U\Sigma V^{\top}, \qquad \Sigma=\operatorname{diag}(\sigma_1,\ldots,\sigma_q), \quad \sigma_1\geq\cdots\geq\sigma_q\geq0.$$

The rank-$r'$ truncation is

$$D_{r'} = U_{:,1:r'}\Sigma_{1:r',1:r'}V_{:,1:r'}^{\top}.$$

A LoRA-compatible factorization can place the singular values on either side. PEFT’s pinned delta-SVD path returns

$$B'=U_{:,1:r'}\Sigma_{1:r',1:r'}, \qquad A'=V_{:,1:r'}^{\top},$$

so $B'A'=D_{r'}$. Under the Frobenius norm, the discarded error is

$$\lVert D-D_{r'}\rVert_F^2 = \sum_{j>r'}\sigma_j^2.$$

For the diagonal witness, $\sigma=[1,0.5]$. Keeping rank 1 gives

$$D_1=\begin{bmatrix}1&0\\0&0\end{bmatrix}, \qquad \lVert D-D_1\rVert_F=0.5.$$

That is the best rank-1 Frobenius approximation, but it is still not the requested exact sum. The deployment decision has traded one capability direction for a smaller adapter. The artifact must record requested rank, actual retained rank, SVD driver, clamping, working dtype, and output digest. A rank choice without an error ledger hides the decision.

The pinned PEFT _svd_generalized_task_arithmetic_weighted_adapter path makes the coordinate distinction executable. It first calls get_delta_weight(adapter), applies task arithmetic, TIES, DARE, or magnitude pruning to those full delta matrices, and only then computes the SVD. Its non-SVD sibling follows a different path.

0x05D Factor-Linear Composition Creates Cross Terms

For same-rank adapters, PEFT’s non-SVD linear path assigns the sign to the $A$ weight and splits coefficient magnitude across both factors. With $c_i=\lambda_is_i$:

$$A_{lin} = \sum_i\operatorname{sign}(c_i)\sqrt{|c_i|}A_i, \qquad B_{lin} = \sum_i\sqrt{|c_i|}B_i.$$

Multiplying the combined factors gives

$$\begin{aligned} B_{lin}A_{lin} &= \sum_i\sum_j \sqrt{|c_i c_j|}\operatorname{sign}(c_j)B_iA_j\\ &= \underbrace{\sum_i c_iB_iA_i}_{\text{desired diagonal terms}} + \underbrace{\sum_{i\neq j} \sqrt{|c_i c_j|}\operatorname{sign}(c_j)B_iA_j}_{\text{cross terms}}. \end{aligned}$$

The diagonal terms are correct. The off-diagonal terms are generally nonzero. Therefore factor-coordinate linear combination is not delta arithmetic.

For the exact witness:

$$A_{lin}=\begin{bmatrix}1&-\sqrt{0.5}\end{bmatrix}, \qquad B_{lin}=\begin{bmatrix}1\\\sqrt{0.5}\end{bmatrix},$$

and

$$B_{lin}A_{lin} = \begin{bmatrix} 1&-0.7071\\ 0.7071&-0.5 \end{bmatrix}.$$

The desired $D$ has zero off-diagonal entries, so the cross-term matrix has Frobenius norm $1.0$. The distinction is not rounding noise.

It is also gauge-sensitive. Replace only adapter 2 with the functionally identical factors

$$A_2'=2A_2, \qquad B_2'=\frac12B_2, \qquad B_2'A_2'=B_2A_2.$$

The exact delta sum remains unchanged, while the factor-linear output changes by $0.7071$ in maximum absolute value. Any factor-wise TIES, DARE, magnitude pruning, averaging, or interpolation inherits this coordinate question unless it first canonicalizes the gauge or reconstructs full deltas.

This does not make factor-space composition useless. It makes it a different hypothesis class with a smaller fixed rank and different interactions. It should be evaluated against exact cat and delta-SVD baselines, not labeled an exact merge by default.

0x05E TIES: Paper Task Vectors Versus PEFT LoRA Paths

TIES begins with full fine-tuned model task vectors, not LoRA factors:

$$\tau_t = \theta_t-\theta_{init}.$$

Its paper algorithm has three ordered steps.

1. Trim: retain the top-$k$ percent of each $\tau_t$ by magnitude and set the rest to zero, producing $\hat\tau_t$.
2. Elect: choose a sign at every parameter coordinate from the aggregate signed movement.
3. Disjoint merge: average only nonzero task-vector values aligned with the elected sign.

Source: TIES-Merging, Figure 1, cropped from the official NeurIPS proceedings PDF. The figure establishes the paper’s full task-vector procedure. It does not establish that applying TIES independently to LoRA $A$ and $B$ factors is equivalent to applying it to $BA$.

For coordinate $p$, a total-magnitude sign election can be written as

$$\gamma_p = \operatorname{sign}\left(\sum_t\hat\tau_{t,p}\right).$$

With aligned set

$$\mathcal{A}_p = \left\{t:\hat\tau_{t,p}\neq0, \operatorname{sign}(\hat\tau_{t,p})=\gamma_p\right\},$$

the disjoint mean is

$$\tau_{merge,p} = \frac{1}{|\mathcal{A}_p|} \sum_{t\in\mathcal{A}_p}\hat\tau_{t,p},$$

with an explicit empty-coordinate policy. For the Loop 6 vectors and density $0.5$, trimming leaves two coordinates per vector, elects signs $[+,-,-,0]$, and returns $[4,-3,-2.5,0]$.

The TIES paper contract and PEFT LoRA paths are different objects:

Surface	Coordinates receiving trim/sign/merge	Final factorization
TIES paper	full task vectors $\tau_t$	merged full parameter vector
PEFT ties	$A_i$ and $B_i$ independently	multiply merged $B_{ties}A_{ties}$; cross terms and gauge dependence can remain
PEFT ties_svd	full LoRA delta matrices $D_i$	SVD refactor at requested rank; may truncate

The pinned code also exposes implementation choices absent from the one-word method name: density, total-versus-frequency sign election, disjoint mean versus sum, target compatibility, rank policy, SVD driver, and optional clamp. These belong in the artifact record.

TIES reports empirical gains under its evaluation protocol. Loop 6 externally verifies the method and executes a small algebraic procedure; it does not reproduce the paper’s benchmark or claim that trimming and sign election universally improve merged quality.

0x05F DARE: An Unbiased Stochastic Preprocessor, Not An Identity

DARE operates on delta parameters. Write retain probability $q=1-p$ and a coordinatewise mask

$$z_j\sim\operatorname{Bernoulli}(q).$$

The dropped-and-rescaled delta is

$$\widehat\delta_j = \frac{z_j}{q}\delta_j.$$

The inverse-density scale gives a coordinatewise expectation identity:

$$\mathbb{E}[\widehat\delta_j] = \frac{\mathbb{E}[z_j]}{q}\delta_j = \delta_j.$$

That derivation explains the scale. It does not say one sampled tensor equals the original delta. For

$$\delta=[2,-1,0.5,3], \quad q=0.5, \quad z=[1,0,1,0],$$

the sampled result is

$$\widehat\delta=[4,0,1,0],$$

whose maximum coordinate difference from $\delta$ is $3$. Averaging over the mask distribution recovers the original coordinates; one release artifact does not.

DARE is a stochastic preprocessor before a subsequent linear or TIES merge. Reproducibility requires retain density plus either the exact mask digest or a fully specified random generator, seed, traversal order, device, and library contract. PEFT’s pinned dare_linear and dare_ties paths apply random pruning to factors separately; their _svd variants apply it to full deltas before refactoring. Those paths should not share one unlabeled “DARE merged” artifact name.

0x05G Dynamic Gates Fail The Static-Matrix Contract

For a learned input-conditioned router:

$$F(x) = \sum_i g_i(x)D_ix, \qquad g(x)=\operatorname{softmax}(Rx).$$

At any one $x$, one can define a token-specific effective matrix

$$D_{eff}(x)=\sum_i g_i(x)D_i.$$

That does not imply one offline matrix works for every input. Any fixed matrix map is additive:

$$Mx_1+Mx_2=M(x_1+x_2).$$

Therefore one counterexample with

$$F(x_1+x_2)\neq F(x_1)+F(x_2)$$

is sufficient to reject a global static merge for that routed function.

The Loop 6 witness uses

$$D_1=\begin{bmatrix}1&0\\0&0\end{bmatrix}, \quad D_2=\begin{bmatrix}0&0\\0&1\end{bmatrix}, \quad R=I,$$

with $x_1=[1,0]^{\top}$ and $x_2=[0,1]^{\top}$. The router gives approximately $[0.7311,0.2689]$ on $x_1$, reverses those weights on $x_2$, and gives $[0.5,0.5]$ on $x_1+x_2$. The additivity residual is

$$F(x_1+x_2)-F(x_1)-F(x_2) = \begin{bmatrix}-0.2311\\-0.2311\end{bmatrix},$$

with norm $0.3268$. The largest difference among the three effective matrices is $0.4621$.

The phrase “dynamic merge” must therefore be read as runtime composition. Its state includes the router, candidate pool, route scope, and selected coefficients. Request-level routing may resolve one static adapter for the whole request. Sequence-, token-, layer-, or matrix-level routing may change the effective operator within the same request. Loop 8 owns those granularities; M6.5 owns loading and residency; Loop 9 owns the physical routed execution.

0x05H Quantization Makes Operation Order Observable

Exact (real-number) addition is commutative and associative:

$$W+D_1+D_2=W+D_2+D_1.$$

The lifecycle ceases to share that equality when a nonlinear representation change is inserted. To expose the issue without reopening M8, define one teaching quantizer:

$$Q_s(z) = s\,\operatorname{clip} \left(\operatorname{round}_{even}(z/s),q_{min},q_{max}\right),$$

with $s=0.25$ and integer range $[-4,4]$. This is only an operation-order witness, not a recommended deployment format.

Using the frozen fixture matrices, the three paths produce:

$$Q(W+D_1+D_2) = \begin{bmatrix}-0.75&0\\0.5&0.5\end{bmatrix},$$ $$Q(W)+D_1+D_2 = \begin{bmatrix}-0.85&0.05\\0.5&0.5\end{bmatrix},$$

and

$$Q(Q(W+D_1)+D_2) = \begin{bmatrix}-1&0\\0.5&0.5\end{bmatrix}.$$

The first and second paths differ by $0.10$ in maximum absolute value; sequential requantization differs from one-shot quantization by $0.25$. Likewise,

$$Q(W+D_1)-D_1 \neq W,$$

with maximum recovery error $0.10$. Subtracting the float delta cannot undo information lost to rounding or clipping.

def publish_composed_artifact(base, deltas, quantizer):
    assert all(delta.parent_base_digest == base.digest for delta in deltas)
    composed = exact_delta_compose(base.float_weight, deltas)
    released = quantizer.quantize_once(composed)
    return released, digest(released), quantizer.calibration_digest

def unsafe_sequential_publish(base, delta1, delta2, quantizer):
    interim = quantizer.quantize(base + delta1)
    return quantizer.quantize(interim + delta2)  # a different artifact

Other order changes matter for the same reason:

train D2 against W + D1       != train D2 against W
TIES then SVD                 != SVD each adapter then TIES factors
DARE mask then gauge change   != gauge change then factor-wise DARE mask
merge, round, then unmerge    != restore immutable W and original deltas

M8 owns quantizer families, calibration, kernels, deployment evaluation, release, and rollback. M6.9 owns recording where composition sits in that lifecycle.

0x05I Composition Artifact And Provenance DAG

A released composition is a directed acyclic graph of parent artifacts and transformations, not merely a filename. The minimum record is:

parents:
    base digest, revision, config, tokenizer
    every adapter digest and its own parent-base digest

target ABI:
    module path, semantic role, exact shape
    factor convention, rank, scale, dtype, orientation

composition:
    method and coordinate space
    adapter order, coefficients, normalization
    cat rank or SVD rank/driver/clamp
    TIES density/sign/reduction policy
    DARE density and seed or exact mask digest

dynamic state, if any:
    router digest, input scope, layer/matrix scope, top-k policy
    component-count and candidate-pool revisions

representation transition:
    merge dtype, quantizer family, scale/zero-point/clipping/rounding
    calibration digest and exact operation order

release:
    library release and commit
    output tensor and package digests
    evaluation-record digest and compatibility decision

The immutable Loop 6 record binds 23 top-level fields. Its static cat artifact rejects a mismatched parent base, wrong target shape, omitted coordinate space, changed coefficient order, missing stochastic identity, changed quantization order, or output digest drift.

def admit_composition(record, base, adapters):
    require(record.base_digest == base.digest)
    require(all(a.parent_base_digest == base.digest for a in adapters))
    require(record.target_shape == base.target_shape)
    require(record.coordinate_space in APPROVED_COORDINATE_SPACES)

    if record.dare.used:
        require(record.dare.seed is not None or record.dare.mask_digest is not None)
    if record.router.mode != "none_static_coefficients":
        reject("dynamic route cannot be labeled one static merged artifact")

    require(record.quantization.operation_order == APPROVED_ORDER)
    require(hash_bytes(load_output(record)) == record.output_digest)
    require(evaluation_passed(record.evaluation_record_digest))

Restoring the base plus original adapters is the reversible rollback path. Subtracting factors from rounded merged weights is not.

0x05J Executable Oracle, Alternatives, And Failure Lab

The Loop 6 fixture is composition_merge_fixture.json. Its pure-Python oracle performs the matrix algebra, two-dimensional SVD, TIES trim/elect/disjoint merge, DARE expectation and sample, dynamic additivity witness, teaching quantizer, and provenance validation without NumPy or a framework runtime.

The checked composition_merge_report.json establishes:

exact cat reconstruction error:         0
composed delta rank:                    2 from two rank-1 adapters
factor-linear cross-term norm:          1.0
gauge-equivalent factor-merge change:   0.7071
rank-1 SVD Frobenius error:             0.5
TIES retained coordinates per vector:  [2,2,2]
DARE expectation error:                 0
DARE sample max difference:             3.0
dynamic additivity residual norm:       0.3268
sequential quantization order change:   0.25
rounded unmerge base error:             0.10

It rejects 128 one-field mutations spanning source revision, method coordinates, dimensions, factors, dynamic route, TIES/DARE policy, quantizer, parent lineage, composition method, stochastic identity, output digest, and unsupported claim promotion.

Need	Conservative choice	More compressed or adaptive choice	New evidence required
exact fixed coefficients	block-cat adapter or dense delta sum	none	rank/storage/runtime ledger
smaller fixed-rank artifact	exact cat at rank sum	delta-SVD at $r'<\operatorname{rank}(D)$	discarded singular energy plus quality evaluation
same-rank factor artifact	delta-SVD at requested rank	factor-linear or factor-wise TIES/DARE	cross-term and gauge sensitivity controls
conflict resolution	explicit weighted delta baseline	TIES	density, sign election, reduction, benchmark protocol
stochastic redundancy removal	deterministic baseline	DARE before linear/TIES merge	seed/mask digest and variance over masks
input-specific specialization	one approved request-static composition	learned runtime gate	router, route scope, candidate residency, cache provenance
low-precision release	compose in approved working dtype, quantize once	sequential merge/requantize	order, calibration, byte digest, rollback proof

Failure	Shape-correct symptom	Root cause	Required detection or repair
factor merge labeled exact delta sum	output factor shapes match rank	$B_iA_j$ cross terms were introduced	compare against materialized $\sum_i c_iD_i$
gauge-equivalent adapters merge differently	every adapter still computes the same $D_i$ alone	operation acted on non-identifiable factors	canonicalize or compose in delta coordinates
truncated SVD called lossless	output rank is accepted by runtime	composed delta rank exceeds retained rank	singular spectrum and reconstruction error ledger
TIES paper and PEFT factor path conflated	both use the string ties	trim/elect/merge happened in different coordinates	pin implementation path and test delta output
DARE release cannot be reproduced	density matches	random mask or generator state is absent	seed plus RNG contract or exact mask digest
dynamic route exported as one merge	one token test passes	$g(x)$ changes with input	additivity test and effective-matrix trace across inputs
quantization order silently changes	shapes and nominal bit width match	rounding/clipping occurred between additions	operation DAG and output digest comparison
unmerge corrupts base	subtraction executes without error	merged state had already rounded	restore immutable parent artifact
same-shape incompatible adapters accepted	tensor load succeeds	parent bases, targets, scales, or conventions differ	provenance admission before arithmetic

Loop 6 closes multi-adapter composition algebra and artifact identity. It does not close generic quantization, runtime residency, paper benchmark reproduction, or continual-learning state. Loop 7 consumes this immutable parent/provenance contract before adding sequential task state.

0x06 Continual Adapters: Ordered State Without A No-Forgetting Myth

Continual learning is not ordinary multi-task training with the dataset rows sorted by task. It is an ordered state transition. Task $D_t$ arrives after $D_{<t}$, earlier examples may be unavailable, and the training artifact at step $t$ becomes the parent of step $t+1$. The learner must therefore answer three different questions:

parameter retention:
    which old tensors are frozen or consolidated?

functional retention:
    how does old-task behavior change after the new transition?

operational retention:
    which parent, selector, task order, evaluation, and rollback state survive?

Freezing one LoRA per completed task answers only the first question. A newly summed delta changes every input that reaches it. A newly trained selector changes how old frozen deltas are combined. A merged checkpoint can preserve the current function while destroying the factor basis required by the next regularizer. Continual state is consequently larger than “the adapters currently loaded on the GPU.”

0x06A Continual Learning Is An Ordered State Transition

Represent the state after task $t$ as:

$$\mathcal{S}_t = \left( H_{base}, \{\Delta W_i\}_{i=1}^{t}, \Phi_t, \pi_{1:t}, \mathcal{P}_t, R_{1:t,:} \right)$$

where $\Phi_t$ is selector or consolidation state, $\pi_{1:t}$ is the ordered task sequence, $\mathcal{P}_t$ is the parent/provenance record, and $R$ is the evaluation matrix. Training is a transition:

$$\mathcal{S}_t = \operatorname{TrainTask} \left( \mathcal{S}_{t-1}, D_t, \mathcal{C}_t \right)$$

under an explicit constraint contract $\mathcal{C}_t$. O-LoRA places an overlap penalty on the new factor subspace. AM-LoRA freezes old task factors and changes an attentional selector while learning the current adapter. A replay method would put old examples or old gradients into $\mathcal{C}_t$. These are different state machines, even if all eventually evaluate $W_0x+\sum_i\Delta W_ix$.

A release must be able to reconstruct or reject this transition:

base revision and digest
ordered task IDs and task-data manifest hashes
parent state digest for every task transition
adapter factor convention, targets, ranks, scales, and dtypes
which old factors, current factors, and selectors were trainable
regularizer definition and coefficients
merge or consolidation history
training-code revision
evaluation-matrix digest and task baselines

In this contract, task order is part of artifact identity. Reversing sentiment $\rightarrow$ topic $\rightarrow$ QA is not a metadata-only edit; it defines a different chain of optimization problems and therefore needs a new state digest and evaluation.

0x06B Three Lifecycles: Sum, Select, Or Consolidate

One new adapter per task does not determine how inference uses the history.

Lifecycle	Effective update	Preserved state	Benefit	Cost or failure mode
cumulative sum	$\sum_{i\le t}\Delta W_i$ for every input	all deltas or one exact merged parent	no task ID and simple forward	every new delta perturbs old inputs
task-ID selection	$\Delta W_{z}$	one adapter per task plus task resolver	isolates task factors and active compute	requires reliable task identity; limited cross-task reuse
learned feature selection	$\sum_i g_i(x)\Delta W_ix$	adapters plus selector state	input-conditioned reuse	selector drift, dense branch work, provenance growth
consolidation	replace history by $\widehat{W}_t$ or compressed factors	consolidated parent plus rollback source	bounded active artifact	approximation, irreversible rounding, future-basis loss

For target matrices $l\in\mathcal{L}$, retaining all task factors costs:

$$M_{adapter}(T) = b_a \sum_{t=1}^{T} \sum_{l\in\mathcal{L}} r_{t,l} \left(d_{in,l}+d_{out,l}\right)$$

before selector, optimizer, registry, and rollback state. A rank-efficient transition is not a bounded-state continual learner. M6.5 owns residency and paging; this section owns what must exist semantically even when only a subset is resident.

0x06C O-LoRA Paper Geometry

O-LoRA approximates earlier task-gradient subspaces with learned LoRA factors, freezes the prior factors, and learns a new low-rank update under an orthogonality regularizer. Its Figure 2 makes that lifecycle visible: the pretrained model and prior task factors are locked while the current pair remains trainable.

Source: O-LoRA official ACL Anthology paper, Figure 2. It establishes the paper architecture and freeze/train lifecycle, not numerical identity with the repository trainer or guaranteed retention.

The paper uses the convention:

$$\Delta W_t=A_tB_t, \qquad A_t\in\mathbb{R}^{d_{out}\times r_t}, \quad B_t\in\mathbb{R}^{r_t\times d_{in}}$$

so:

$$\Delta W_tx=A_t(B_tx)$$

first projects $x$ into $r_t$ coordinates through $B_t$, then maps those coordinates into the column space of $A_t$. O-LoRA therefore uses $\operatorname{col}(A_t)$ as a tractable proxy for the current task’s update directions. If the prior bases are concatenated as:

$$U_{t-1} = \left[A_1\;A_2\;\cdots\;A_{t-1}\right] \in \mathbb{R}^{d_{out}\times R_{t-1}}, \qquad R_{t-1}=\sum_{i<t}r_i,$$

exact subspace orthogonality would require:

$$U_{t-1}^{\top}A_t=0.$$

The method enforces a finite penalty instead. It does not project every optimizer step onto the exact null space, and it does not retain old examples to check old functions directly.

0x06D O-LoRA Paper Loss And Gradient

The paper’s pairwise overlap loss can be written:

$$\mathcal{L}_{orth}^{paper} = \sum_{i<t} \left\lVert A_i^{\top}A_t\right\rVert_F^2 = \left\lVert U_{t-1}^{\top}A_t\right\rVert_F^2.$$

The final form follows from the trace identity:

$$\begin{aligned} \left\lVert U^{\top}A\right\rVert_F^2 &= \operatorname{tr}\!\left((U^{\top}A)^{\top}(U^{\top}A)\right) \\ &= \operatorname{tr}\!\left(A^{\top}UU^{\top}A\right). \end{aligned}$$

Because $UU^{\top}$ is symmetric, differentiating gives:

$$\nabla_{A_t}\mathcal{L}_{orth}^{paper} = 2U_{t-1}U_{t-1}^{\top}A_t = 2\sum_{i<t}A_iA_i^{\top}A_t.$$

This gradient removes the component of the current basis that overlaps directions represented by the old factors. It is not automatically scale-normalized. Multiplying one old basis by a constant changes the penalty unless the bases are orthonormalized first.

The Loop 7 witness uses:

$$U= \begin{bmatrix} 1&0\\ 0&1\\ 0&0 \end{bmatrix}, \qquad A_t= \begin{bmatrix} 0.5\\-0.25\\1 \end{bmatrix}.$$

Then:

$$U^{\top}A_t= \begin{bmatrix}0.5\\-0.25\end{bmatrix}, \qquad \mathcal{L}_{orth}^{paper}=0.5^2+(-0.25)^2=0.3125,$$

and:

$$\nabla_{A_t}\mathcal{L}_{orth}^{paper} = \begin{bmatrix}1\\-0.5\\0\end{bmatrix}.$$

The third coordinate receives no penalty because the old proxy basis contains no component there. This explains the geometry of the regularizer; it does not establish that a finite optimizer step preserves every old network output.

0x06E The Official O-LoRA Code Is A Different Numerical Contract

The official repository is pinned at 07117e1. The paper and pinned code must remain separate rows. In the traced PEFT fork, old and current down-projection factors use the implementation orientation:

old lora_A:      [r_sum, d_in]
current loranew_A: [r, d_in]
old lora_B:      [d_out, r_sum]
current loranew_B: [d_out, r]

The trainer computes:

$$\mathcal{L}_{overlap}^{code} = \sum_l \left\lVert A_{old,l}A_{new,l}^{\top} \right\rVert_{1,entry},$$

not the squared Frobenius expression above. It then adds unsquared Euclidean norms for every current loranew_A and loranew_B tensor:

$$\mathcal{L}^{code} = \mathcal{L}_{task} + \lambda_1\mathcal{L}_{overlap}^{code} + \lambda_2 \sum_l \left( \lVert A_{new,l}\rVert_2+ \lVert B_{new,l}\rVert_2 \right).$$

Away from zero overlap entries, an $L_1$ subgradient is:

$$\partial_{A_{new}} \left\lVert A_{old}A_{new}^{\top}\right\rVert_{1,entry} = \operatorname{sign} \left(A_{old}A_{new}^{\top}\right)^{\top} A_{old}.$$

For the same numerical overlap $[0.5,-0.25]^{\top}$, the code loss is $0.75$, and the overlap subgradient in code coordinates is $[1,-1,0]$. This is not a harmless notation transpose: squared $L_2$ overlap and entrywise $L_1$ overlap produce different magnitudes, smoothness, and rank/task-count sensitivity. The code also matches old and current factors by a parameter-name prefix, making naming part of the implementation contract.

The initialization creates another training-state detail. Current loranew_A is Kaiming-initialized, while current loranew_B is zero. For a row batch:

$$Z=XA_{new}^{\top}, \qquad Y_{new}=ZB_{new}^{\top},$$

and upstream gradient $G=\partial\mathcal{L}_{task}/\partial Y_{new}$:

$$\nabla_{B_{new}}\mathcal{L}_{task}=G^{\top}Z,$$ $$\nabla_{A_{new}}\mathcal{L}_{task} = \left(GB_{new}\right)^{\top}X.$$

At $B_{new}=0$, the task-loss gradient into $A_{new}$ is exactly zero, while the task-loss gradient into $B_{new}$ can be nonzero. The regularizers can nevertheless move $A_{new}$ on the first step. In the executable witness, the maximum task-gradient magnitude is $0$ for $A_{new}$ and $0.875$ for $B_{new}$, while the regularizer produces a nonzero $A_{new}$ gradient. Training therefore begins with two distinct forces: task loss opens the zero $B$ branch, and regularization can rotate or shrink $A$ immediately.

def pinned_olora_loss(task_loss, named_parameters, lambda1, lambda2):
    overlap_l1 = 0.0
    factor_l2 = 0.0

    for module_prefix in target_module_prefixes(named_parameters):
        old_A = named_parameters[module_prefix + "lora_A"]
        new_A = named_parameters[module_prefix + "loranew_A"]
        overlap_l1 += abs(old_A @ new_A.T).sum()

    for name, factor in named_parameters.items():
        if "loranew_" in name:
            factor_l2 += norm(factor, p=2)  # unsquared in the pin

    return task_loss + lambda1 * overlap_l1 + lambda2 * factor_l2

This pseudocode describes the pinned revision only. It is not a replacement definition for the O-LoRA paper.

0x06F Cumulative Rank, Checkpoint Concatenation, And Scaling

The pinned save path accumulates completed and current ranks by concatenating $A$ along rank rows and $B$ along rank columns:

$$A_{cat} = \begin{bmatrix} A_{old}\\A_{new} \end{bmatrix}, \qquad B_{cat} = \begin{bmatrix} B_{old}&B_{new} \end{bmatrix}.$$

Multiplication proves the exact sum:

$$B_{cat}A_{cat} = \begin{bmatrix}B_{old}&B_{new}\end{bmatrix} \begin{bmatrix}A_{old}\\A_{new}\end{bmatrix} = B_{old}A_{old}+B_{new}A_{new}.$$

No cross term appears because the factors are block-concatenated, as in Loop 6’s exact cat composition. The cumulative rank becomes:

$$r_{sum,t}=\sum_{i=1}^{t}r_i.$$

Exactness requires the scale contract to survive. If old and new deltas use $s_{old}$ and $s_{new}$ but the cumulative forward applies one scale $s$, naive concatenation computes:

$$sB_{cat}A_{cat} = sB_{old}A_{old}+sB_{new}A_{new},$$

which equals the intended $s_{old}B_{old}A_{old}+s_{new}B_{new}A_{new}$ only when scales are compatible or encoded into one factor before concatenation. The oracle’s homogeneous-scale concatenation has zero error. Changing only the new-task scale from $1$ to $2$ yields maximum error $1.0$ under naive concatenation; multiplying the new $B$ block by its scale repairs the witness exactly.

Merging the cumulative delta into $W_0$ may reduce inference bookkeeping, but it does not make the continual state disappear. Future O-LoRA training still needs a representation of prior basis directions; rollback still needs the immutable parent or exact deltas; and a merged parent gets a new base digest. Inference state, future-training state, and rollback state are three separate ledgers.

0x06G Orthogonality Capacity And Function-Space Limits

If every task contributes a full-rank $r_t$ subspace and all task subspaces must be mutually orthogonal in an ambient dimension $d$, then necessarily:

$$\sum_{t=1}^{T}r_t\le d.$$

In the paper’s $A_t\in\mathbb{R}^{d_{out}\times r_t}$ convention, $d=d_{out}$. In the implementation’s row-space $A_t\in\mathbb{R}^{r_t\times d_{in}}$ convention, the corresponding ceiling is $d_{in}$. The Loop 7 witness sets $d=3$ and accepts ranks $[1,1,1]$; a fourth rank-one task cannot be exactly orthogonal to all three independent directions.

Before that hard ceiling, finite regularization can still leave overlap. After it, one of rank, exact orthogonality, or prior-subspace independence must give way. Practical alternatives include:

normalize or orthonormalize the stored bases before measuring overlap
penalize projector overlap ||Q_old^T Q_new||_F^2
inspect principal angles rather than raw factor magnitudes
allocate rank adaptively and stop when the null-space budget is exhausted
retain replay examples or distillation targets for function-space constraints
consolidate old tasks and accept an explicit approximation contract

M6.7 owns the general geometry of projectors, principal angles, factor gauge, and functional diagnostics. Loop 7 consumes that theory to set the claim ceiling: orthogonal proxy factors do not prove unchanged hidden states, logits, or old-task accuracy in a nonlinear network.

0x06H AM-LoRA Feature Selector Tensor Trace

AM-LoRA addresses a different failure. It keeps one task-specific LoRA per task, freezes completed adapter factors, and learns how to combine branch features. The paper’s Figure 2 separates the frozen old LoRAs from the current LoRA and the trainable feature editor/raters.

Source: AM-LoRA arXiv v1 paper, Figure 2. It establishes the paper architecture and trainability legend. No official AM-LoRA code revision is claimed here.

For a row batch $X\in\mathbb{R}^{N\times d_{in}}$, define branch $i$‘s feature:

$$F_i = X\Delta W_i^{\top} \in \mathbb{R}^{N\times d_{out}}.$$

Let one rater vector $u_i\in\mathbb{R}^{d_{out}}$ score that branch:

$$s_i=F_iu_i\in\mathbb{R}^{N}.$$

The logits must be stacked before normalization:

$$S= \begin{bmatrix} s_0&s_1&\cdots&s_t \end{bmatrix} \in \mathbb{R}^{N\times(t+1)},$$ $$G_{n,i} = \frac{\exp S_{n,i}} {\sum_{j=0}^{t}\exp S_{n,j}}.$$

The softmax is over the adapter-branch axis. Applying a softmax independently to each scalar $s_i$ would return one for every branch and would not define a selector. The mixed output is:

$$Y = XW_0^{\top} + \sum_{i=0}^{t} G_{:,i}\odot F_i,$$

where $G_{:,i}$ is broadcast over $d_{out}$. This dimensionally explicit equation also restores the base input multiplication that can be obscured by compact paper notation.

AM-LoRA includes a zero branch $\Delta W_0=0$. Therefore $F_0=0$. If its rater has no bias, $s_0=0$ supplies a fixed reference logit. The zero branch can absorb probability mass and reduce the adapter correction, but it does not save a matmul that was already executed for the nonzero branches.

The oracle uses two rows and four branches [zero, task-1, task-2, task-3]:

Row	Stacked logits $S_n$	Branch gates $G_n$
1	$[0,1,0,1]$	$[0.13447,0.36553,0.13447,0.36553]$
2	$[0,0,1,0]$	$[0.17488,0.17488,0.47537,0.17488]$

Every row sums to one over branches. The zero branch receives positive mass, and no gate is exactly zero.

0x06I AM-LoRA Trainability And Sparsity Contract

At task $t$, the base and prior task LoRA factors are frozen, and the current task LoRA is trainable. Selector trainability needs a more careful statement. The paper’s ablation reports that freezing previous rater dense layers performs worse than the stronger configuration that continues to train the selector raters. Consequently:

old LoRA factors fixed
does not imply
old branch contribution fixed

Changing $u_i$ changes $G_{:,i}(X)$ and therefore changes the composed function even when every $\Delta W_i$ is byte-identical. In the oracle, changing only the task-1 rater from $[1,0]$ to $[2,0]$ changes output by maximum absolute value $0.17426$ while all adapter deltas remain fixed.

The paper adds an $L_1$ penalty to rater weights:

$$\mathcal{L}_{AM} = \mathcal{L}_{task} + \lambda \sum_{i=0}^{t} \left\lVert u_i\right\rVert_1.$$

This encourages parameter shrinkage. It is not an $L_1$ penalty on gate probabilities, not a top-$k$ operator, and not a certificate that a branch can be skipped. Softmax assigns strictly positive probability to every finite logit. Sparse execution needs a separate threshold/top-$k$ policy, a route plan, and a kernel that does not materialize rejected branch features.

def am_lora_forward(X, W0, task_deltas, rater_vectors):
    # X: [N, d_in]; every F_i: [N, d_out]
    features = [X @ delta.T for delta in task_deltas]
    features = [zeros_like(features[0])] + features

    # Each branch produces one scalar per row. Stack first.
    logits = stack([
        features[i] @ rater_vectors[i]
        for i in range(len(features))
    ], axis=1)                              # [N, T+1]

    gates = softmax(logits, axis=1)         # adapter-branch axis
    mixed = sum(
        gates[:, i, None] * features[i]
        for i in range(len(features))
    )
    return X @ W0.T + mixed, gates

The pseudocode is a dimensionally valid reconstruction of the paper equations, not a traced official implementation.

0x06J State, Training, And Runtime Growth

For $T$ task adapters with common rank $r$ at one $d_{in}\rightarrow d_{out}$ target, factor state is:

$$P_{adapter}(T) = Tr(d_{in}+d_{out}).$$

One bias-free rater vector per task branch plus the zero branch adds:

$$P_{rater}(T) = (T+1)d_{out}.$$

Across target modules $l$:

$$P_{continual}(T) = \sum_{l\in\mathcal{L}} \left[ T r_l(d_{in,l}+d_{out,l}) + (T+1)d_{out,l} \right].$$

Storage, trainable state, and active runtime state differ:

Ledger at task $T$	O-LoRA-style cumulative factors	AM-LoRA stronger selector configuration
stored factors	all previous and current ranks, or exact merged delta plus retained basis provenance	all task LoRAs plus all raters
trainable factors	current loranew_A/B	current LoRA plus trainable raters
optimizer state	current factors and regularizer path	current LoRA and every trainable rater
inference branches	cumulative merged branch or accumulated factors	every materialized task branch unless execution prunes
rollback	immutable parent or exact task deltas	parent adapters plus selector revision

For the oracle’s $T=3$, $r=1$, $d_{in}=d_{out}=2$ layer, task adapters contain $12$ parameters and four raters contain $8$, for $20$ continual parameters. At task 3, two old adapters contribute $8$ frozen parameters. Training the current adapter and all raters exposes $12$ trainable parameters. A dense forward materializes three nonzero adapter branches and four selector logits per row.

These toy counts prove the ledger algebra, not the paper’s total model memory. Real releases must add every target layer, dtype, optimizer tensor, distributed replica/shard, selector bias if present, registry object, and parent artifact.

0x06K Continual Evaluation Matrix

The GEM evaluation protocol defines $R_{i,j}$ as performance on test task $j$ after learning through task $i$. For $T$ tasks:

$$ACC_T = \frac{1}{T} \sum_{j=1}^{T}R_{T,j}.$$

Final average accuracy does not identify whether earlier tasks improved or degraded. Backward transfer compares each old task’s final score with its score immediately after learning it:

$$BWT_T = \frac{1}{T-1} \sum_{j=1}^{T-1} \left(R_{T,j}-R_{j,j}\right).$$

Forward transfer needs a pretraining or random-initialization baseline $\bar b_j$ for the future task:

$$FWT_T = \frac{1}{T-1} \sum_{j=2}^{T} \left(R_{j-1,j}-\bar b_j\right).$$

A taskwise forgetting measure records the fall from the best score observed after learning that task:

$$F_j = \max_{i\in\{j,\ldots,T\}}R_{i,j} - R_{T,j}.$$

For the immutable teaching matrix:

$$R= \begin{bmatrix} 0.80&0.25&0.18\\ 0.72&0.82&0.25\\ 0.68&0.78&0.85 \end{bmatrix}, \qquad \bar b= \begin{bmatrix}0.10&0.20&0.15\end{bmatrix},$$

the derived ledger is:

final ACC:          (0.68 + 0.78 + 0.85) / 3 = 0.770
BWT terms:          [0.68 - 0.80, 0.78 - 0.82] = [-0.12, -0.04]
mean BWT:           -0.080
FWT terms:          [0.25 - 0.20, 0.25 - 0.15] = [0.05, 0.10]
mean FWT:           +0.075
per-task forgetting:[0.12, 0.04]
maximum forgetting: 0.120

The final mean looks strong while backward transfer is negative. A release protocol should additionally record base-capability retention, task-order permutations, task-ID oracle versus task-ID-free inference, state bytes, training work, active branch count, and rollback success. The O-LoRA paper’s reported final average accuracy is a source result under its protocol, not a substitute for this complete matrix.

0x06L Alternatives And Paper-By-Contract Comparison

Design	Task boundary required	Old data	State growth	How interference is constrained	Inference contract
immutable adapter + oracle task ID	yes	no	one adapter per task	isolate factors and select exactly one	task resolver chooses one artifact
cumulative adapter sum	task order only	no	cumulative rank or merged parent	no explicit functional constraint	all updates always active
O-LoRA paper	sequential tasks	no replay in reported method	cumulative task factors/basis state	squared factor-subspace overlap	cumulative adapted model; paper reports task-ID-free comparison
O-LoRA pinned code	sequential tasks and cumulative checkpoint	no replay in traced trainer	r_sum plus current factors	entrywise overlap $L_1$ plus unsquared factor norms	old and current branches share one scale before save concatenation
AM-LoRA paper	sequential tasks	no replay in core method	one LoRA and rater state per task	feature selector with rater-weight $L_1$	soft mixture over zero, old, and current branches
replay or distillation	task/stream protocol dependent	examples, logits, or teacher state	replay buffer or teacher artifacts	directly checks old examples/targets	method-specific
Online-LoRA	no explicit task ID in its task-free setting	method-specific online buffers/state	online regularization state	importance-weight regularization and loss-dynamics shift detection	task-free online boundary, not O-LoRA equivalence
consolidation/distillation	chosen release boundary	optional	bounded release, retained rollback parent	approximate function transfer	one consolidated artifact after evaluation

Online-LoRA is included to expose a missing axis: O-LoRA and AM-LoRA assume a task sequence, while a production stream may not provide clean boundaries. Loop 7 does not derive or reproduce Online-LoRA, replay, or consolidation methods. They are alternatives that require their own source, state, and evaluation contracts.

0x06M Executable Oracle And Failure Closure

The checked-in continual_adapter_state_fixture.json and continual_adapter_state_oracle.py produce continual_adapter_state_report.json. The deterministic report checks:

O-LoRA paper squared-overlap loss:             0.3125
pinned-code entrywise overlap loss:            0.75
task gradient into new A at zero new B:        0
maximum task gradient into new B:              0.875
homogeneous checkpoint-cat error:              0
heterogeneous naive-cat error:                 1.0
remaining exact orthogonal dimensions:         0
minimum AM-LoRA softmax gate:                  0.13447
AM output drift after old-rater-only mutation: 0.17426
continual parameters in the AM toy layer:      20
ACC / BWT / FWT:                               0.770 / -0.080 / +0.075
maximum forgetting:                            0.120

It also binds task order, parent chain, adapter and selector digests, targets, ranks, scales, training code, merge history, and evaluation identity. Two hundred four one-field mutations are rejected across sources, method contracts, numerical policy, O-LoRA factors, AM-LoRA branches, evaluation values, provenance, and forbidden claims.

Failure	Shape-correct symptom	Root cause	Required repair
paper/code loss collapse	both losses mention orthogonality	squared Frobenius was substituted for entrywise $L_1$, or vice versa	pin equation and code revision as separate contracts
old/new factors fail to match	training continues with some modules unregularized	name-prefix pairing differs between checkpoint and trainer	validate complete target-module pair map
regularizer dominates over time	new task underfits as task count grows	overlap penalty is not normalized by accumulated old rank/tasks	log task loss, overlap loss, factor loss, and gradient norms separately
zero-$B$ schedule misunderstood	$A$ moves before task gradient can reach it	regularizer updates $A$ while task path is blocked	trace task and regularizer gradients independently
heterogeneous scale drift	concatenated shapes load successfully	one forward scale was applied to factors trained under different scales	encode scales into blocks or preserve separate metadata/branches
orthogonal capacity exhausted	overlap cannot approach zero at retained rank	cumulative rank exceeds ambient dimension	reduce rank, share/project, replay, or consolidate under an explicit loss contract
frozen adapter still forgets	old factor files are unchanged	selector raters or newly summed deltas changed the function	evaluate old functions and version selector state
fake AM sparsity	rater weights are small but latency grows with $T$	softmax stayed dense and every branch feature was computed	add executable pruning policy and kernel trace
zero branch misread as free abstention	gate assigns mass to zero but work is unchanged	nonzero branches were materialized before selection	measure branch matmuls, not only gate values
final mean hides damage	ACC stays high	early-task loss is averaged with strong recent tasks	report $R$, BWT, per-task forgetting, and critical-task floors
task reorder treated as same release	tensor filenames and shapes match	ordered parent transition was omitted from identity	hash task order and rerun the full evaluation matrix
merged parent cannot resume	inference works, next task cannot reconstruct constraint	old basis/provenance was discarded	retain immutable factors/basis state or declare a new continual method

Loop 7 closes sequential adapter state. It does not close task/request/token routing granularity, sparse kernels, replay, task-free online continual learning, or paper benchmark reproduction. Loop 8 consumes this immutable state before changing route lifetime and scope.

0x07 Routing Granularity: Lifetime Is Not Placement

The owner question for this section is narrower than “which adapter wins?”:

At what lifetime is a routing decision made, at which layers and matrices does it apply, which information is legally available when it is made, and does the execution path actually skip unselected adapter work?

Those are four different questions. A route can be fixed for an entire task but still carry a different coefficient for every target matrix. A request-level route can be fixed throughout generation but differ at every layer. A token route can change at every row while still being shared across all adapted projections. Finally, a top-$k$ tensor is not evidence of sparse execution unless the runtime omits the other adapter matmuls.

Loop 8 therefore treats granularity as a product of decision lifetime, placement scope, feature availability, training lifecycle, and execution operator. It consumes the immutable adapter identities and sequential state from Loops 2—7. It does not redo generic MoE capacity from M6.1/M6.2, adapter residency from M6.5, multimodal architecture from M6.8, or grouped low-rank kernels from Loop 9.

0x07A Route Lifetime And Placement Are Different Axes

Let $B$ be requests, $S$ token positions, $L$ routed layers, $P$ adapted target matrices, and $J$ adapter choices. A useful logical route tensor is:

$$G \in \mathbb{R}^{B \times S_r \times L_r \times P_r \times J}.$$

The route extents $S_r$, $L_r$, and $P_r$ are policy choices. They need not equal the model extents:

Axis	Extent 1 means	Full extent means	Independent question
$S_r$	route persists across positions	route can change at every token	when does the decision change?
$L_r$	one route is shared across layers	every layer has its own route	where in depth does it apply?
$P_r$	one route is shared across target matrices	every adapted matrix has its own route	which projection does it modify?
$J$	one adapter candidate	an ordered adapter registry	which artifact/version is selected?

Define index maps $\phi_S(s)$, $\phi_L(\ell)$, and $\phi_P(p)$ that either preserve an index or map it to zero when that axis is shared. For input row $h_{bs\ell p}$ and adapter $j$ with scale $\gamma_{\ell pj}$, the common routed-adapter envelope is:

$$\Delta y_{bs\ell p} = \sum_{j=1}^{J} G_{b,\phi_S(s),\phi_L(\ell),\phi_P(p),j} \gamma_{\ell pj} B_{\ell pj}A_{\ell pj}h_{bs\ell p}.$$

This equation derives the broadcast rules. A task route sets $B=S_r=L_r=P_r=1$ and reuses one coefficient vector everywhere its artifact says to apply. A sequence-by-layer route uses $S_r=P_r=1$ but $L_r=L$, so one request keeps one distribution per layer. A token-by-layer route uses $S_r=S$ and $L_r=L$. Adding matrix-local routing makes $P_r=P$.

The number of logical gate entries follows immediately from the Cartesian product:

$$N_G = B S_r L_r P_r J.$$

For the Loop 8 toy with $B=2$, $S=3$, $L=2$, $P=2$, and $J=3$:

task:                       3 entries
request:                    2 * 3 = 6
sequence by layer:          2 * 2 * 3 = 12
token by layer:             2 * 3 * 2 * 3 = 36
token by layer and matrix:  2 * 3 * 2 * 2 * 3 = 72

This is a semantic count, not a command to materialize a dense five-dimensional tensor. A static route may be one artifact digest. A sparse route may be IDs and gates with logical shape:

$$I,V \in \mathbb{R}^{B \times S_r \times L_r \times P_r \times k},$$

where $I$ stores adapter IDs and $V$ stores selected-axis weights. Loop 9 will derive the packed execution plan. Loop 8 only establishes what those records mean.

0x07B Task-Level Composition: LoRAHub

LoRAHub is a task-level composition method, not an online request router. For a new task $\tau$ with a few-shot set $Q_\tau$, it searches one coefficient vector $w_\tau\in\mathbb R^J$ over an existing adapter pool. Its paper objective is:

$$w_\tau^* = \arg\min_w \left[ \mathcal L(Q_\tau;w) + \alpha\sum_{j=1}^{J}|w_j| \right].$$

The cost is paid while constructing the task artifact. Once $w_\tau^*$ is chosen, ordinary examples from task $\tau$ do not rerun CMA-ES or another gradient-free search. The deployed state is one fixed composition and therefore has task lifetime even when many requests use it.

The paper’s Figure 1 says this directly: composition is conducted per task rather than per example.

Primary-source figure. LoRAHub Figure 1, arXiv v3/COLM 2024. The diagram supports task-level timing. Its benchmark bars are source-reported and are not reproduced by this article.

The paper and pinned official repository must still remain separate numerical contracts:

Contract	Paper	Pinned code
candidate admission	equal-rank factor discussion	checks matching state-dict tensor shapes
regularizer	$\alpha\sum_j	w_j
search	gradient-free, primarily CMA-ES in reported setups	bounded Nevergrad NGOpt budget
output	one task-adapted module	coordinate-wise state-dict sum, then merge_and_unload

Loop 6 already showed that the paper’s factor-wise expression and a coordinate-wise state-dict sum are not universally interchangeable because factor products can introduce cross terms. Loop 8 uses only the timing invariant: one search result becomes one reusable task artifact.

def build_task_adapter(few_shot_task_set, ordered_adapter_pool, budget):
    assert all_compatible_with_same_base_and_targets(ordered_adapter_pool)

    def objective(weights):
        candidate = compose_under_pinned_coordinate_contract(
            ordered_adapter_pool, weights
        )
        return task_loss(candidate, few_shot_task_set) + regularizer(weights)

    weights = gradient_free_search(objective, budget=budget)
    return materialize_versioned_task_artifact(weights, ordered_adapter_pool)

def serve_task_request(request, task_artifact):
    # No gradient-free search in the request path.
    return generate(request, adapter=task_artifact)

0x07C Prompt-Persisted Request Routing: MiLoRA

MiLoRA occupies a different coordinate. It computes routes from a request’s prompt hidden states immediately before generation, then persists those decisions for subsequent tokens. Its route lifetime is therefore prompt/request, while its placement is layer specific and module specific.

For layer $\ell$, let $H_{b\ell}^{prompt}\in\mathbb R^{S_b\times d}$ be the prompt representation and $M_b$ its attention mask. A pooler produces one request feature:

$$\bar h_{b\ell} = \operatorname{Pool}_{\ell} \left(H_{b\ell}^{prompt},M_b\right) \in\mathbb R^d.$$

The paper sends that feature through a layer-specific activation and router, then applies top-$k$ over LoRA module choices:

$$z_{b\ell} = g_\ell(\bar h_{b\ell})W_{r,\ell}, \qquad \mathcal E_{b\ell} = \operatorname{TopK} \left(\operatorname{softmax}(z_{b\ell}),k\right).$$

For a LLaMA-style block, the candidates in the paper are module placements such as Q, K, V, O, gate, up, and down. A request can therefore select one module at layer 3 and another at layer 20. Calling this simply “sequence routing” would hide the layer and module axes.

Primary-source figure. MiLoRA Figure 1, Findings of EMNLP 2024. It supports prompt-conditioned, per-layer module selection and route reuse. It is not official-code evidence and does not reproduce the paper’s latency results.

The paper considers last-token, average, max, and self-attention pooling. It also uses learnable rational activation functions and describes bi-level optimization for activation parameters $\Theta$ versus remaining router/expert parameters $\Omega$. Its load term follows a Switch-style prompt assignment ledger. For layer $\ell$ and module expert $i$:

$$\mathcal L_{lb} = \frac{1}{L} \sum_{\ell=1}^{L} N_{mod} \sum_{i=1}^{N_{mod}} f_{\ell i}\hat p_{\ell i},$$

where $f_{\ell i}$ is the fraction of prompts assigned to module $i$ and $\hat p_{\ell i}$ is its mean router probability. The final training objective and the activation-search procedure are paper-specific; neither follows from prompt persistence alone.

The serving state must include more than selected integers:

def prepare_prompt_route(prompt, model, routers, adapter_registry):
    hidden_by_layer = prefill_and_capture_router_inputs(prompt, model)
    layer_routes = []

    for layer, hidden in enumerate(hidden_by_layer):
        pooled = routers[layer].pool(hidden, mask=prompt.attention_mask)
        logits = routers[layer].activation(pooled) @ routers[layer].weight
        ids, gates = selected_axis_topk(logits, k=routers[layer].k)
        layer_routes.append({
            "ids": adapter_registry.resolve_versions(ids),
            "gates": gates,
            "router_revision": routers[layer].revision,
            "pooling_policy": routers[layer].pooling_policy,
        })

    return persist_request_route(layer_routes)

def decode_with_prompt_route(request, token_row):
    assert request.route_state.is_pinned_and_resident()
    return model_decode(token_row, layer_routes=request.route_state)

Recomputing this route after preemption, or silently substituting a newly deployed router, changes the model. The route is request state and must be checkpointed with scheduler state.

0x07D Sequence- And Instance-Level Routing

Two methods can both emit one route per example and still have different contracts.

0x07D.1 Mixture-of-LoRAs: Sequence By Layer

The Mixture-of-LoRAs lifecycle was derived in 0x03G. It separately trains domain specialists, then performs a balanced Stage 2 in which one router at each layer predicts a sequence-level domain distribution. Its logical route is:

$$G^{MoA} \in \mathbb R^{B\times 1\times L\times 1\times J}.$$

Domain labels supervise $\mathcal L_{cls}$ during Stage 2. They are targets, not proof that an oracle domain ID is supplied at inference. The sequence representation, router architecture, layer voting policy, and final selection policy remain part of the artifact.

0x07D.2 Conditional MixLoRA: Instance-Local Factor Construction

Conditional MixLoRA routes rank-one factors, not complete pre-trained LoRA modules. At one target layer, the paper starts from preceding-layer hidden states $H_b\in\mathbb R^{S_b\times d_{in}}$ and forms one instance feature:

$$r_{IFS}^{A}(H_b) = \operatorname{Avg}_{s}(H_{bs}) \in \mathbb R^{d_{in}}.$$

An independent factor selector chooses $r$ entries from an expanded pool of $E$ input factors:

$$g_b^A = \operatorname{TopR} \left(\operatorname{softmax} \left(W_{IFS}^{A}r_{IFS}^{A}(H_b)\right) \right).$$

If $a_e\in\mathbb R^{1\times d_{in}}$ and $b_e\in\mathbb R^{d_{out}\times1}$, selected factors construct:

$$A_b = \begin{bmatrix} a_{i_1}\\ \vdots\\ a_{i_r} \end{bmatrix}, \qquad B_b = \begin{bmatrix} b_{j_1}&\cdots&b_{j_r} \end{bmatrix}.$$

The conditional factor selector for $B$ additionally depends on selected $A$ factors. The resulting $B_bA_b$ is one instance-local low-rank update at that wrapped layer and is broadcast across the sequence rows for request $b$.

Primary-source figure. Conditional MixLoRA Figure 3, ACL 2024. This section uses the figure only to establish instance-level A/B factor selection. The paper’s multimodal architecture and training context are outside this article’s scope.

The pinned official code makes the shapes concrete:

selector input:               [batch, sequence, hidden]
input_x.mean(dim=1):          [batch, hidden]
selected A factor IDs:        [batch, rank]
gathered A factors:           [batch, rank, d_in]
gathered B factors:           [batch, rank, d_out]
constructed A after rearrange:[batch, d_in, rank]
constructed B:                [batch, rank, d_out]
x @ (A @ B):                  [batch, sequence, d_out]

That code trace also exposes a batching defect that a shape table alone misses: the inspected selector uses input_x.mean(dim=1) without an attention mask. In a padded batch, zero or nonzero pad rows participate in the route feature unless an outer caller has removed them.

The paper is multimodal and does not establish a general autoregressive decode policy for repeatedly averaging an evolving text sequence. A text-serving implementation must choose explicitly between prompt-only pooling, a running causal pool, or rerouting. It cannot average future tokens that do not yet exist.

0x07E Token Routing: SiRA

SiRA supplies the sparse token-level counterpoint. For token row $x_s$:

$$G(x_s) = \operatorname{TopK} \left( \operatorname{softmax}(\theta_g^\top x_s) \right),$$

and the adapter output is:

$$y_s = W_0x_s + \sum_{e=1}^{E} G_{s,e}W_e(x_s).$$

Here $G_{s,e}$ changes with $s$. The paper also adds expert dropout, a balancing term, and a capacity limit. If all $K$ selected experts are full, the overflow token uses only the frozen base path in the reported design. Capacity therefore changes the executed function: “selected by score” and “admitted to an expert buffer” are distinct states.

Primary-source figure. SiRA Figure 1, arXiv v1. It supports the paper’s token top-$k$, dropout, capacity, and base fallback. It is not an official code or GPU-kernel trace.

M6.1 and M6.2 own the generic capacity formula, load balance, token dropping, and distributed dispatch. The M6.9-specific consequence is that adapter route metadata now has at least one record per token, layer, target scope, and selected slot:

(request_id, token_position, layer_id, target_id,
 route_slot, adapter_id, gate, local_row, adapter_version)

Without that mapping, inverse scatter can be shape-correct while assigning an expert output to the wrong token.

0x07F Token-By-Layer Dense Routing: X-LoRA

X-LoRA makes both token and layer axes explicit. Adding batch to the paper’s notation gives:

$$\Lambda \in \mathbb R^{B\times S\times L\times J}, \qquad \sum_{j=1}^{J}\Lambda_{bs\ell j}=1.$$

The default route is dense. At wrapped layer $\ell$:

$$y_{bs\ell} = W_{0,\ell}x_{bs\ell} + \sum_{j=1}^{J} \Lambda_{bs\ell j} \gamma_{\ell j} B_{\ell j}A_{\ell j}x_{bs\ell}.$$

The paper and pinned code use a two-pass schedule:

1. Scaling pass. Run the base with adapters inhibited or fixed, collect hidden states, and predict $\Lambda$.
2. Forward pass. Run the adapter-wrapped model using the predicted token-by-layer scalings.

Primary-source figure. X-LoRA Figure 19, arXiv v2. It supports the two-pass graph and adapter scaling. It does not establish sparse default execution or validate every optional code path.

The pinned official implementation exposes four consequential details:

classifier output: [batch, sequence, wrapped_layers, adapters]
layerwise_scalings=False:
    predict [batch, sequence, adapters]
    repeat the route across wrapped layers
Python integration:
    assert model.config.use_cache is False
default wrapped linear:
    loop over every active adapter and add every branch

The paper warns that the same K/V cache cannot simply be reused between the scaling and forward passes because hidden state and cache accumulation belong to different effective executions. It describes a separate mistral.rs path and a non-granular scaling option that caches scalings after a configured number of completion runs. That option is a different route-lifetime policy, not merely a kernel optimization.

0x07F.1 Pinned Optional Top-K Normalization Counterexample

The pinned optional top-$k$ helper masks selected entries, then applies:

softmax(xlora_scalings[nonzero_mask], dim=-1)

Boolean indexing flattens all selected values from all token rows. The correct selected-axis normalization for row $r$ is:

$$\hat g_{rj} = \frac{g_{rj}\mathbf 1[j\in\mathcal E_r]} {\sum_{q\in\mathcal E_r}g_{rq}}.$$

The pinned operation instead has one denominator across every selected pair:

$$\tilde g_{rj} = \frac{\exp(g_{rj})} {\sum_{(r',q)\in\mathcal U}\exp(g_{r'q})}, \qquad \mathcal U=\bigcup_{r'}\{(r',q):q\in\mathcal E_{r'}\}.$$

For dense inputs [.70,.20,.10] and [.10,.60,.30] with $k=2$:

correct row-wise top-k:
    [.77778, .22222, 0]
    [0, .66667, .33333]
    row sums = [1.0, 1.0]

pinned flattened selected-value softmax:
    [.31430, .19063, 0]
    [0, .28439, .21068]
    row sums = [.50493, .49507]

The global selected sum is one, but neither token row is normalized. This is a pinned-code caveat, not a claim that the X-LoRA paper defines top-$k$ this way.

0x07G Causality, Pooling, And Feature Availability

Routing granularity is only meaningful if the route feature exists when the decision is needed.

Lifetime	Legal feature examples	Illegal shortcut	State consequence
task search	few-shot task inputs and targets in the declared search set	test labels	one task artifact and search provenance
prompt/request	prompt tokens, prompt mask, prompt hidden states	future generated tokens	persisted per-request route
sequence	encoder/full-sequence state when the task contract permits it	hidden rows outside the declared input	one route per sample, possibly per layer
token	current layer input and causal prefix	future tokens or same-layer output that depends on the route being chosen	route metadata per token/layer

The same-layer circularity is easy to miss. If $\Delta_\ell$ needs route $G_\ell$ but $G_\ell$ is predicted from the post-adapter output $h_{\ell+1}$, then:

$$h_{\ell+1} = F_\ell(h_\ell,G_\ell), \qquad G_\ell = R_\ell(h_{\ell+1}),$$

which is an implicit equation, not the advertised one-pass router. A conventional routed layer predicts $G_\ell$ from $h_\ell$, metadata, or a prior pass.

0x07G.1 Masked Pooling Derivation

For variable prompt lengths, average pooling must exclude padding:

$$\bar h_b = \frac{ \sum_{s=1}^{S}M_{bs}H_{bs} }{ \sum_{s=1}^{S}M_{bs} }.$$

The denominator is the count of valid positions, so identical content has the same pooled representation regardless of batch padding. The padding-blind mean uses $S$ instead and therefore scales or contaminates the feature.

The oracle’s first request has valid rows [2,0], [0,2] and a padded sentinel [100,100]:

mask:                 [1, 1, 0]
masked mean:          [1, 1]
padding-blind mean:   [34, 34]
maximum difference:  33

The second request has no padding and both means equal [2/3, 2/3]. This paired witness proves that the discrepancy comes from padding, not from the pooling implementation in general.

Last-token pooling has a parallel constraint: in a left- or right-padded batch, “last array row” is not necessarily “last valid prompt token.” Self-attention pooling must mask keys and normalize only over valid rows. Pooling policy, mask convention, and tokenizer padding side are model state.

0x07H Route Persistence, Decode, And K/V Provenance

Prompt routing trades adaptation frequency for persistent state. Suppose MiLoRA-style prefill chooses IDs and gates $\rho_{b\ell}$ once. The request state is:

$$\rho_b = \left( \text{router revision}, \text{pooling policy}, \{\text{adapter ID},\text{version},\text{gate}\}_{\ell} \right).$$

It must survive scheduler preemption, migration, beam expansion, and restore. Reconstructing it from a mutable registry is not equivalent to restoring it.

The conservative prefix/KV identity is:

$$K_{prefix} = H\left( \text{base revision}, \text{router revision}, \text{adapter versions}, \rho_b, \text{prefix tokens}, \text{cache policy} \right).$$

Why include the route when LoRA does not directly target K or V? Because an adapter applied to Q, O, or an MLP changes residual states; later layers project those states into K/V. A route change invalidates K/V provenance even if the changed matrix is not named k_proj or v_proj.

def restore_request(snapshot, registry):
    route = snapshot.route_state
    assert registry.has_exact_versions(route.adapter_versions)
    assert route.router_revision == snapshot.router_revision
    assert kv_cache.key == hash_execution_config(snapshot, route)
    pin(route.adapter_versions)
    return RequestState(route=route, kv=kv_cache)

def change_route_after_kv_exists(request, new_route):
    if new_route.signature != request.route.signature:
        invalidate_or_rebuild_affected_kv(request)
    request.route = new_route

For X-LoRA’s two-pass graph, the issue is stricter: the scaling pass and adapter-weighted forward are different executions. The pinned Python path asserts use_cache=False; a runtime claiming cache reuse must document how it separates, repairs, or recomputes cache state.

0x07I Routing Scope Does Not Determine Training Lifecycle

The same route shape can be produced by incompatible training processes:

Method	Adapter origin	Router/scaler training	Route lifetime	Runtime artifact
LoRAHub	pre-trained task adapters	gradient-free task search	task	static composed adapter
MiLoRA	jointly initialized module-level LoRA experts	routers, experts, and learnable activations under paper schedule	prompt/request	persisted per-layer module route
Mixture-of-LoRAs	independently trained domain adapters	balanced Stage 2 updates routers and LoRAs	sequence by layer	routers, adapter pool, ordering, voting policy
Conditional MixLoRA	expanded rank-one factor pools	IFS/CFS selectors and factors in multimodal tuning	instance by layer/target	selectors plus factor pools
SiRA	LoRA experts initialized for sparse adaptation	token gate, experts, dropout, capacity/balance objective	token	sparse route plus capacity policy
X-LoRA	separately trained adapters	frozen base/adapters, train scaling head in reported core	token by layer	scaling head plus full adapter pool

Consequently:

route shape != training provenance
route shape != expert coordinate compatibility
route shape != sparsity
route shape != mergeability

A design review must record all four independently.

0x07J Dense Weights Versus Sparse Execution

For dense softmax routing, every finite logit has a positive gate:

$$g_j = \frac{e^{z_j}}{\sum_qe^{z_q}} > 0.$$

If the implementation loops over all $J$ adapters, routing changes arithmetic weights but saves no adapter branch matmul. That is the pinned X-LoRA default path and the dense envelope used by several methods earlier in this article.

For sparse top-$k$ routing, the mathematical output has at most $k$ nonzero branches:

$$y = W_0x + \sum_{j\in\mathcal E(x)} \hat g_jB_jA_jx, \qquad |\mathcal E(x)|=k.$$

Actual compute is sparse only if the execution trace satisfies:

unselected adapter IDs absent from packed segments
unselected A/B tensors not read for that row
no dense all-adapter output tensor formed before masking
selected rows mapped back through route-slot metadata
capacity or residency fallback recorded explicitly

Top-$k$ after all expert outputs have already been formed is sparse weighting, not sparse computation. Soft gates with small values are not zero. An $L_1$ penalty on router parameters is not an execution plan. Loop 9 will convert these conditions into traffic, launch, and grouped-GEMM contracts.

0x07K Paper-By-Contract Comparison

Contract	LoRAHub	MiLoRA	Mixture-of-LoRAs	Conditional MixLoRA	SiRA	X-LoRA
decision source	few-shot task set	pooled prompt hidden	sequence representation	preceding-layer sequence mean	token hidden row	scaling-pass hidden row
lifetime	task	request/prompt	sequence	instance at each wrapped layer	token	token
layer axis	artifact-defined	layer-specific	layer-specific	target-layer-local	target-layer-local	explicit layer axis
selected unit	adapter composition	module-level LoRA placement	domain adapter	rank-one A/B factors	LoRA expert	complete adapter output
dense/sparse	static coefficients	paper top-$k$	distribution then selection/vote	top-$r$ factors	top-$k$ + capacity	dense default
decode state	static artifact	persist route	selection policy	not closed for general text decode	route each token	two-pass/cache-specific
privileged signal	task examples during search	none required at inference	domain label is training target	multimodal conditioning fields are method-specific	no task ID required in paper setting	none required beyond hidden states

This table is not a leaderboard. It prevents one method’s route semantics from being used to explain another method’s latency or training claims.

0x07L Executable Oracle And Failure Closure

The checked-in routing_granularity_fixture.json and routing_granularity_oracle.py produce routing_granularity_report.json. The exact report checks:

logical entries, task/request/sequence-layer/token-layer/full: 3/6/12/36/72
masked request-0 mean:                                        [1, 1]
padding-blind request-0 mean:                                 [34, 34]
prompt route reused over decode steps:                        4
route change changes K/V provenance digest:                   true
correct top-k row sums:                                       [1.0, 1.0]
pinned X-LoRA optional top-k row sums:                         [.50493, .49507]
dense materialized branches / sparse selected branches:       3 / 2
LoRAHub search evaluations per task / request:                 12 / 0

Every scalar in the fixture receives a one-field mutation. The test suite rejects changes to sources, method contracts, route axes, route values, pooling masks, persistence state, causality, the pinned X-LoRA witness, execution counts, and forbidden claims.

Failure	Shape-correct symptom	Root cause	Required repair
lifetime/placement collapse	one route tensor broadcasts	”layer routing” was treated as a lifetime	record $S_r$, $L_r$, and $P_r$ separately
task search in request path	outputs look valid but latency explodes	task artifact construction confused with online routing	materialize and version task result once
prompt route silently changes	decode remains numerically finite	router/registry changed after prefill	persist exact IDs, gates, revisions, and policy
padding changes adapter	batch shape is valid	sequence mean ignored mask	masked pool with valid-token denominator
oracle label leak	evaluation accuracy rises	domain/task label used as inference input though only trained as target	enumerate legal runtime features
same-layer circular router	graph appears to have one route and one adapter	route predicted from output that already depends on route	use layer input, metadata, or explicit prior pass
fake sparse token route	top-$k$ IDs are logged	all adapter outputs computed before mask	trace packed rows and omitted tensor reads
selected-axis bug	global selected gates sum to one	Boolean indexing flattened rows before softmax	normalize independently per logical route row
stale K/V accepted	cache dimensions match	route state omitted from cache identity	invalidate/rebuild or prove unaffected dependency cone
MixLoRA called token routing	per-token outputs differ after base attention	one instance matrix was broadcast across rows	trace selector input and selected-ID shapes
MiLoRA called global routing	one request has one route object	object contains different layer/module choices	inspect internal route axes, not container count
dense X-LoRA called sparse	many gates are small	every adapter remains in the forward loop	count branch matmuls and tensor reads

Loop 8 closes routing semantics and persistent state. It does not claim to execute a GPU kernel, reproduce paper benchmarks, or close Loop 9 runtime packing.

0x07M Loop 9 Handoff Sketches

The following sketches are deliberately incomplete. They show the data that Loop 9 must consume; they are not yet a launch, traffic, occupancy, or scheduler proof.

0x07M.1 Independent Factors

def routed_lora_independent(x, route_ids, route_gates, adapters):
    # x: [T, d_in], route_ids/gates: [T, k]
    plan = build_route_plan(route_ids)
    # Every entry binds (token_row, route_slot, expert_id, local_row).
    packed_x = gather_rows(x, plan.token_rows)

    packed_dy = empty_output(plan.num_routes, adapters.d_out)
    for expert_id, segment in plan.expert_segments:
        adapter = adapters.pin(expert_id)
        z = segment.rows @ adapter.A.T
        segment.output = adapter.scale * (z @ adapter.B.T)

    packed_dy *= route_gates[plan.token_rows, plan.route_slots, None]
    return scatter_add(packed_dy, plan.token_rows, output_rows=x.shape[0])

0x07M.2 Shared A

def routed_lora_shared_a(x, route_ids, route_gates, shared_A, B_experts):
    z = x @ shared_A.T                     # [T, r], once per row
    plan = build_route_plan(route_ids)
    packed_z = gather_rows(z, plan.token_rows)
    packed_dy = grouped_expert_linear(
        packed_z,
        expert_ids=plan.expert_ids,
        local_offsets=plan.local_offsets,
        expert_weights=B_experts,
    )
    packed_dy *= route_gates[plan.token_rows, plan.route_slots, None]
    return scatter_add(packed_dy, plan.token_rows, output_rows=x.shape[0])

Shared A removes repeated input projections. It does not remove row packing, expert-specific B reads, gate weighting, inverse scatter, residency, or version checks.

0x07M.3 Persisted Prompt Route

def decode_step(request, x_t):
    route = request.persisted_route
    assert route.signature == request.kv.execution_signature.route
    assert all_exact_adapter_versions_resident(route)
    return base_linear(x_t) + apply_layer_local_adapter_route(x_t, route)

0x08 Runtime Execution: From Logical Routes To Versioned Work

Loop 8 ended with logical route IDs, gates, decision lifetime, and K/V provenance. Those records still do not execute a low-rank branch. Loop 9 owns the missing transformation:

logical route rows
    -> route occurrences
    -> stable physical plan
    -> grouped low-rank work
    -> gate and scatter
    -> versioned completion under an M6.5 lease

The owner boundary matters. M6.5 owns adapter admission, paging, resident slots, request scheduling, hotswap publication, K/V compatibility, and reclamation. This section consumes those services. It owns only the additional work created when one row can select several adapters, when factors may be shared, and when route metadata changes by token or layer.

0x08A Runtime ABI: A Row Can Produce Several Occurrences

Let the layer input be $X\in\mathbb R^{T\times d_{in}}$. For row $t$, the router supplies a selected support $\mathcal S_t$, adapter IDs $I_{t,s}$, and selected-axis gates $g_{t,s}$. The number of physical route occurrences is

$$R = \sum_{t=1}^{T}|\mathcal S_t|.$$

Only under fixed top-$k$ without capacity rejection is $R=Tk$. An overflow or base-only row can make $R<Tk$; an all-expert soft mixture makes $R=TJ$. The physical ABI therefore flattens selected pairs rather than pretending that every batch has the same work:

token_row   [R] int32
route_slot  [R] int32
expert_id   [R] int32
local_row   [R] int32, assigned after packing
gate        [R] compute or accumulation dtype
scale       [R] or one scale per expert

For the teaching route

t0 -> (e0, .75), (e2, .25)
t1 -> (e1, .60), (e0, .40)
t2 -> (e2, .50), (e1, .50)
t3 -> (e0, .80), (e1, .20)

the row-major occurrence IDs are $[0,1,2,3,4,5,6,7]$. A stable sort by (expert_id, token_row, route_slot) gives

permutation:       [0, 3, 6, 2, 5, 7, 1, 4]
segment experts:   [0, 1, 2]
segment offsets:   [0, 3, 6, 8]

The first segment contains three copies of input rows $[t_0,t_1,t_3]$ for expert 0. The second contains $[t_1,t_2,t_3]$ for expert 1. The final segment contains $[t_0,t_2]$ for expert 2. This is not a token permutation alone: one token can occur in two segments.

One exact pack plan is:

def build_route_plan(route_ids, route_gates, expert_count):
    occurrences = []
    for token_row, (ids, gates) in enumerate(zip(route_ids, route_gates)):
        for route_slot, (expert_id, gate) in enumerate(zip(ids, gates)):
            occurrences.append({
                "token_row": token_row,
                "route_slot": route_slot,
                "expert_id": expert_id,
                "gate": gate,
            })

    perm = stable_argsort(
        occurrences,
        key=lambda r: (r["expert_id"], r["token_row"], r["route_slot"]),
    )
    offsets = histogram_prefix_sum(
        [occurrences[p]["expert_id"] for p in perm],
        expert_count,
    )

    for local_row, occurrence_id in enumerate(perm):
        occurrences[occurrence_id]["local_row"] = local_row

    return occurrences, perm, offsets

Sorting is one implementation, not a mathematical requirement. A bounded expert pool can use histogram counts, prefix sums, and a stable scatter. A direct-indexed kernel can retain row-major occurrences and fetch factors by expert_id. Whichever plan is chosen, (token_row, route_slot) -> (expert_id, local_row, gate) must remain invertible until scatter-add completes. Indexing an expert-major output buffer by original token ID is wrong.

0x08B Independent-Factor Forward: Pack, Shrink, Expand, Scatter

For independent factors

$$A_j\in\mathbb R^{r\times d_{in}}, \qquad B_j\in\mathbb R^{d_{out}\times r},$$

one occurrence $q=(t,s)$ computes

$$z_q=A_{I_q}x_t\in\mathbb R^r,$$ $$d_q=B_{I_q}z_q\in\mathbb R^{d_{out}},$$

and the adapter output is

$$y_t^{adapter} = \sum_{q:\,token(q)=t} g_qs_{I_q}d_q.$$

scale here is the LoRA scale $s_j$, not the router gate. Folding $s_j$ into $A_j$ or $B_j$ can remove one runtime multiply, but it changes artifact and pointer-table identity. The plan must say where scale lives.

The grouped schedule is:

def independent_route_forward(X, plan, factors):
    delta = zeros([X.rows, factors.d_out])

    for expert_id, begin, end in plan.segments:
        occurrence_ids = plan.permutation[begin:end]
        token_rows = [plan.occurrence[q].token_row for q in occurrence_ids]
        X_j = gather_rows(X, token_rows)                 # [R_j, d_in]
        Z_j = X_j @ factors[expert_id].A.T              # [R_j, r]
        D_j = Z_j @ factors[expert_id].B.T              # [R_j, d_out]

        for local, occurrence_id in enumerate(occurrence_ids):
            row = plan.occurrence[occurrence_id]
            coefficient = row.gate * factors[expert_id].scale
            delta[row.token_row] += coefficient * D_j[local]

    return delta

The scalar coefficient may be applied to $z_q$ before the linear $B_j$ expansion or to $d_q$ afterward because

$$B_j(c_qz_q)=c_qB_jz_q.$$

That algebra does not authorize moving token-dependent dropout, nonlinear normalization, or a vector-valued gate across $B_j$. It also does not eliminate the need to retain the coefficient for router and factor gradients during training.

0x08C Shared A: Reuse Proof And Its Exact Preconditions

HydraLoRA-style ownership changes the operator to

$$y_t^{adapter} = \sum_{q:\,token(q)=t} g_qs_{I_q}B_{I_q}Ax_t.$$

Because every selected branch consumes the same $A$ and the same row $x_t$, compute

$$z_t=Ax_t$$

once per token, gather $z_t$ by route occurrence, and group only the $B_j$ expansions. This is exact because substitution gives

$$\sum_{q:\,token(q)=t}g_qs_{I_q}B_{I_q}z_t = \sum_{q:\,token(q)=t}g_qs_{I_q}B_{I_q}Ax_t.$$

It is not valid merely because several independently owned $A_j$ tensors happen to start with equal values. After one optimizer step, separate moment state or gradients can make them diverge. Reuse requires all of the following at the launch boundary:

the same A storage/version and multiplication orientation
the same input row before A
the same dropout mask or inference-time identity dropout
compatible logical and physical rank
compatible placement of LoRA scale
no branch-local nonlinear operation between A and B

With $R$ route occurrences, the arithmetic ledgers are

$$F_{independent} \approx 2Rr(d_{in}+d_{out}),$$ $$F_{sharedA} \approx 2Trd_{in}+2Rrd_{out},$$

so the saved shrink work is

$$\Delta F_A = 2(R-T)rd_{in}.$$

For fixed top-$k$, this reduces to $2T(k-1)rd_{in}$. In the exact fixture, $T=4$, $R=8$, $r=2$, $d_{in}=3$, and $d_{out}=2$:

independent factors:  2 * 8 * 2 * (3 + 2) = 160 FLOPs
shared A:             2 * 4 * 2 * 3 + 2 * 8 * 2 * 2 = 112 FLOPs
saved shrink work:    48 FLOPs

These are arithmetic counts under the article’s multiply-add convention, not latency predictions.

0x08D Packed Backward: Segment Reductions And Shared Ownership

Let $G_t=\partial\mathcal L/\partial y_t^{adapter}$. For occurrence $q=(t,s)$, define

$$c_q=g_qs_{I_q}, \qquad u_q=c_qG_t.$$

For independent factors, $z_q=A_{I_q}x_t$ and the local derivatives are

$$dB_j = \sum_{q:I_q=j}u_qz_q^{\top},$$ $$dz_q=B_j^{\top}u_q,$$ $$dA_j = \sum_{q:I_q=j}dz_qx_t^{\top},$$ $$dx_t^{adapter} = \sum_{q:token(q)=t}A_{I_q}^{\top}dz_q.$$

The selected gate derivative before the selected-axis softmax is assembled from

$$\frac{\partial\mathcal L}{\partial g_q} = s_{I_q}G_t^{\top}B_{I_q}z_q.$$

Loop 3 owns the softmax Jacobian and centered router credit. This packed backward holds the discrete top-$k$ support fixed. It does not claim a derivative through support changes at a tie or selection boundary.

For shared $A$, compute all branch-local $dz_q$ as above, then reduce by source token:

$$dz_t = \sum_{q:token(q)=t}dz_q.$$

Only after that reduction are the shared-factor gradients valid:

$$dA = \sum_t dz_t x_t^{\top}, \qquad dx_t^{adapter}=A^{\top}dz_t.$$

Computing one dA per expert and applying the optimizer three times would violate shared ownership even if all three arrays alias the same storage. A kernel may use segmented reductions, block-local accumulation, atomics, or a second reduction pass; the mathematical owner remains one shared gradient tensor.

def shared_A_backward(X, G, plan, A, B, scales, saved_Z):
    dB = zeros_like(B)
    dZ = zeros([X.rows, A.rank])
    dgate = zeros([plan.route_count])

    for q in plan.occurrences:
        t, j = q.token_row, q.expert_id
        z = saved_Z[t]                         # or recompute A @ X[t]
        u = q.gate * scales[j] * G[t]
        dB[j] += outer(u, z)
        dZ[t] += B[j].T @ u
        dgate[q.id] = scales[j] * dot(G[t], B[j] @ z)

    dA = dZ.T @ X
    dX_adapter = dZ @ A
    return dX_adapter, dA, dB, dgate

The Loop 9 oracle checks this backward against three independently represented clones of the same $A$, sums the three clone gradients, and then checks $X$, shared $A$, selected $B$, and gate coordinates by centered finite differences. The maximum teaching error is below $10^{-8}$.

0x08E Workspace, Traffic, And Liveness

The FLOP count hides several physically different schedules. If route occurrences are packed explicitly, a conservative workspace ledger contains

$$M_{meta} \approx R(4b_i+b_g),$$

for token row, route slot, expert ID, local row, and one gate. Materializing packed inputs and both low-rank intermediates adds

$$M_{pack} \approx R(d_{in}b_x+rb_z+d_{out}b_d).$$

An indexed batched-matrix implementation that copies every selected factor pair adds

$$M_{factor\ copy} \approx Rr(d_{in}+d_{out})b_f.$$

A direct-indexed kernel can make this particular copy term zero by loading the selected factors through expert_id. It still reads those factors from memory, still needs route indices, and can suffer irregular access or poor small-matrix utilization. “No selected-factor copy” is not “no factor traffic.”

The fixture uses four 32-bit indices and one FP64 gate per occurrence, so $R=8$ requires 192 bytes of metadata. FP64 occurrence intermediates require 128 bytes each for $Z_R$ and $D_R$. Copying eight selected FP16 factor pairs at $r=2$, $d_{in}=3$, and $d_{out}=2$ requires another 160 bytes. The numbers are tiny because the fixture is tiny; the formulas, not the byte magnitudes, are the reusable contract.

Training introduces a liveness decision:

Policy	Saved forward state	Backward work	Risk
save independent $Z_R$	$Rrb_z$ bytes	reuse each occurrence projection	activation growth with $R$
recompute independent $Z_R$	route plan and $X$	repeat about $2Rrd_{in}$ FLOPs	factors and dropout state must match
save shared $Z_T$	$Trb_z$ bytes	reuse once-per-token projection	exact only under shared-A contract
recompute shared $Z_T$	route plan and $X$	repeat about $2Trd_{in}$ FLOPs	same deterministic-state requirement

If dropout was active before $A$, recomputation must reproduce its mask or checkpoint the post-dropout input. If an adapter or router generation changes between forward and backward, recomputation is invalid even when shapes still match.

0x08F Prefill And Decode Are Different Regimes

For prefill, $T$ is often the sum of prompt rows admitted to the target layer. Larger segments can make grouped matrix work efficient, but selected-factor copies and $R$-scaled intermediates grow with prompt length. For one-token decode, $T$ is approximately the number of active requests. Segments can be very short, so host dispatch, sorting, pointer assembly, and launches can dominate the low-rank arithmetic.

prefill teaching regime:
    T=128, k=2 -> R=256 route occurrences
    many rows can amortize one grouped launch

decode teaching regime:
    T=16, k=2 -> R=32 route occurrences
    one row per active request; route skew can create tiny segments

Sequence routing can group requests by a route signature

$$\rho = (I_1,g_1,\ldots,I_k,g_k,\text{adapter versions},\text{router revision}),$$

but exact floating-point gates can create many unique signatures. Rounding or quantizing gates to improve batching changes the function and needs its own quality evaluation. Token routing cannot assume one signature persists across rows unless Loop 8’s route policy proves it.

Measurements must separate at least:

router time
route expansion and plan construction
residency/admission wait
pack or factor-index preparation
shrink and expand kernels
gate and scatter/reduction
prefill versus decode
cold versus warm adapter state

0x08G Residency, Generations, Leases, And K/V Handoff

A physical plan is valid only for the M6.5 state snapshot from which it was built. Bind the request to semantic revisions first, acquire resident slots and leases second, then record a launch snapshot:

base revision
router revision
selected adapter artifact versions
catalog generation
slot generation for every selected adapter
pointer-table generation for every selected adapter
route-plan generation
route/KV execution signature

def execute_routed_layer(request, X, route_state, runtime):
    assert route_state.digest == request.kv.execution_signature.route_digest
    assert route_state.router_revision == request.router_revision

    lease = runtime.acquire_exact_adapter_set(
        versions=route_state.selected_adapter_versions,
        deadline=request.deadline,
    )
    try:
        plan = build_route_plan(route_state.ids, route_state.gates, lease.pool_size)
        launch = runtime.snapshot(lease, plan.generation)

        assert launch.catalog_generation == request.catalog_generation
        assert launch.slot_generations == lease.slot_generations
        assert launch.pointer_generations == lease.pointer_generations

        delta = routed_low_rank_kernel(X, plan, launch.pointer_table)
        event = runtime.record_last_use(lease)
        return base_linear(X) + delta, event
    finally:
        runtime.release_after_last_use_event(lease)

The lease must cover plan construction through the final kernel use. Otherwise an adapter can be evicted or a slot can be republished after its pointer enters the plan but before the GPU consumes it. Comparing only adapter names is insufficient because one alias can resolve to a new artifact generation.

Even when adapters do not target K or V directly, their residual updates can change later hidden states that produce K/V. A cached prefix therefore belongs to

$$\text{KV key} = (\text{base},\text{router},\text{adapter versions}, \text{routing policy},\text{resolved route state},\text{prefix}).$$

Deterministic token routing may reconstruct historical routes from the same inputs and exact revisions rather than serializing every row. The cache still needs a digest proving that reconstruction contract. Sequence/prompt routing should persist the resolved route explicitly. Never continue an in-flight request with a newly published adapter generation and old K/V.

A miss on one selected adapter has only three honest outcomes: wait/admit it, reject/defer the request, or enter an explicitly evaluated degraded mode. Silently replacing it with a default adapter changes the function.

0x08H MeteoRA: Indexed BMM Versus Direct Factor Indexing

Primary-source figure. MeteoRA Figure 2, arXiv:2405.13053/ICLR 2025. It supports the routed LoRA architecture and top-$k$ branch combination. It does not reproduce the paper’s latency, memory, or task-quality measurements.

MeteoRA exposes three implementation contracts for the same logical token-routed operator.

1. Loop over adapters. Select each expert’s rows and run small matrix multiplications. The paper argues that this underutilizes hardware when each expert receives few rows, especially in decode when $b\times s<n$.
2. Indexed batched matrices. Flatten $b\times s\times k$ selections, index the corresponding $A/B$ factor pairs, and execute two BMMs. This parallelizes the arithmetic but materializes selected-factor copies.
3. Direct-indexed Triton kernel. Load the selected factor pair through the route ID inside the kernel, block the vector-by-low-rank products, and avoid the explicit $bsk$ factor-pair allocation.

The distinction is visible in the pinned official repository at revision 2d192ad. The traced Torch path performs

selected_loras = selected_loras.view(-1)
blora_A = lora_A_weights[selected_loras].contiguous()
blora_B = lora_B_weights[selected_loras].contiguous()
torch.bmm(torch.bmm(X_occurrence, blora_A), blora_B)

whereas the traced Triton v4 path passes the factor pools and flattened selected IDs to a kernel that indexes each selected factor. The repository also contains distinct backward kernels. That is useful implementation evidence, but this article did not compile or run those GPU paths.

MeteoRA’s paper reports an average speedup of roughly $4\times$ for its indexed Torch BMM over its original loop in its experiments, while its Triton path retains much of that time improvement with lower selected-factor-copy memory. Those are source-reported results under the paper’s shapes and hardware. They are not a universal ranking against M6.5’s Punica/S-LoRA paths, and they are not reproduced here.

0x08I LoRA-Switch: Change The Route Contract To Change The Kernel

Primary-source figure. LoRA-Switch Figure 2, arXiv:2405.17741. It supports the paper’s pre-gated architecture, fused weight-switching path, and different training/decode graphs. It is not evidence that arbitrary layer-local routed adapters can use the same kernel.

LoRA-Switch changes the model before it changes the kernel. It places a top-2 router at the first expanded linear layer and reuses that token route for all later expanded layers. If $\Delta W_j^{(\ell)}$ is adapter $j$‘s update for layer $\ell$, the fused weight for token $t$ is

$$\widetilde W_t^{(\ell)} = W^{(\ell)} + \sum_j g_{t,j}\Delta W_j^{(\ell)}.$$

Because the current fused weights already contain the previous token’s route, the next state can be written

$$\widetilde W_t^{(\ell)} = \widetilde W_{t-1}^{(\ell)} - \sum_j g_{t-1,j}\Delta W_j^{(\ell)} + \sum_j g_{t,j}\Delta W_j^{(\ell)}.$$

The paper concatenates the negative previous and positive current factors, then applies the updates for many layer matrices through one segmented gather matrix multiplication (SGMM) launch. Pointer and shape arrays describe heterogeneous layer segments; tiled work and double buffering address launch and loading overhead.

That schedule is not a drop-in optimization for arbitrary M6.9 routes:

required by LoRA-Switch:
    route known before all expanded layers
    same token route reused across those layers
    mutable fused weight state carried between decode steps
    explicit previous-route state for subtraction

not provided by a generic layer-local router:
    future layer routes before those layers execute
    one shared route across target matrices
    permission to mutate a multitenant shared base in place

In a batched server, requests can choose different routes. One globally mutated fused backbone cannot represent all request-specific weights simultaneously. A production integration would need route-homogeneous batching, isolated weight replicas, another indirection, or a different kernel. M6.5’s immutable-generation and lease rules still apply to whichever physical state is chosen.

0x08J Paper-By-Contract Runtime Comparison

Path	Logical route preserved?	Selected factor handling	Intermediate/workspace	Training support in this audit	Main regime/risk
Python expert loop	yes	one expert at a time	row-index lists and many launches	reference only	tiny segments and launch overhead
M6.5 sort/segment	yes	pointer per expert segment	permutation, offsets, packed rows	exact CPU oracle; GPU wrapper owned by M6.5	route planning and segment skew
MeteoRA indexed BMM	yes	copies $R$ selected $A/B$ pairs	$O(Rr(d_{in}+d_{out}))$ factor-copy bytes	pinned forward/backward source inspected	memory grows with route occurrences
MeteoRA direct Triton	yes	indexes pools in kernel	masks, blocked temporaries, route IDs	pinned code inspected, not executed	layout constraints and irregular loads
exactly shared $A$	yes under stated object graph	one $A$ projection, routed $B_j$	$Z_T$ plus B segments	exact forward/backward oracle	representational and optimizer coupling
LoRA-Switch SGMM	paper’s pre-gated route	updates many layer weights before forward	previous/current route and fused weights	paper algorithm audited, kernel not run	architecture change and mutable weight state
dense all-expert	yes for dense gate	materializes all $J$ branches	all expert outputs or streaming sum	logical reference	top-$k$ name does not prove this work vanished

No row in this table is universally best. The decision depends on route lifetime, factor sharing, rank, segment size, prefill/decode mix, resident pool, training needs, and whether changing the architecture is allowed.

0x08K Executable Oracle And Failure Closure

The checked-in runtime_execution_fixture.json and runtime_execution_oracle.py produce runtime_execution_report.json and make the section executable without claiming a GPU benchmark. The oracle proves:

R=8 route occurrences and segment offsets [0,3,6,8]
dense row-major reference == stable grouped pack/scatter
output is invariant to within-segment occurrence order
shared-A schedule == independent clones only when every A is identical
replacing unequal A_j with one shared A changes the output
shared dA == sum of cloned independent dA_j
shared dB and dX match the cloned independent graph
selected X, A, B, and gate gradients match centered finite differences
all selected adapters are resident under valid generation-bound leases
route state matches the K/V execution signature

Its work report records 160 independent-factor FLOPs, 112 shared-$A$ FLOPs, 192 bytes of route metadata, 160 bytes of teaching FP16 selected-factor copies for indexed BMM, and zero copy bytes for direct indexing while retaining factor reads. All GPU, latency, throughput, production-scheduler, and paper-benchmark claims remain false.

The generated one-field mutation corpus rejects changes to dimensions, route IDs or gates, factors, sharing preconditions, source digests, residency, slot or pointer generations, leases, K/V identity, execution alternatives, regimes, and claim ceilings. Loop 10 will consume this exact report rather than reconstructing the runtime graph from prose.

0x09 Cross-Contract Oracle: Compose State Without Collapsing Methods

Loops 2-9 each close one local question. That is necessary but not sufficient for a runnable system. A route can be numerically correct while naming the wrong adapter generation. A valid shared factor can be paired with a training history that produced independent factors. A correct continual checkpoint can be reopened under a reordered task lineage. An exact static merge can be attached to a live router even though those two composition contracts are mutually exclusive.

Loop 10 therefore asks a different owner question:

Can one request be traced from immutable model identity through training provenance, factor ownership, composition, continual state, route state, physical execution, K/V provenance, and rollback without silently substituting one method’s contract for another?

The answer is not one universal routed-adapter schema. The answer is a typed compatibility graph whose legal paths remain distinct.

0x09A A Contract Is A Typed Product, Not A Bag Of Fields

Represent one deployable adapter state as

$$\mathcal C = (I_B,I_M,I_T,I_F,I_C,I_S,I_R,I_E,I_K),$$

where

I_B  base model, config, tokenizer, and target-module identity
I_M  method family and paper/code contract
I_T  training lifecycle, trainable set, frozen set, and optimizer provenance
I_F  factor topology, rank, orientation, scale, storage owner, and versions
I_C  static or dynamic composition contract and artifact lineage
I_S  ordered continual state, task lineage, selector, and parent chain
I_R  route scope, support, gates, persistence, and causal feature provenance
I_E  physical plan, residency, slots, pointers, generations, and leases
I_K  K/V and prefix provenance affected by the routed model state

The tuple is typed. none is a valid value only when the selected lane does not own that state. Static fusion has no online router; a HydraLoRA-style shared-factor transaction has no O-LoRA task chain; O-LoRA has no attentional selector; AM-LoRA’s softmax selector does not imply a sparse physical plan.

Separate semantic and execution identities:

$$I_{model} = H(I_B,I_M,I_T,I_F,I_C,I_S),$$ $$I_{exec} = H(I_{model},I_R,I_E,I_K).$$

H is SHA-256 over canonical, typed, length-preserving serialization in the teaching oracle. A production system may use another collision-resistant content-addressing scheme. It must not hash an ambiguous concatenated string, omit field names, or treat two equal-shaped tensors as one object.

Runtime generations do not change the abstract function by themselves. They still belong to $I_{exec}$ because they decide which concrete slots and pointer-table rows the launch may dereference. This is why a rollback can restore the old function while producing a new execution identity.

0x09B Replay Before Composition

The Loop 10 oracle imports every prior fixture and oracle, rebuilds every report, compares the result with the checked-in report, and validates four immutable file digests per loop:

Loop 2: adapter-family identity
Loop 3: common routed operator and fixed-support gradients
Loop 4: method-specific training states
Loop 5: independent/shared-A/shared-B ownership and hypothesis class
Loop 6: static/dynamic composition, quantization order, and artifact identity
Loop 7: ordered continual state and evaluation matrix
Loop 8: route lifetime, causality, persistence, and K/V provenance
Loop 9: route occurrences, physical execution, backward, and leases

For loop $\ell$, let $F_\ell$, $O_\ell$, $P_\ell$, and $C_\ell$ denote the fixture, oracle, report, and contract bytes. Admission requires

$$H(F_\ell)=h^F_\ell, \quad H(O_\ell)=h^O_\ell, \quad H(P_\ell)=h^P_\ell, \quad H(C_\ell)=h^C_\ell,$$

and

$$O_\ell(F_\ell)=P_\ell.$$

The semantic digest stored inside $P_\ell$ is checked separately from the JSON file digest. The first identifies the report’s canonical meaning; the second catches serialization or checked-in byte drift. Loop 10 refuses to compose stale prose with current fixtures or current prose with stale reports.

0x09C Four Legal Lanes

The exact fixture admits four lanes. They share the frozen-base deployment envelope, not a training algorithm or runtime graph.

Lane	Training provenance	Factor topology	Composition	Route and materialization	Release state
shared-A selected	joint asymmetric training	one owned $A$, several $B_j$	dynamic input-conditioned	selected top-2 route occurrences; pack routed $B_j$	base + router + shared $A$ + $B$ pool
static fusion	post-training fixed coefficients	independent $A_j,B_j$	exact delta-space block concatenation	no online route; one fixed child delta	versioned merged or unmerged child
O-LoRA ordered	sequential tasks	ordered task-factor pairs	scale-aware cumulative concatenation	global cumulative update; no selector	ordered parents or exact deltas required for rollback
AM-LoRA selector	sequential tasks plus selector	task pairs plus zero branch	dynamic attentional mixture	dense softmax; all nonzero task branches in the teaching path	ordered history plus selector state

A compact legality predicate is

$$\operatorname{legal}(\mathcal C) = P_B\land P_M\land P_T\land P_F\land P_C \land P_S\land P_R\land P_E\land P_K.$$

This notation does not reduce compatibility to nine undocumented booleans. Each predicate has concrete checks:

P_B: exact parent base/config/tokenizer/target identity
P_M: method key resolves to one frozen paper/code contract
P_T: trainable and frozen sets match that method and stage
P_F: ranks, orientations, scales, ownership, and versions are compatible
P_C: dynamic routes are not packaged as static deltas; static coefficients are fixed
P_S: task order, parents, selector, and continual state identity agree
P_R: route features are causal and route lifetime/persistence is explicit
P_E: selected artifacts are resident under generation-bound leases
P_K: route/model state matches the cache and prefix execution signature

The predicate is intentionally stricter than shape checking. Every lane uses low-rank matrices, but that common shape does not make its factor owners, route scopes, update histories, or release artifacts interchangeable.

0x09D End-To-End Shared-A Request Transaction

The executable path uses the Loop 9 teaching tensors and the local shared_a_selected lane. It is not a reproduction of HydraLoRA training or a MeteoRA benchmark. It composes the already proved shared-A topology with the already proved selected-route runtime.

The request binds:

request                         request-loop10-0001
base                            teaching-base-r17
router                          teaching-router-r9
route-plan/catalog generations 41 / 31
selected adapter versions       adapter-0-r5, adapter-1-r3, adapter-2-r8
slot generations                11, 12, 13
pointer-table generations       21, 22, 23
shared-A storage/generation     shared-A-storage-r17 / 17
route/KV state digest           3d37e7...44c721
route occurrences               8 in three expert segments

The transaction identity is

$$I_{tx} = H( I_B,I_{lane},I_{router},I_{route},I_{factor}, I_{slot},I_{pointer},I_{lease},I_K ).$$

The exact teaching digest is:

a3593145c02635cf0b76d2035453c0ee8fdd3d2655567bd38a517ebb08f3604a

It identifies the whole request binding, not the adapter checkpoint alone.

One transaction trace is:

def execute_bound_shared_a_request(request, registry, cache):
    lane = resolve_exact_lane("shared_a_selected")
    assert lane.training_lifecycle == "joint_asymmetric_training"
    assert lane.factor_topology == "shared_A_multiple_B"
    assert not lane.static_merge_legal

    model_state = bind_model_identity(
        base=request.base_revision,
        router=request.router_revision,
        factor_pool=request.selected_adapter_versions,
        shared_A=(request.shared_A_storage_id, request.shared_A_generation),
    )
    assert model_state.semantic_id == request.expected_model_id

    route = bind_route(
        ids=request.route_ids,
        gates=request.route_gates,
        generation=request.route_plan_generation,
    )
    assert route.digest == cache.execution_signature.route_state_digest

    leases = registry.acquire_exact_versions(
        request.selected_adapter_versions,
        slot_generations=request.slot_generations,
        pointer_generations=request.pointer_table_generations,
    )
    try:
        assert all(lease.valid for lease in leases)
        assert one_exact_shared_A_owner(leases)
        plan = stable_route_occurrence_plan(route)
        delta = execute_shared_A_pack_segment_scatter(request.X, plan, leases)
        record_last_use_event(leases, delta)
        return request.base_output + delta
    finally:
        release_only_after_last_use(leases)

The report reproduces the Loop 9 shared-A output:

[[-0.1875,  0.0000],
 [-2.6000,  2.1000],
 [-0.3750,  2.2500],
 [-2.4000,  1.5000]]

Its output digest is:

1f5e91f4aab7bea475dd00644f4bf0f9588cab039c28204f9a21325743618f45

The plan retains eight occurrences, the shared-A schedule saves 48 arithmetic FLOPs in the tiny fixture, and the maximum finite-difference error is approximately $7.11\times10^{-10}$. None of those facts measures production latency.

0x09E Backward And Optimizer Ownership Survive Composition

An integrated forward does not authorize mixing training objectives. The shared-A lane consumes:

Loop 3: selected-axis gate derivative on fixed support
Loop 5: one shared A owner and coupled branch gradients
Loop 9: per-occurrence dB, token-reduced dZ, one dA, and one dX

It does not automatically consume LoRAMoE’s localized balance objective, Mixture-of-LoRAs domain supervision, MoLE’s candidate-output gate, O-LoRA’s orthogonality penalty, or AM-LoRA’s selector penalty. Those losses belong to different method states from Loops 4 and 7.

For a legal shared-A step, the ownership graph is

$$dB_j = \sum_{q:I_q=j}u_qz_{token(q)}^\top,$$ $$dz_t = \sum_{q:token(q)=t}B_{I_q}^\top u_q,$$ $$dA = \sum_t dz_t x_t^\top.$$

The optimizer receives one $A$ parameter identity, one $A$ gradient, and one set of moments. Replacing that state with three equal-valued independent $A_j$ tensors is illegal even when the current forward is equal: the optimizer objects and future trajectories differ.

The route derivative still holds support fixed. A full training transaction must additionally record the top-$k$ support, tie policy, gate dtype, balance objective, and whether the router’s input path itself depends on routed outputs. Loop 3 remains the diagnostic owner when that gradient fails.

0x09F Composition And Continual State Are Lane-Specific

The static lane may materialize

$$\Delta W_{static} = \sum_{j=1}^{J}\lambda_jB_jA_j$$

by exact block concatenation before later rounding. Its router is none, its route lifetime is artifact_global, and its runtime can use an ordinary fixed child weight. Retaining an online router in that release would make the artifact description internally contradictory.

The shared-A and AM-LoRA lanes are input-conditioned. Loop 6’s additivity witness has residual norm $0.326766\ldots$, so one global static matrix does not represent that dynamic witness. Factor-coordinate linear averaging is not a repair: its cross-term norm is $1.0$ in the exact fixture and changes under a gauge transform.

O-LoRA and AM-LoRA both carry ordered state, but they still do not share a runtime:

O-LoRA:
    cumulative update, no selector
    exact homogeneous block concatenation
    scale-aware blocks required when scales differ

AM-LoRA:
    task adapters plus zero branch and mutable raters
    adapter-axis softmax
    three nonzero task branches per row in the teaching state
    zero hard-zero gates, despite L1 pressure

The continual state identity is

$$I_S = H( I_B, [task_1,task_2,\ldots,task_n], [I_{adapter,1},\ldots,I_{adapter,n}], I_{selector}, I_{parents}, I_{evaluation} ).$$

For the exact fixture, the original state digest begins 87c42c59...; reordering the same task records produces 2b55c98b.... Same members plus a different order is a different continual state. A resume path must reject the reordered state before loading optimizer moments or selector parameters.

0x09G Publication, Drain, And Rollback

The local transition has three monotonically increasing plan generations:

generation 41: adapter-0-r5, adapter-1-r3, adapter-2-r8
generation 42: adapter-0-r5, adapter-1-r4, adapter-2-r8
generation 43: adapter-0-r5, adapter-1-r3, adapter-2-r8  # rollback publication

Generation 42 accepts new requests only after publication. Requests already bound to generation 41 retain their old leases until their last-use events complete. Rollback does not mutate generation 42 in place; it publishes the old artifact set as generation 43.

Let $A(g)$ be the ordered artifact-version tuple and $I_P(g)$ the publication identity. Then

$$A(43)=A(41),$$

but

$$I_P(43)\ne I_P(41)$$

because the publication generation and lifecycle history differ. Functional rollback is not temporal erasure.

The conservative cache rule is correspondingly strict:

generation-42 K/V is not reusable after rollback
generation-41 K/V is not silently relabeled generation 43
reuse requires an explicit semantic compatibility proof
old in-flight generation-41 requests may finish under their original leases

Some architectures may prove that an adapter change does not affect a specific cache dependency cone. That narrower proof can relax the rule. Shape equality or restored adapter filenames cannot.

0x09H Same-Shape Counterfactuals

The integrated oracle carries eight substitutions that remain shape-correct but violate ownership or lifecycle semantics.

Substitution	Why tensors still fit	Why the transaction rejects it
package dynamic route as static merge	every delta has the same matrix shape	input-conditioned operator fails the fixed-matrix/additivity contract
use factor-linear average as delta sum	$A$ and $B$ shapes remain legal	cross terms appear and result is gauge-sensitive
reuse shared-A schedule for unequal $A_j$	each $A_j$ is $[r,d_{in}]$	one projection no longer equals branch-local projections
call AM-LoRA softmax sparse execution	gates are finite and normalized	all three task branches remain nonzero/materialized
naively concatenate heterogeneous O-LoRA scales	block dimensions match	effective delta differs by $1.0$ in the witness
reorder continual tasks under old state ID	all parent records remain present	task order changes state identity and selector history
reuse K/V after route change	cache tensor shapes match	route provenance digest changes
infer skipped work from top-$k$ gates	support contains two IDs	no physical plan or tensor-read trace proves omission

This is the main value of the cross-contract oracle: local equations can all be right while the assembled system is wrong.

0x09I Selection Logic After Contract Closure

Need	Conservative design	More adaptive design	Added contract
known disjoint tasks	external task ID + one adapter	learned request router	router identity and route persistence
heterogeneous corpus	one larger LoRA	joint routed experts	coupled expert/router training state
reduce routed parameters	independent small experts	exactly shared A or B	shared storage and optimizer ownership
combine trained adapters	evaluated static delta composition	input-conditioned gate	dynamic artifact and runtime route state
continual retention	immutable adapter per task	orthogonal or attentional state	ordered parent chain and selector provenance
simple serving	one approved merged child	live token routing	occurrences, kernels, generations, leases, and K/V identity

The correct baseline is often not another router. Reliable external task identity can make explicit request-level selection easier to evaluate and serve. A single higher-rank adapter can avoid router and packing overhead. A routed training system can also be distilled into one approved static artifact, but M6.10 owns that model-transfer objective and its quality contract.

0x09J Executable Oracle And Mutation Closure

The checked-in cross_contract_fixture.json and cross_contract_oracle.py produce cross_contract_report.json. The report:

replays exact upstream reports:             8 / 8
legal lifecycle lanes reconciled:           4 / 4
integrated checks passing:                 16 / 16
same-shape counterfactuals rejected:        8 / 8
route occurrences / expert segments:       8 / 3
shared-A arithmetic saved in fixture:      48 FLOPs
shared-A finite-difference maximum error:   7.11e-10
publication generations:                   41 -> 42 -> 43
one-field negative fixtures rejected:      222 / 222

Every scalar in the Loop 10 fixture receives a one-field mutation. The suite separately classifies source, identity, lane, route/factor/runtime, continual, composition, publication/rollback, and claim-ceiling drift. Prior-loop numerical fixtures remain owned and diagnosed by their original tests; Loop 10 proves composition rather than replacing them with one opaque test.

The claim ceiling remains narrow. No paper training run, checkpoint, framework autograd path, GPU kernel, production scheduler, latency, throughput, quality result, or benchmark is reproduced. Loop 11 still owns broad evaluation and injected-failure closure; Loop 12 owns final publication reconciliation.

0x0A Evaluation, Failure Recovery, And Static Publication Closure

Loops 2-10 answered whether a declared routed-adapter state is internally coherent. They did not answer whether one state is better than another. That second question needs a different contract. A quality score without the candidate’s parent checkpoint, training history, route scope, active budget, evaluation split, nuisance coordinates, and runtime state is not an evaluation record. It is a number detached from the system that produced it.

Loop 11 therefore owns three publication-facing surfaces and no new mechanism:

evaluation design:       which typed states may be compared, under which estimand
failure closure:         which invariant failed, who repairs it, and what proves recovery
static evidence surface: which sources, links, figures, and assets support the article

It consumes the exact Loop 10 transaction rather than constructing a new method state. It does not execute paper training, a framework model, a GPU kernel, a production scheduler, or a benchmark. The checked numbers below are synthetic teaching records used to test the evaluation algebra and publication schema.

0x0AA Candidate Lanes Are Not Automatically Comparable

The candidate identifier must include the complete parent state:

$$I_{\mathrm{candidate}} = H\!\left( I_{\mathrm{base}}, I_{\mathrm{training}}, I_{\mathrm{factor}}, I_{\mathrm{route}}, I_{\mathrm{composition}}, I_{\mathrm{continual}}, I_{\mathrm{runtime}} \right).$$

Two candidates with the same rank and tensor shapes can still have different frozen specialists, supervision, task order, route lifetime, selected-branch materialization, or publication generation. A comparison table must not erase those coordinates.

The Loop 11 design freezes six inference candidates:

Candidate	Question it can answer	Required matching	Interpretation limit
one mixed rank-16 adapter	is routing worth its extra state?	trainable budget, targets, data, optimizer	different hypothesis class from specialists
oracle task-ID specialists	what if reliable task identity were available?	exact specialist artifacts	upper-bound control, not task-ID-free routing
uniform shared-$A$ mixture	does the learned router add value over fixed mixing?	same shared $A$, $B_j$, materialization	dense fixed control, not sparse execution
learned shared-$A$ top-2	does input-conditioned selection help?	same parent factors and route budget	dynamic route cannot become one fixed matrix
exact static fusion child	what does a fixed release artifact trade away?	same parent deltas and conversion policy	deployment alternative, not the same function
AM-LoRA dense selector	how does continual attentional composition behave?	ordered task state and selector lineage	all nonzero branches remain physical work

No single ordering of these six rows is scientifically meaningful. The oracle-task control uses information the learned route may not have. The static child removes route overhead by changing the function class. AM-LoRA consumes an ordered continual state. The one-adapter control changes capacity allocation. The valid output is a set of predeclared contrasts, not a universal adapter leaderboard.

0x0AB Freeze Unit, Workload, Nuisance, Split, And Attempt Identity

One cross-sectional evaluation cell is

$$u = \left( c,w,s,p,a \right),$$

where $c$ is a candidate identity, $w$ is a workload, $s$ is a paired nuisance tuple, $p$ is a split identity, and $a$ is an attempt identity. The nuisance tuple is itself structured:

$$s = \left( s_{\mathrm{init}}, s_{\mathrm{data}}, s_{\mathrm{route\ tie}}, s_{\mathrm{request\ order}} \right).$$

The same $s$ must be used on both sides of a paired contrast. “Four seeds” is too weak if candidate A sees initialization/data-order pairs $(1,1),(2,2),(3,3),(4,4)$ while candidate B sees a different pairing. Route tie-breaking and request order also matter because a numerically tied top-k route or a different cold-adapter arrival sequence can change physical execution without changing tensor shapes.

The design contains four fixed workloads:

domain_alpha:    target capability A
domain_beta:     target capability B
base_retention:  capability inherited from the frozen base
unseen_transfer: held-out domain not used to train the router or factors

It crosses six candidates, four workloads, and four nuisance tuples, producing $6\times4\times4=96$ synthetic quality cells. The split manifest keeps training, route-validation, candidate-selection, and held-out test identities disjoint. Missing paired cells are not replaced with a convenient new seed; the contrast remains incomplete. An infrastructure retry retains the failed attempt and may rerun only the same semantic unit. A numerical failure is an outcome, not permission to search for a better seed.

0x0AC Derive The Paired Estimand Before Looking At Scores

Let $m_{cws}$ be a metric for candidate $c$, workload $w$, and paired nuisance tuple $s$. Against a declared baseline $b$, the paired difference is

$$d_{cws} = m_{cws}-m_{bws}.$$

Pairing removes the nuisance contribution only to the extent that the same nuisance realization affects both candidates. It does not make the result causal when training histories, information sets, or budgets differ.

For $N$ complete pairs, the mean difference is

$$\bar d_c = \frac{1}{N} \sum_{w,s} d_{cws},$$

with sample variance

$$s_{d,c}^{2} = \frac{1}{N-1} \sum_{w,s} \left(d_{cws}-\bar d_c\right)^2$$

and standard error

$$\operatorname{SE}(\bar d_c) = \frac{s_{d,c}}{\sqrt N}.$$

This formula does not automatically create a valid confidence interval. Workloads may not be exchangeable, repeated examples may be dependent, and four hand-picked nuisance tuples do not define a population of all training runs. The report must name the sampling unit and dependence assumptions before attaching inferential language. Otherwise it should publish the paired values, mean, dispersion, and missingness without calling the interval a confidence statement.

The teaching matrix deliberately uses a common nuisance offset, so the paired difference is easy to audit:

Candidate	Mean synthetic quality	Paired difference from one mixed adapter	What the number proves
one mixed adapter	0.7175	0.0000	arithmetic baseline only
oracle specialists	0.7725	+0.0550	paired-summary implementation
uniform shared-$A$	0.7150	-0.0025	paired-summary implementation
learned shared-$A$ top-2	0.7575	+0.0400	paired-summary implementation
exact static child	0.7375	+0.0200	paired-summary implementation
AM-LoRA dense selector	0.7450	+0.0275	paired-summary implementation

These values are generated fixtures, not observations about LoRAMoE, HydraLoRA, O-LoRA, AM-LoRA, or any cited implementation. Their only purpose is to make incorrect grouping, unpaired subtraction, workload pooling, test reselection, and hidden missing-cell substitution executable failures.

0x0AD Quality Is A Vector, Not One Average

For candidate $c$, the minimum publication unit is a quality vector

$$q_c = \left( q_c^{\mathrm{target}}, q_c^{\mathrm{base}}, q_c^{\mathrm{unseen}}, q_c^{\mathrm{calibration}}, q_c^{\mathrm{safety}} \right).$$

Target quality can rise while base retention falls. Oracle routing can improve a known-domain score while being unusable when the task ID is absent. A static fusion child can reduce runtime state while losing input-conditioned specialization. For that reason, Loop 11 forbids “best average quality” from silently compensating a retention or safety regression.

An actual run must predeclare:

primary target estimand and direction
non-inferiority margins for base retention and safety
unseen-domain transfer metric
calibration and decoding policy
validation-only selection rule and deterministic tie-break
one held-out opening after candidate identity is frozen
release or no-release outcome; no runner-up reselection

Parameter matching also needs two ledgers. Matched total trainable parameters controls optimizer state and artifact growth. Matched active parameters per token controls routed low-rank arithmetic. A shared-$A$ top-2 candidate and an independent-factor top-2 candidate can match active rank while differing in stored parameters and hypothesis class. Both ledgers must be visible.

0x0AE Routing Diagnostics Must Join Logical And Physical State

Let $p_j$ be the fraction of eligible route mass assigned to adapter $j$. Route entropy is

$$H(p) = -\sum_{j=1}^{J}p_j\log p_j.$$

$H=0$ indicates one selected branch in the teaching oracle, while a uniform three-branch route has $H=\log 3$. Entropy alone is insufficient: a high-entropy soft gate can still materialize every branch, and balanced route counts can still assign poor experts.

For selected occurrence counts $n_j$, define

$$\mu_n = \frac{1}{J}\sum_j n_j, \qquad \operatorname{CV}_{n} = \frac{\sqrt{J^{-1}\sum_j(n_j-\mu_n)^2}}{\mu_n}.$$

The coefficient of variation exposes load skew, but it must be reported beside domain-conditioned quality. Driving $\operatorname{CV}_n$ to zero can destroy specialization.

Route stability between two selected sets $S$ and $S'$ can be summarized by Jaccard overlap:

$$J(S,S') = \frac{|S\cap S'|}{|S\cup S'|}.$$

The report must say what changed between $S$ and $S'$: adjacent token, layer, nuisance seed, prompt paraphrase, or publication generation. Those are different stability questions.

Finally, logical sparsity must reconcile with physical materialization:

$$\rho_{\mathrm{mat}} = \frac{R_{\mathrm{materialized}}}{R_{\mathrm{selected}}}.$$

For a correct selected top-k schedule, both counts describe the same admitted occurrences and $\rho_{\mathrm{mat}}=1$. If all $J$ branches run before a top-k mask, the denominator must not be falsified to $Tk$; the physical ledger is $TJ$. AM-LoRA’s softmax selector has no hard-zero execution claim in the teaching state, so its materialized branch count remains dense.

Required routing evidence is therefore:

gate probabilities and selected IDs by domain, token, layer, and nuisance tuple
entropy, top-k frequency, count CV, and task/domain confusion
selected versus evaluated versus materialized branch occurrences
route stability under named perturbations
expert output similarity and gradient utilization during training
quality under forced oracle, uniform, balanced, and sparse controls

0x0AF Runtime Evaluation Is Phase-Specific Measurement

A routed-adapter request decomposes into

$$T_{\mathrm{request}} = T_{\mathrm{queue}} +T_{\mathrm{base}} +T_{\mathrm{route}} +T_{\mathrm{pack}} +T_{\mathrm{low\ rank}} +T_{\mathrm{scatter}} +T_{\mathrm{collective}} +T_{\mathrm{cache}}.$$

This is an accounting identity, not permission to add overlapping GPU intervals. Events must be recorded on the owning streams, synchronized at declared boundaries, and reconciled with one end-to-end wall-clock interval. CPU queueing, adapter admission, cold copy, and GPU execution need different clocks.

Prefill and one-token decode must remain separate. Prefill has many rows and can amortize route and packing work; decode exposes launch, pointer, and selected-factor overhead at small row counts. The same candidate can win one phase and lose the other. Actual measurement must freeze:

hardware, driver, framework, kernel revision, clocks, and power policy
batch and sequence-length distribution, prompt/decode split, and request popularity
warmup count, measured repetitions, synchronization, and percentile estimator
resident versus cold-adapter state and copy overlap policy
route-signature distribution and prefix/KV compatibility policy
raw attempt records, failures, retries, and censoring

Cold-start cost is

$$\Delta T_{\mathrm{cold}} = T_{\mathrm{cold}}-T_{\mathrm{resident}},$$

but it is meaningful only when both terms use the same request and route state. Resident bytes, selected bytes, copied bytes, workspace bytes, and K/V bytes are separate ledgers.

The executable fixture contains twelve synthetic phase ledgers, two phases for each candidate. For example, the shared-$A$ top-2 row totals 132 synthetic prefill work units and 22 decode units, while the static child totals 112 and 12. Those numbers test phase aggregation only. They are explicitly not latency measurements, because no kernel, overlap schedule, hardware, or scheduler ran.

0x0AG Continual Evaluation Consumes The Full Task-Time Matrix

Loop 7 already derived the task-by-time matrix contract. Loop 11 consumes that exact report rather than recomputing a convenient final average. For task $i$ evaluated after learning task $t$, retain every $R_{t,i}$. Then final average accuracy, backward transfer, forward transfer, and forgetting remain different estimands:

$$\operatorname{ACC}_T = \frac{1}{T}\sum_{i=1}^{T}R_{T,i},$$ $$\operatorname{BWT}_T = \frac{1}{T-1} \sum_{i=1}^{T-1} \left(R_{T,i}-R_{i,i}\right),$$ $$\operatorname{FWT}_T = \frac{1}{T-1} \sum_{i=2}^{T} \left(R_{i-1,i}-b_i\right),$$

where $b_i$ is the declared pre-training baseline for task $i$. The Loop 7 teaching witness remains ACC $=0.770$, BWT $=-0.080$, and FWT $=+0.075$: the same final average coexists with negative backward transfer.

State growth is part of the result:

$$M_{\mathrm{continual}}(T) = M_{\mathrm{base}} +M_{\mathrm{factors}}(T) +M_{\mathrm{selector}}(T) +M_{\mathrm{optimizer}}(T) +M_{\mathrm{provenance}}(T).$$

Report artifact, CPU, GPU, optimizer, and temporary bytes separately, together with branch materialization at inference. Evaluate oracle task-ID and task-ID-free inference as distinct protocols. A rollback test must load each immutable parent state, reproduce its declared evaluation digest, and create a new publication generation rather than rewriting history.

0x0AH Failure Injection Needs A Recovery Owner

A red flag in prose is not a closed failure. Each Loop 11 record binds ten coordinates:

loop and category
invariant
controlled fault injection
detector
consequence
recovery owner
mechanism-local executable test
release gate after repair
evidence limit that remains in force

The manifest contains four failures for every mechanism loop from 2 through 10: 36 typed failures total. It reconciles nine prior invalid-fixture families and all 1,296 one-field negatives without replacing the original tests. Loop 3 still diagnoses a wrong selected-axis softmax; Loop 6 still diagnoses a gauge-sensitive factor merge; Loop 9 still diagnoses a stale lease. Loop 11 only proves that publication cannot call the problem recovered until the named owner restores the invariant and the named local test passes.

Detection and recovery remain separate predicates:

$$R_f = \mathbf 1[\text{invariant restored}] \land \mathbf 1[\text{named test passes}] \land \mathbf 1[\text{claim ceiling unchanged}].$$

A detected failure with no repair blocks release. A repair with no replayable test also blocks release. Passing one synthetic oracle cannot widen the claim to a paper training run, GPU implementation, or production scheduler.

0x0AI Complete Source, Figure, Link, And Visual Surface

The publication audit separates frozen evidence identities from routes visible to the reader:

19 Loop 1 source identities:      7 local/consumed owners + 12 primary/code records
29 exact external URLs:           every paper, official implementation, and handoff link
7 normalized local article links: all resolved to source Markdown
91 source-figure dispositions:    53 from A22 + 38 from A29
2 retained source-local figures:  A22 Figure 51 + A29 Figure 33
29 rendered visual elements:      Markdown images plus responsive picture contracts
40 desktop/mobile asset routes:   every route exists, has alt text, digest, and provenance

The 29-URL publication registry is intentionally broader than the 12-source Loop 1 core. Later loops added LoRAHub, MiLoRA, Conditional MixLoRA, SiRA, X-LoRA, GEM, Online-LoRA, TIES, DARE, MeteoRA, LoRA-Switch, PEFT, and pinned code evidence. Loop 11 does not retroactively pretend those sources were in the Loop 1 manifest; it records when and why they entered the article.

Every A22/A29 figure has a final owner and disposition. Rejection is useful evidence: a figure can be technically related yet belong to M6.3, M6.4, M6.5, M6.8, or another mechanism loop. Reusing all source images would blur article ownership instead of adding depth.

Static SVG checks verify file existence, digest, viewBox, role, title, description, and responsive-pair semantics. They do not verify browser selection, clipping, MathJax interaction, dark-theme contrast, or mobile overflow. Loop 12 owns that normalized browser evidence.

0x0AJ Executable Loop 11 Oracle And Release State

The checked-in evaluation_publication_fixture.json and evaluation_publication_oracle.py produce evaluation_publication_report.json. The exact report closes:

Loop 10 contract/report digest chain:          exact
candidate / workload / nuisance cross-product: 6 / 4 / 4
synthetic quality cells:                       96
routing profiles:                              16
synthetic prefill/decode phase ledgers:         12
typed failures:                                36
prior mechanism negatives:                  1,296
published external/local routes:              29 / 7
A22/A29 figure dispositions:                  53 / 38
responsive Loop 11 visual routes:              4
new one-field publication mutations:          30

The static release predicate is

$$R_{\mathrm{static}} = R_{\mathrm{loop10}} \land R_{\mathrm{design}} \land R_{\mathrm{failure}} \land R_{\mathrm{source}} \land R_{\mathrm{figure}} \land R_{\mathrm{visual}} \land R_{\mathrm{boundary}}.$$

Its state is awaiting_loop12. Browser rendering, live external availability, empirical quality, GPU runtime, production scheduling, deployment, publication acceptance, and rollback-artifact execution all remain false. Loop 12 must independently reconcile the final formula registry, source surface, responsive rendering, accessibility, and cross-article boundaries before the article is publishable.

0x0B Failure Lab

The table below remains the reader-facing symptom index. The executable failure_closure_manifest.json is stricter: it records 36 failures, four for each mechanism loop from 2 through 10, and binds every row to an injected fault, detector, recovery owner, mechanism-local test, release gate, and evidence limit. A symptom can appear in several rows when it violates different invariants. Recovery is accepted only by the original mechanism owner; Loop 11 does not replace 1,296 earlier one-field negatives with a single publication test.

Failure	Symptom	Root cause	Required evidence/fix
router collapse	one adapter receives almost all mass	early advantage and softmax saturation	gate, gradient, and output-similarity traces; balance or routing redesign
fake sparse routing	latency scales with all $J$ experts despite top-k	experts evaluated before masking	kernel trace proving unselected matmuls are absent
shared-factor bottleneck	more B experts stop improving quality	common A cannot represent needed input subspaces	compare independent experts at matched active/total parameters
specialist overwrite	domain adapter loses its original behavior during router stage	”composition” stage also updated specialist factors	freeze ablation and per-specialist regression tests
task-label leakage	router looks excellent only with labels unavailable at inference	training route consumes privileged metadata	task-id-free evaluation and explicit inference contract
unsafe merge	composed artifact silently changes after quantization/rounding	lifecycle operation is order-sensitive	preserve base/deltas and test conversion order
stale route state	resumed request uses new adapter version with old K/V	hotswap ignored request provenance	version pinning or full state invalidation
continual state explosion	task count increases latency and memory linearly	every old adapter and selector remains active	growth ledger, sparse execution, consolidation/distillation policy
orthogonality overclaim	low factor overlap but old-task quality falls	factor subspace is only a proxy for functional interference	full task-time matrix and function-space diagnostics
balance harms specialization	uniform routes reduce domain quality	auxiliary objective encodes wrong utilization prior	domain-conditioned utilization and quality ablation
scatter uses token ID as expert-local row	outputs are attached to the wrong tokens after grouping	route occurrence metadata was discarded during pack	preserve occurrence ID, token row, route slot, expert ID, local row, and inverse scatter
fake shared-$A$ reuse	packed shared schedule disagrees with independent reference	value-equal but independently owned $A_j$, different dropout, rank, or scale policy	require one exact shared object/version and run the unequal-$A$ counterexample
shared gradient applied once per expert	shared $A$ update depends on segment order or optimizer call count	branch-local dA_j was not reduced into one shared gradient	reduce all token/branch $dZ$ first and step one shared parameter state
selected-factor copy explosion	memory rises with prompt length, batch size, or $k$	indexed BMM materializes $R$ copies of both factors	record $Rr(d_{in}+d_{out})b_f$ and compare direct indexing or segmentation
stale route plan	shape-correct launch reads a republished slot	plan retained an old slot or pointer-table generation	generation-bound lease from plan construction through the last-use event
route/KV mismatch	resumed generation changes after hotswap or reroute	K/V signature omitted router, adapter, or resolved-route state	reject reuse or rebuild all dependent cache state
LoRA-Switch treated as a kernel swap	layer-local model changes function or cannot know later routes	pre-gated cross-layer route assumption was omitted	preserve the paper architecture or choose a generic routed execution path
dynamic route packaged as a static child	one artifact passes shape checks but disagrees across inputs	input-conditioned coefficients were erased during release	retain a live route contract or distill into a separately evaluated static student
shared-A lane reopened with independent factors	current values agree but resumed optimizer trajectories diverge	storage and optimizer ownership were inferred from tensor equality	bind one shared storage/version and one optimizer state in the model identity
continual task order changes under the old state ID	all task adapters load, but selector behavior and replay history change	an ordered parent chain was treated as an unordered adapter set	derive a new continual-state identity and reject old optimizer/selector state
AM-LoRA soft gates called sparse execution	every branch is still materialized despite small coefficients	numerical pressure was confused with a physical omission plan	report materialized branches and require an explicit sparse execution trace
rollback relabels old K/V as the new generation	shapes match after artifact restoration, but request provenance does not	functional artifact equality was confused with publication identity	publish a new generation and require an explicit cache-compatibility proof
stale upstream evidence composed with current state	local equations pass while the assembled transaction uses mixed revisions	fixture, oracle, report, or contract digests were not replayed before composition	rebuild all eight upstream reports and reject any file or semantic digest drift

0x0C Claim Status

Claim	Status	Boundary
LoRAMoE freezes the base, routes several low-rank experts, and uses a localized balancing constraint tied to expert/sample roles	externally verified	mechanism from the ACL 2024 paper; benchmark gains are source-reported
Mixture-of-LoRAs separately trains domain adapters and then learns sequence-level routing with domain supervision	externally verified	paper-specific lifecycle; not universal to all “mixture of LoRA” methods
MoLE composes trained LoRA outputs with hierarchical learned gates and a gate-balancing loss	externally verified	exact runtime cost depends on gating hierarchy and implementation
HydraLoRA shares A and routes multiple B matrices	externally verified	the claim that A captures universal commonality is an empirical interpretation, not a theorem
shared A reduces parameter count and permits one A projection per row	derived here	follows from tensor shapes and assumes compatible ranks/targets
an input-dependent adapter mixture cannot generally become one fixed merged matrix	derived here	static coefficients remain mergeable
block-concatenated LoRA factors exactly reconstruct a compatible weighted delta sum at rank at most the sum of input ranks	derived here and checked against pinned PEFT code	exact before later rounding; output rank can become expensive
PEFT factor-wise linear/TIES/DARE paths are not generally equivalent to delta-space composition	pinned-code trace plus local derivation	factor cross terms and gauge dependence are explicit; _svd paths instead operate on delta matrices before refactoring
TIES trims task vectors, elects aggregate signs, and disjointly merges aligned coordinates	externally verified	paper task-vector contract; quality gains remain source-reported
DARE’s inverse-density rescale is coordinatewise unbiased	externally verified and derived here	one stochastic sample is not the original delta and needs reproducible randomness
quantization can make merge order and rounded unmerge observable	derived teaching witness	M8 owns generic quantizer methods, calibration, and release policy
the O-LoRA paper penalizes squared Frobenius overlap between new and old low-rank factor subspaces	externally verified	orthogonality is a proxy, not guaranteed functional retention
the pinned O-LoRA trainer uses entrywise absolute overlap plus unsquared current-factor norms	pinned-code trace	this numerical objective differs from the paper equation and is revision-specific
block concatenation exactly accumulates compatible O-LoRA factors while cumulative rank grows additively	pinned-code trace plus local derivation	heterogeneous scales need scale-aware blocks or preserved branch metadata
exact mutually orthogonal full-rank task subspaces consume a finite ambient-dimension budget	derived here	a finite penalty can leave overlap before the hard dimension ceiling
AM-LoRA freezes prior task adapters and learns a sparse-pressured attentional selector	externally verified	the stronger paper ablation can continue updating prior raters; $L_1$ pressure does not guarantee sparse execution
ACC, BWT, and FWT require a task-by-time evaluation matrix and FWT baselines	externally verified and derived here	the teaching matrix is not an O-LoRA or AM-LoRA benchmark reproduction
routed adapter identity belongs in KV/prefix provenance	runtime inference	follows from M5/M6.5 state dependence; exact safe reuse can be proven for narrower architectures
a token top-$k$ route expands to $R=\sum_t	\mathcal S_t	$ route occurrences before grouped execution
stable pack/segment/scatter preserves the independent routed-adapter output	derived here and checked by exact oracle	requires complete route-occurrence metadata and scalar gate/scale semantics
exactly shared $A$ saves $2(R-T)rd_{in}$ arithmetic FLOPs relative to independent factors	derived here and checked by exact oracle	excludes routing, pack, scatter, traffic, and launch overhead; requires one compatible shared $A$ object
shared-$A$ backward reduces all branch-local $dZ$ by token before producing one $dA$ and $dX$	derived here and checked by finite differences	the discrete top-$k$ support is held fixed; Loop 3 owns router-softmax credit
MeteoRA compares adapter loops, indexed selected-factor BMM, and a direct-indexed Triton path	externally verified plus pinned-code trace	its performance numbers remain source-reported and no GPU path was executed here
LoRA-Switch uses one token pre-route across expanded layers and applies previous-negative/current-positive low-rank weight updates through SGMM	externally verified	this changes the model/runtime contract and is not a drop-in kernel for arbitrary layer-local routes
routed execution must bind adapter residency, slot/pointer generations, leases, and K/V route provenance	runtime integration derived from M6.5	exact engine APIs vary; this article validates a teaching state machine rather than a production scheduler
all eight Loop 2-9 fixtures rebuild their checked-in semantic reports before cross-contract composition	local executable evidence	proves revision consistency of the teaching evidence only; it does not reproduce the cited training runs or runtime systems
shared-A selected execution, static fusion, O-LoRA ordered state, and AM-LoRA selection are four non-interchangeable lifecycle lanes	local derivation checked by the cross-contract oracle	the lanes share a frozen-base envelope, not training, routing, composition, or release semantics
the teaching transaction ID binds model, route, factor, slot, pointer, lease, and K/V state	local executable evidence	the SHA-256 identity is a canonical teaching contract, not a prescribed production wire format
rollback can restore the generation-41 artifact tuple while still requiring a new generation-43 publication identity	local derivation checked by the publication oracle	functional restoration does not prove K/V compatibility or erase lifecycle history
222 one-field mutations and eight same-shape cross-lane substitutions are rejected	local executable evidence	the mutation corpus covers the declared fixture schema; it is not an exhaustive proof over every framework or deployment
the Loop 11 paired evaluation oracle computes 96 quality cells, 16 route profiles, and 12 phase ledgers	local synthetic teaching evidence	validates grouping, pairing, aggregation, and evidence firewalls only; no model, paper protocol, or GPU measurement ran
36 typed failures preserve four recovery records for every mechanism loop from 2 through 10	local executable evidence	closes the declared failure inventory, not the universe of possible implementation faults
29 external URLs, seven local routes, 91 source-figure dispositions, and 40 image routes form a complete static surface	local static publication evidence	live availability, browser rendering, theme contrast, and final publication acceptance remain Loop 12
the Loop 11 release state is awaiting_loop12	local release-state contract	static inventory closure is not publication acceptance

0x0D Publish Gate

[x] route scope is explicit at task, sequence, token, layer, and matrix levels
[x] soft routing is not described as sparse compute
[x] factor-sharing parameter counts and subspace restrictions are derived
[x] joint, staged, and post-hoc training histories remain separate
[x] static merging is separated from input-conditioned composition
[x] plain delta addition is not incorrectly called non-commutative
[x] exact cat, delta-SVD, and factor-coordinate composition are separated
[x] TIES and DARE paper contracts are separated from PEFT factor-wise paths
[x] quantization order and stochastic identity are preserved in provenance
[x] O-LoRA paper and pinned-code losses, shapes, gradients, and initialization remain separate
[x] cumulative checkpoint concatenation includes rank, scale, resume, and rollback contracts
[x] AM-LoRA softmax axis, selector trainability, state growth, and runtime density are explicit
[x] continual retention is evaluated with ACC, BWT, FWT, forgetting, task order, and state growth
[x] route rows are expanded into invertible route-occurrence metadata before grouped execution
[x] independent and shared-A forward schedules, FLOPs, and exact sharing preconditions are derived
[x] packed independent and shared-A backward ownership is derived and finite-difference checked
[x] workspace separates metadata, packed activations, saved intermediates, and selected-factor copies
[x] prefill and decode ledgers remain separate from measured latency claims
[x] runtime plan includes residency, route metadata, slot/pointer generations, leases, batching, and KV provenance
[x] MeteoRA indexed-BMM and direct-indexed paths remain separate from M6.5 segmented serving
[x] LoRA-Switch is presented as an architecture/runtime co-design, not a generic kernel substitution
[x] Loop 9 oracle and one-field mutation corpus are checked in and GPU/benchmark claims remain false
[x] all Loop 2-9 fixtures, oracles, reports, and contracts replay exactly before composition
[x] shared-A selected, static fusion, O-LoRA ordered, and AM-LoRA selector lanes remain typed and non-interchangeable
[x] one request transaction binds exact model, route, factor, residency, generation, lease, and K/V identity
[x] static and dynamic composition, continual task order, and optimizer ownership survive end-to-end composition
[x] rollback restores artifacts under a new publication identity and never silently relabels prior K/V
[x] eight same-shape substitutions and all 222 one-field Loop 10 mutations are rejected
[x] Loop 10 cross-contract fixture, oracle, source trace, visual provenance, and report are checked in
[x] the responsive Loop 10 contract graph passes desktop/mobile and light/dark browser validation
[x] six typed evaluation candidates remain separate across training, factor, route, composition, continual, and runtime identities
[x] four workloads and four complete nuisance tuples produce 96 paired synthetic teaching cells with no replacement seeds
[x] quality, routing, prefill/decode runtime, and continual estimands are reported separately
[x] validation-only selection freezes one candidate before a held-out split is opened
[x] route entropy, load CV, stability, selected occurrences, and physical materialization remain distinct
[x] runtime phase ledgers are labeled synthetic and are not called latency or throughput measurements
[x] the exact Loop 7 task-time matrix and ACC/BWT/FWT/forgetting report are consumed by digest
[x] four typed failures cover each mechanism loop from 2 through 10 with named recovery owner and test surface
[x] all 1,296 prior mechanism-local one-field negatives remain reconciled under their original owners
[x] all 29 external URLs, seven local routes, 53 A22 figures, and 38 A29 figures have a static publication disposition
[x] all 29 rendered visual elements and 40 asset routes have nonempty alternatives, local assets, digests, and provenance
[x] the two Loop 11 responsive visual pairs pass static SVG role, title, description, and semantic-pair checks
[x] the Loop 11 oracle rejects all 30 one-field evaluation/publication mutations
[x] the historical Loop 11 release state remains recorded as `awaiting_loop12`; static closure is not retroactively called browser, deployment, rollback, or publication evidence
[x] Loop 12 independently closes formula/source identity, normalized browser rendering, accessibility, concrete owner boundaries, local-route acceptance, and article-source rollback as `publishable_with_declared_scope`
[x] paper benchmark claims remain source-reported

0x0E Final Formula, Source, Rendering, Accessibility, Boundary, And Rollback Closure

Loop 12 is not a twelfth routed-adapter mechanism. It is a transaction over the article that already exists. The transaction asks whether the explanatory surface, executable evidence, local assets, owner boundaries, and release state all describe the same candidate. A passing mechanism test cannot answer that question by itself: a page can preserve the right tensor algebra while shipping a missing mobile figure, a stale source identity, a broken owner route, or a release state that cannot return to its parent.

The parent is the exact Loop 11 article digest. The final candidate adds this closure section and replaces only the final publication-gate state. The rollback oracle removes that exact span in memory, restores the exact Loop 11 gate row, and requires the reconstructed bytes to hash to the Loop 11 digest. It never edits the working article.

0x0EA One Candidate, Several Independent Surfaces

The final audit does not accept one aggregate “documentation score.” It reconciles these surfaces independently and then joins them by the final article digest:

Surface	Final identity	Acceptance condition
formulas	255 source-ordered display blocks	exact LaTeX text, order, section, line span, and digest
sources and claims	19 frozen source records, 12 claim records, and 29 published external URLs	local digests exact; external URL set exact; no new Loop 12 scientific claim
figures and visuals	all 53 A22 and 38 A29 dispositions; 40 asset routes yielding 29 rendered image elements	every asset local and digest-bound; every alternative nonempty; every SVG has role, title, description, and view box
executable evidence	eleven historical loop contracts, 36 typed failures, 1,296 mechanism mutations, and 30 Loop 11 publication mutations	each historical diagnostic owner remains intact; Loop 12 does not collapse them into one final test
browser surface	desktop/mobile crossed with light/dark	MathJax parity, responsive source selection, local route and image responses, headings, anchors, language, theme, and article-local overflow all pass
release and rollback	one final article digest and one Loop 11 parent digest	publication is allowed only after browser and local-route acceptance; rollback reconstructs the exact parent bytes

The source and visual registries used here are final-candidate registries, not mutations of the historical Loop 11 files. That distinction keeps the evidence chain honest: Loop 11 remains replayable against its own article digest, while Loop 12 can bind the final article without rewriting history.

0x0EB Normalized Browser And Accessibility Contract

Four profiles cover desktop light, desktop dark, mobile light, and mobile dark. Each profile loads the built local route, waits until every MathJax source has a rendered peer, eagerly resolves article images, and then inspects the complete article DOM rather than only the screenshot viewport. The audit requires:

math sources == rendered MathJax nodes
display-math sources == 255
rendered image elements == 29
responsive picture elements == 11
broken images == 0
empty image alternatives == 0
duplicate heading IDs == 0
broken internal anchors == 0
article-owned horizontal overflow == 0
article-owned console/page errors == 0
all image and local owner routes return the expected content type
mobile profiles select mobile sources; desktop profiles select desktop sources

The four screenshots are normalized top-of-article evidence; the DOM checks cover the full article. Fixed menu or theme controls are site-shell state and are reported separately if they intersect the long title. Browser acceptance does not claim every browser, assistive technology, zoom policy, live external URL, or production deployment.

0x0EC Concrete Owner Boundaries And Program Handoff

The final boundary audit resolves owners to actual local files and rejects long exact prose duplication. It preserves the following non-transferable contracts:

Relation	Owner	Contract preserved here
consumes	M6.2	generic router, selection, weighting, and balance vocabulary
consumes	M6.3	vanilla LoRA operator, gradients, checkpoint, and merge baseline
consumes	M6.4	changed-object variant axes and quantized-base boundaries
consumes	M6.5	independent-adapter residency, scheduling, K/V compatibility, and lifecycle
consumes	M6.7	factor geometry, non-identifiability, and function-space diagnostic limits
defers	M6.6	generic expert-parallel placement, all-to-all, grouped GEMM, and topology
defers	M7.8	speculative proposal, verification, branch state, and release evaluation
defers	M8.10	compression-specific deployment evaluation, packaging, release, and rollback
next program owner	M6.10	teacher-student objectives, supervision artifacts, architecture alignment, and transfer runtime

M6.9 therefore closes routed, shared-factor, composed, and continual adapter state without reopening generic MoE execution, ordinary adapter serving, speculative verification, quantized packaging, or teacher-student transfer.

0x0ED Release Verdict And Narrow Rollback

The final state is publication_closed with the verdict publishable_with_declared_scope. Browser acceptance and local-route acceptance are true. Live external availability remains unclaimed. The rollback artifact is executable only at article-source scope: it proves that the Loop 12 closure can return to the exact Loop 11 Markdown bytes. It does not roll back a model, adapter pool, optimizer, router, K/V cache, service, deployment, or infrastructure generation.

The evidence ceiling remains unchanged. No paper training run, checkpoint, framework model, GPU kernel, production scheduler, latency, throughput, quality benchmark, live-link availability, or deployment was reproduced by Loop 12. The final audit proves publication consistency for this repository snapshot, not empirical superiority of any routed-adapter method.

0xFF References

Transformer System and source-local material

• M6.2 - Expert Parallelism and DeepSeek MoE
• M6.3 - LoRA Foundations and Training Mechanics
• M6.4 - The Low-Rank Adaptation Design Space
• M6.5 - Multi-Adapter Serving Runtime
• M6.7 - Low-Rank Theory
• Exploring the Transformer Series (22) - LoRA
• Exploring the Transformer Series (29) - DeepSeek MoE

Primary papers

• LoRAHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
• Conditional Mixture of LoRA Experts for Visual Instruction Tuning
• SiRA: Sparse Mixture of Low Rank Adaptation
• X-LoRA: Mixture of Low-Rank Adapter Experts, a Flexible Framework for Large Language Models with Applications in Protein Mechanics and Molecular Design
• LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• Mixture of LoRA Experts (MoLE)
• Mixture-of-LoRAs: An Efficient Multitask Tuning Method for Large Language Models
• Orthogonal Subspace Learning for Language Model Continual Learning
• Learning Attentional Mixture of LoRAs for Language Model Continual Learning
• Gradient Episodic Memory for Continual Learning
• Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation
• TIES-Merging: Resolving Interference When Merging Models
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models
• LoRA-Switch: Boosting the Efficiency of Dynamic LLM Adapters via System-Algorithm Co-design

Pinned official implementations

• LoRAHub implementation at df73afe5f38d9ff0fd1cd43774be51c79c581cc3
• Conditional MixLoRA implementation at af5cc3f4b42eada1e41691022a500e5711ab58e1
• X-LoRA implementation at 8217992c8dcf5a1c025749a48e665c59cc0b6350
• LoRAMoE implementation at 117728885a4069b74d3fe0f2d71ca84fa5918df4
• HydraLoRA implementation at afb693bc12386b96431636a27168c9f656437638
• MoLE implementation at 8841fe0f0fc86b21c1dea7a1e7e250f0e69e26ba
• O-LoRA implementation at 07117e1fc4a5f5ad9308a815a42cee8f46502dc8
• PEFT model merging implementation at ba6a19060d6ab54a87538a6e77e3e4d5a907375b
• PEFT model-merging guide
• MeteoRA implementation at 2d192ad5a63aa9e4beb6bf78e1d71903d5b76ed1

Runtime handoff

• Punica: Multi-Tenant LoRA Serving
• S-LoRA: Serving Thousands of Concurrent LoRA Adapters

M6 now has the adapter lifecycle end to end: operator and optimization, method variants, serving, low-rank geometry, routed composition, and continual state. M6.10 applies the same discipline to teacher-student distillation and model transfer.