M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch

0x00 Overview: Why Routed Capacity Exists

The usual way to increase a neural network’s capacity is to make every layer wider or to add more layers. In a dense Transformer, that also enlarges the path executed by every token. The same expanded feed-forward matrices process punctuation, copied text, a rare technical term, and a difficult reasoning step:

larger dense FFN
    -> more stored parameters
    -> more active parameters per token
    -> more matrix-multiply work per token

For a SwiGLU FFN with $d=4096$ and $d_{ff}=14336$, the three projection matrices contain approximately:

$$3dd_{ff} = 3\cdot4096\cdot14336 = 176{,}160{,}768$$

parameters. Under a two-FLOP multiply-add convention, one token’s three projections account for roughly 352 million arithmetic operations before activation and systems overhead. Making that FFN wider increases both the stored function family and the work charged to every token.

Mixture-of-Experts starts from a different scaling question:

Can the model own many parameterized transformations while executing only a small, input-dependent subset for each token?

The question is not whether unused parameters can be deleted. It is whether they can remain available for other inputs without participating in this input’s active path.

0x00A Four Pressures Behind Conditional Capacity

A21 motivates MoE through several related but distinct pressures. Keeping them separate prevents the method from looking like a universal solution to one vague problem called “scale.”

Latent activation sparsity. Dense FFNs can produce many inactive or weak intermediate features for a particular input. ReLU makes the observation especially visible because values can become exactly zero. But a zero produced after a dense projection does not refund the projection. The implementation needs explicit block structure and pre-execution selection before sparsity can skip a parameter block.

Dense overwork. A dense network activates the same enlarged parameter surface for every token. If a token needs only a narrow subset of the model’s learned transformations, the remaining work still executes. MoE attempts to make the selected subset explicit.

Representation interference. Shared neurons and parameters can participate in unrelated features or objectives. Updates useful for one input region can interfere with another. Conditional experts provide separate parameter surfaces that may specialize under different routing exposure. They do not guarantee clean semantic modules or eliminate shared representation interference.

Finite compute and memory budgets. More parameters can improve the available function class, but training and serving remain bounded by accelerator memory, memory bandwidth, network topology, and latency targets. Conditional computation tries to grow stored capacity faster than per-token arithmetic. The full model still has to be stored, trained, checkpointed, and often distributed.

These motivations point toward explicit routed blocks. Whether the resulting layer is actually faster depends on routing quality, packing, expert kernels, load balance, communication, and workload shape.

0x00B Division Of Labor, And Where The Analogy Breaks

The common analogy is a team of specialists. A coordinator receives a request, chooses the relevant specialists, and combines their work. The analogy usefully identifies three roles:

router:
    estimates which parameter blocks should process this token

experts:
    apply different learned transformations

combine:
    returns selected expert outputs to one residual-stream row

It also breaks quickly.

1. A routed expert is not guaranteed to correspond to a human domain such as law or medicine. It may specialize by syntax, token frequency, position, modality, or an uninterpretable distributed feature.
2. One token can use more than one expert, and one expert can serve many unrelated token regions.
3. Experts do not negotiate independently. The router, expert parameters, shared attention path, and training losses are optimized together.
4. A specialist on another device is not free to consult. Dispatch, synchronization, and combine can dominate the logical expert computation.

The safe claim is that routing creates opportunities for parameter specialization. Semantic specialization is an empirical outcome, not an architectural invariant.

0x00C From A Logical Choice To An Executable Layer

The idea predates modern language models. Early mixtures combined local expert models through a learned gate; later sparse-gated systems made selection discrete enough that unselected experts did not have to run. Transformer MoE usually applies conditional computation to the FFN sublayer: keep attention dense, replace selected dense FFNs with a router and a bank of expert FFNs, and route each token to one or a few experts.

This creates three distinct views of one layer:

total expert capacity:
    every expert parameter stored by the layer

active expert path:
    only the experts selected for this token

systems path:
    score, select, permute, dispatch, execute, combine, and restore order

Only the active expert path is sparse for one token. The full expert set must still be trained, stored, loaded, and often distributed. Routing also creates tensors and policies that a dense FFN does not need:

router logits and selected IDs
gate weights and route slots
expert capacities and local row indices
token-major <-> expert-major permutations
overflow decisions and inverse-combine metadata

TopK(router(h)) is therefore a logical routing decision, not an implementation. The runtime must preserve enough metadata to recover every output row and must execute variable expert workloads efficiently.

Source-local overview from A21 - MoE. The image captures the high-level gate-and-expert mental model. The Transformer System tensor and runtime sections below make the omitted capacity, permutation, and physical placement state explicit.

0x00D What M6.1 Establishes

This article builds that foundation in three layers:

conceptual:
    capacity, sparsity, conditional computation, specialization

mathematical:
    router logits, top-k routes, gate weights, expert outputs

runtime:
    permutation, capacity buffers, overflow policy, combine metadata

The owner question is:

How does an input-dependent top-k choice become a shape-correct, capacity-aware expert execution without confusing total model capacity with active work?

M6.1 stays with the logical and single-layer contract. M6.2 studies load control and DeepSeek-style expert structure. M6.6 owns physical expert placement, collectives, kernels, and prefill/decode runtime behavior.

0x01 Prerequisites: Capacity, Sparsity, Experts, And Routing

0x01A The Lineage From Local Experts To Transformer MoE

Modern Transformer MoE is not one invention copied unchanged across papers. Each stage modified a different contract.

Adaptive mixtures of local experts addressed heterogeneous input regions. A learned gate assigned responsibility to expert networks, allowing experts to compete for and cooperate over examples instead of forcing one global function to fit every region equally. This established specialization and learned routing, but it did not yet guarantee sparse execution. A mixture can weight several experts while still evaluating all of them.

Sparsely gated MoE changed that execution contract. Noisy top-k routing allowed a model to own a large expert bank while activating only a small subset for one input. Skipping experts created conditional computation, but it also created load imbalance, expert undertraining, routing instability, and communication requirements.

GShard placed MoE inside Transformer FFNs and made group-wise top-2 routing, expert capacity, auxiliary balancing, and automatic sharding part of the system design. Capacity bounded expert buffers so distributed execution had predictable shapes. It also meant that a desired top-2 route could overflow, be padded, dropped, or handled by a policy outside the router score itself.

Switch Transformer simplified the selected path from top-2 to top-1. One route per token reduces combine work and can simplify dispatch and communication. It does not remove expert capacity, load imbalance, auxiliary control, all-to-all exchange, or the possibility that an expert becomes undertrained.

Modern decoder and dropless MoE explores several responses to the remaining physical costs. Mixtral demonstrates top-2 experts in a decoder-only model. Expert Choice reverses the selection direction to give experts bounded token capacities. Soft MoE uses differentiable expert slots. MegaBlocks-style dropless execution uses block-sparse kernels to execute imbalanced expert workloads without padding every expert to one fixed capacity. These are alternatives with different routing and runtime contracts, not successive universal replacements.

The progression can be summarized as:

Previous limitation	Intervention	New state or failure introduced
one global function for heterogeneous inputs	learned expert responsibility	gate/expert co-adaptation
dense mixture executes every expert	sparse top-k	imbalance and route discontinuity
sparse experts are difficult to place in Transformers	grouped routing, capacity, sharding	overflow and distributed exchange
top-2 adds route and combine complexity	top-1 Switch routing	less redundancy; balance still required
fixed capacity pads or drops irregular loads	dropless/ragged/block-sparse execution	kernel scheduling and locality pressure

This problem-response reading is more useful than a date list because it explains why capacity, auxiliary balancing, top-1 routing, and specialized kernels exist.

0x01B Capacity Is Not One Number

“Model size” collapses several quantities that become different in an MoE model. Keep four ledgers separate:

Ledger	Question	Dense FFN	MoE FFN
total parameters	What weights exist?	one FFN	router plus all experts
active parameters	What weights participate for one token?	the whole FFN	selected experts only
arithmetic	What matrix work executes?	one dense path per token	router plus k expert paths per token
residency and movement	What must be in memory or moved?	shared FFN weights	expert weights plus routing/dispatch state

M4.1 introduced the same discipline for parameters, FLOPs, and memory traffic. MoE makes it essential. If each expert contains $P_{expert}$ parameters, the layer has $E$ experts, and each token selects $k$ experts, the first approximation is:

$$P_{total,experts}=E P_{expert}$$ $$P_{active,experts/token}=k P_{expert}$$

The ratio $E/k$ describes the gap between stored expert capacity and the selected expert path. It does not include router work, dispatch overhead, imbalance, communication, or the non-expert parts of the Transformer.

0x01C Two Different Meanings Of Sparsity

A21 motivates MoE partly from sparse FFN activations: nonlinearities such as ReLU can leave many intermediate activations at zero. That observation is useful, but it is not yet an executable sparse architecture.

activation sparsity:
    some values inside a dense computation are zero

architectural sparsity:
    the runtime deliberately skips unselected parameter blocks

A dense FFN can produce many zeros after its activation and still execute both dense matrix multiplications. Sparse MoE makes a coarser decision: a token is dispatched to selected expert networks, while the other expert networks are not evaluated for that token. The benefit comes from making the conditional path explicit enough for the implementation to avoid work.

MoEfication makes the bridge concrete for a ReLU FFN. It first identifies inactive intermediate neurons, groups neurons that tend to co-activate into expert blocks, and learns a selector that predicts which blocks should run. The source figure moves from a dense FFN, through unused neurons, to an explicit expert partition and router:

Source: MoEfication, Figure 1. The figure illustrates one way to recover routed expert blocks from a ReLU FFN. It is not a proof that every modern MoE originates from exact activation zeros: GeLU and SiLU-family activations need not create the same hard-zero pattern, and many MoE models train experts and routers directly from scratch.

This distinction also prevents a bad causal claim. MoE is not automatically efficient merely because neural activations are sparse. It is efficient only when routing, packing, expert kernels, and communication preserve enough useful sparsity to outweigh their overhead.

0x01D MoE Versus Its Neighbors

Several methods can look like “use only part of a network,” but they select different objects at different times.

Dropout is mainly a training regularizer; its random mask is normally disabled at inference and does not provide learned input-dependent expert capacity. A dense ensemble combines separately trained models but normally executes every member. A soft mixture can learn input-dependent weights while still evaluating all expert branches. Static structured sparsity removes the same blocks for every input. Activation sparsity can appear after dense work has already executed.

Sparse MoE’s distinctive contract is narrower:

before expert execution:
    make an input-dependent decision over parameter blocks

during expert execution:
    run only selected blocks for this token

after expert execution:
    combine selected outputs and restore token order

If a system computes every expert and masks outputs afterward, it may implement mixture weighting, but it has not preserved the sparse-compute promise.

0x01E What Counts As An Expert?

In Transformer MoE, an expert is usually an FFN-shaped subnetwork:

$$E_e(x)=W_{down,e}\,\phi(W_{up,e}x)$$

or a gated MLP variant with two input projections. Experts normally share the same input and output width so their results can return to the residual stream:

expert input:  [d_model]
expert output: [d_model]

“Expert” does not guarantee a human-readable subject such as law, medicine, or code. Specialization is learned from routing and gradient exposure. An expert may specialize by syntax, token type, domain, frequency pattern, or a distributed feature that has no simple label. Some experts may remain broad; poorly balanced experts may remain undertrained.

The specialization motivation is related to representation interference. A dense FFN reuses the same parameters across heterogeneous tokens and tasks; one neuron or feature can participate in several unrelated patterns. Routed experts give the model a way to allocate different parameter subsets to different token regions, potentially reducing interference. They do not eliminate polysemantic representations by construction. Shared upstream states, shared attention, top-k routing, and finite data can still produce distributed or overlapping specialization.

“Expert” has also referred to different computational objects across the literature:

Expert form	Routed unit	Typical consequence
local expert network	sample or input region	arbitrary specialized subnetwork; early mixture formulation
Transformer FFN expert	token row	common modern language-model MoE contract
fine-grained FFN expert	token row over smaller expert slices	more combinations and routing metadata; owned by M6.2
shared expert	every token	always-active common path plus routed paths; owned by M6.2
attention/depth/modality expert	head, token, layer, or modality	changes the routing axis; owned by M6.8

Logical function and physical ownership are separate. Expert_3 can be an FFN function in the model graph while its weights live on one rank, are replicated on several ranks, or are executed by a grouped kernel with other experts. M6.1 names logical routes; M6.6 maps them to devices and kernels.

MoE also differs from a conventional dense ensemble:

dense ensemble:
    run every member, then combine every output

sparse MoE:
    choose a small subset, run only that subset, then combine selected outputs

The gating decision is therefore part of the computation graph and the runtime schedule, not just a final voting rule.

0x01F Conditional Computation And Router Vocabulary

For token state $h_t\in\mathbb{R}^{d}$, a simple router is a learned projection:

$$z_t = h_t W_r, \qquad W_r\in\mathbb{R}^{d\times E}$$

The vector $z_t\in\mathbb{R}^{E}$ contains one affinity or logit per expert. A token-choice top-k router then selects expert IDs:

$$\mathcal{E}_t=\operatorname{TopK}(z_t,k)$$

and derives combine weights for the selected routes. Terminology used throughout M6:

Term	Meaning
router or gate	function producing expert affinities
route	one token-to-expert assignment
route slot	one of the k assignments owned by a token
token-choice routing	each token chooses experts
expert-choice routing	each expert chooses a bounded set of tokens
gate weight	coefficient applied when selected expert outputs are combined
dispatch	reorder/copy token rows into expert-owned groups
combine	restore token order and reduce selected expert outputs

For example, with four experts and $k=2$:

router logits for token t:
    [1.2, -0.3, 2.0, 0.7]

selected expert IDs:
    [2, 0]

selected-only softmax weights:
    softmax([2.0, 1.2]) ~= [0.690, 0.310]

The top-k result is not the MoE output. The runtime must still preserve the mapping from token t to expert 2 and expert 0, execute both experts, multiply their outputs by the two gate weights, add them, and restore the row to token t’s original position.

0x01G The Complete Layer Lifecycle

Before opening the equations, keep the whole lifecycle visible:

1. flatten token rows                     H: [T,d]
2. compute router logits                  Z: [T,E]
3. select expert IDs and gate weights     ids, gates: [T,k]
4. apply capacity/overflow policy
5. permute or pack rows by expert
6. run variable expert batches
7. weight and combine expert outputs
8. invert the permutation                 Y: [T,d]

Steps 4 through 8 are why TopK(router(x)) is not a sufficient implementation description. They also explain the progression of the rest of this article: dense baseline, conditional capacity, router math, dispatch/combine, capacity, and finally implementation assertions.

0x01H Claim Status And Scope

M6.1 owns the single-layer contract:

dense FFN -> router -> selected experts -> dispatch -> expert compute -> combine

M6.2 will handle DeepSeek-style scaling, fine-grained experts, shared experts, device-limited routing, and load-balancing details. M6.3 will handle LoRA foundations, and M6.5 will handle multi-adapter serving.

0x02 Dense FFN Baseline

In a standard Transformer block, every token row passes through the same feed-forward network. In a simple two-matrix FFN:

x:      [tokens, d]
W_up:   [d, d_ff]
W_down: [d_ff, d]

0x02A Why A Two-Matrix FFN Has $2dd_{ff}$ Parameters

The factor of two is not a convention imported from a scaling-law table. It follows from the two stored linear maps. Using mathematical right-multiplication notation:

input rows:
    X [T,d]

up projection:
    X [T,d] @ W_up [d,d_ff]
        -> U [T,d_ff]

down projection:
    phi(U) [T,d_ff] @ W_down [d_ff,d]
        -> Y [T,d]

The first matrix owns d * d_ff scalar weights. The second owns d_ff * d. Addition is commutative, so:

P_two_matrix
    = elements(W_up) + elements(W_down)
    = d*d_ff + d_ff*d
    = 2*d*d_ff

PyTorch stores nn.Linear(in_features, out_features).weight in [out_features, in_features] order. The checkpoint therefore usually records the same two maps as:

stored W_up:   [d_ff,d]
stored W_down: [d,d_ff]

The stored axes are transposed relative to the mathematical matrices above, but their scalar counts are unchanged. A tensor trace has to name which convention it is using; otherwise a shape can look plausible while the input and output axes are reversed.

Parameter count, ignoring biases:

P_dense = d * d_ff + d_ff * d
        = 2 * d * d_ff

For a SwiGLU-style FFN with two input projections and one output projection:

P_swiglu ~= 3 * d * d_ff

0x02B Why SwiGLU Changes The Factor From Two To Three

SwiGLU needs two independent projections from the residual width into the intermediate width. One branch is activated and the other gates it elementwise:

u = x @ W1^T          # [T,d] -> [T,d_ff]
v = x @ W3^T          # [T,d] -> [T,d_ff]
h = silu(u) * v       # [T,d_ff]
y = h @ W2^T          # [T,d_ff] -> [T,d]

With PyTorch storage axes:

W1: [d_ff,d]       d_ff*d scalars
W3: [d_ff,d]       d_ff*d scalars
W2: [d,d_ff]       d*d_ff scalars
                    ----------------
one expert:         3*d*d_ff scalars

The pointwise silu and multiplication add arithmetic but no learned matrix weights. Omitting W2 undercounts the checkpoint by one full d*d_ff matrix; treating W2 as [d_ff,d] may preserve the scalar count while still reversing its semantic axes. Loop 2 tests both failures separately.

For a sequence with N_tokens = B * S, the dense FFN path is straightforward:

for every token:
    run the same FFN

That gives a clean implementation contract:

total FFN parameters = active FFN parameters

If the model owner wants more FFN capacity, the dense solution is usually to make d_ff larger or add layers. That increases both stored parameters and active compute. MoE changes this coupling.

Dense FFN FLOP ledger, ignoring activation functions and bias terms:

F_dense ~= 2 * T * (d * d_ff + d_ff * d)
        ~= 4 * T * d * d_ff

For a SwiGLU-style FFN with three projections:

F_swiglu ~= 2 * T * (3 * d * d_ff)

0x02C The FLOP Factor Comes After The Parameter Derivation

This article uses the M4 convention that one multiply-add contributes two FLOPs. For a bias-free matrix multiply, each stored weight scalar participates in one multiply-add for each input token row. Therefore:

matmul FLOPs
    = tokens
    * active weight scalars per token
    * 2 FLOPs per multiply-add

For one token at d=4096 and d_ff=14336:

FFN path	active weight scalars	matmul FLOPs per token
two-matrix FFN	2*d*d_ff = 117,440,512	234,881,024
one SwiGLU expert	3*d*d_ff = 176,160,768	352,321,536

These are matrix-multiply ledgers. They exclude activation functions, elementwise products, bias, routing, packing, communication, and kernel implementation overhead. Counting one fused multiply-add as one operation would halve the reported FLOPs without changing the tensor graph, so the convention must travel with every number.

The constants depend on how multiply-adds are counted. The important part is the contract:

dense FFN:
    total FFN parameters ~= active FFN parameters

MoE FFN:
    total expert parameters != active expert parameters per token

0x02D From Mixtral Configuration To Layer-0 Tensor Names

The Loop 1 evidence bundle pins mistralai/Mixtral-8x7B-v0.1 at revision fc7ac94680e38d7348cfa806e51218e6273104b0 and reconciles its config, safetensors index, layer-0 safetensors header, and Transformers v4.57.0 implementation. The relevant config fields are:

hidden_size             -> d       = 4096
intermediate_size       -> d_ff    = 14336
num_local_experts       -> E       = 8
num_experts_per_tok     -> k       = 2
stored weight dtype                = BF16
projection and router bias         = false

Those fields predict one router tensor and three projection tensors for each of eight experts:

The layer-0 set therefore contains 1 + 8*3 = 25 MoE tensors. This is a source-to-tensor reconciliation, not model execution: it proves what is stored and how the axes map, but it does not reproduce quality, latency, throughput, or a production dispatcher.

0x03 MoE As Conditional FFN Capacity

An MoE FFN replaces one FFN with E experts:

Expert_0, Expert_1, ..., Expert_{E-1}

Each expert is often shaped like the FFN it replaces, or a fraction of it in fine-grained designs. The router maps each token state to expert scores:

h_t:          [d]
router(h_t): [E]

Then the layer chooses only k experts:

selected_experts_t = TopK(router(h_t), k)

The MoE output is a weighted sum of selected expert outputs:

$$y_t = \sum_{e \in \mathcal{E}_t} g_{t,e} E_e(h_t)$$

where:

h_t             token hidden state
E_e             expert e
mathcal{E}_t    selected experts for token t
g_{t,e}         gate weight for selected expert e

The important ledger becomes:

total expert parameters:
    P_experts_total = E * P_expert

active expert parameters per token:
    P_experts_active/token = k * P_expert

This is why MoE model names can be confusing. A model can expose many more total parameters than it activates for one token. The Mixtral paper gives a concrete modern example: each layer has eight FFN experts, the router selects two experts per token, and the paper distinguishes accessible parameters from active parameters during inference.

The safe wording is:

MoE increases total capacity while keeping the active path sparse.

Not:

MoE makes the model small.

The full expert set usually still has to exist somewhere in memory. M6.2 will handle what happens when those experts are distributed across devices.

Numeric parameter ledger:

d = 4096
d_ff = 14336
SwiGLU expert params ~= 3 * d * d_ff
                    ~= 176,160,768 parameters

E = 8 experts
k = 2 selected experts

total expert params ~= 8 * 176,160,768
                    ~= 1.409B

active expert params/token ~= 2 * 176,160,768
                           ~= 352.3M

This excludes attention, embeddings, normalization, router weights, shared experts, and non-expert layers. It is only the expert-FFN ledger.

0x03A Derive Router, Expert, Total, And Active Counts Separately

The router projects every token from d hidden coordinates to E expert scores. Its stored weight is [E,d], so:

P_router = E*d
         = 8*4096
         = 32,768 scalars

top_k does not appear in this count. The router has to produce the expert score vector before top-k can select two entries. Replacing E by k would count only the selected outputs after pretending the selection had already happened.

For the expert bank:

P_one_expert
    = elements(w1) + elements(w2) + elements(w3)
    = 3*d*d_ff
    = 176,160,768 scalars

P_all_experts
    = E*P_one_expert
    = 1,409,286,144 scalars

P_active_experts_per_token
    = k*P_one_expert
    = 352,321,536 scalars

Adding the router produces two different layer-local views:

View	Derivation	Scalar parameters
stored router + all experts	E*d + E*3*d*d_ff	1,409,318,912
logical active path for one token	E*d + k*3*d*d_ff	352,354,304

Under the two-FLOP convention, the router plus two expert matmuls account for:

F_active/token
    = 2*(E*d) + 2*(k*3*d*d_ff)
    = 704,708,608 matmul FLOPs

The result is not k/E times the whole layer because the router remains active and because this table intentionally excludes the layer’s dense attention and normalization paths.

0x03B Active Parameters Do Not Determine Process Residency

An active-parameter count answers which weights participate in one token’s logical path. A resident-byte count answers which weights a stated process placement holds. The two questions need different inputs.

For BF16 payloads at two bytes per scalar:

single-process full-layer residency:
    (router + all 8 experts) * 2 bytes
    = 2,818,637,824 bytes
    = 2.625061 GiB

ideal EP=4 rank residency:
    (replicated router + 2 resident experts) * 2 bytes
    = 704,708,608 bytes
    = 0.656311 GiB

The ideal shard is an accounting coordinate, not a measured deployment. It assumes exactly E/EP=2 experts on the illustrated rank and a resident router. Real implementations can replicate hot experts, shard one expert further, align allocations, retain framework state, or stage remote transfers. M6.6 owns those physical placement and topology choices.

There is a useful numerical trap here: 704,708,608 is both the ideal rank’s BF16 resident bytes and the active path’s matmul FLOPs per token under these particular dimensions and conventions. The equal integer does not make the quantities interchangeable. Their units and causal inputs are different.

0x03C Routing Also Creates Schedule-Dependent Transient State

Parameters and resident weights still do not describe the temporary routing schedule. Consider an explicit teaching implementation with T=8192, E=8, k=2, BF16 packed activations, FP32 router logits and gate weights, and INT32 expert IDs:

Logical buffer	Derivation	Payload
router logits	T*E*4	262,144 B
selected expert IDs	T*k*4	65,536 B
selected gate weights	T*k*4	65,536 B
packed expert inputs	T*k*d*2	134,217,728 B
packed expert outputs	T*k*d*2	134,217,728 B
declared total	sum above	268,828,672 B = 256.375 MiB

This is a logical materialization ledger, not a universal peak-memory formula. A fused kernel can avoid, alias, tile, or stream some buffers. Fixed-capacity routing can add padding and slack. Distributed execution can add send/receive staging. Backward adds saved state and gradients. Loop 5 owns executable dispatch/combine state; M6.6 owns physical kernels and communication.

0x03D Executable Reconciliation And Failure Mutations

The Loop 2 oracle derives every value above from the fixture’s stored tensor shapes and declared dtypes, then reconciles those tensors with the pinned Loop 1 checkpoint manifest. It rejects one-field mutations for:

missing w2 projection
transposed w2 semantic axes
missing or duplicate expert IDs
config versus tensor-index disagreement
top_k substituted for E in the router count
active-path bytes substituted for full-process residency
top_k greater than the expert count
placement records that omit the resident router

Run it with:

python3 docs/numinens-evidence/m6-1-routed-capacity/parameter_residency_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop2_parameter_residency.py

The oracle does not claim that the checkpoint was fully downloaded or executed. Its proof surface is narrower and auditable: config fields, layer-0 tensor names, stored shapes, scalar counts, declared dtypes, accounting units, and deterministic negative tests.

0x04 Router Scores, Top-k, And Gate Weights

A common router is a learned linear projection:

gate = nn.Linear(d_model, num_experts, bias=False)

For input states:

x: [B, S, d_model]

flatten tokens:

h: [T, d_model]       # T = B * S

router logits:

z = gate(h)           # [T, E]

0x04A The Stored Matrix And The Runtime Equation Use Opposite Orientations

The mathematical router is often written with $W_r\in\mathbb{R}^{d\times E}$:

$$Z = H W_r, \qquad H\in\mathbb{R}^{T\times d}, \quad Z\in\mathbb{R}^{T\times E}.$$

PyTorch nn.Linear(d,E,bias=False) stores its weight as [out_features,in_features] = [E,d]. The executable equation for that stored tensor is therefore:

$$Z = H W_{r,\mathrm{stored}}^\top, \qquad W_{r,\mathrm{stored}}\in\mathbb{R}^{E\times d}.$$

For the pinned Mixtral layer-0 checkpoint, this trace is concrete:

hidden rows H                              [T, 4096]
stored gate.weight                         [8, 4096]
runtime transpose in linear semantics      [4096, 8]
router logits Z                            [T, 8]
router bias                                absent

The transpose is a semantic view, not a request to rewrite the checkpoint. Silently reading the stored tensor as [d,E] either fails for non-square dimensions or produces the wrong expert coordinate system after an accidental reshape.

The checkpoint fixes the learned router matrix. It does not, by itself, fix the complete routing algorithm. Reproduction also needs the implementation’s score transform, selection order, normalization domain, precision, tie behavior, and train/evaluation perturbation policy.

Top-k selection:

topk_scores, topk_idx = topk(z, k)   # [T,k], [T,k]

The phrase “top-k routing” does not uniquely determine the gate weights. There are at least two common constructions.

The first computes a probability distribution over all experts, selects the top-k entries, and optionally renormalizes the selected mass:

$$p_{t,e} = \frac{\exp(z_{t,e})}{\sum_{j=1}^{E}\exp(z_{t,j})}$$ $$\mathcal{E}_t = \operatorname{TopK}(p_t,k), \qquad g_{t,e} = \frac{p_{t,e}}{\sum_{j\in\mathcal{E}_t}p_{t,j}}$$

The second selects logits first and applies softmax only across the selected entries:

$$g_{t,e} = \frac{\exp(z_{t,e})} {\sum_{j \in \mathcal{E}_t} \exp(z_{t,j})}$$

for:

e in selected expert set mathcal{E}_t

These two normalized forms are algebraically equivalent when both begin from the same logits and the first explicitly renormalizes after selection. They are not equivalent to selecting top-k probabilities and leaving their sum below one. They are also not universal: DeepSeek-V3, discussed in M6.2, uses sigmoid affinities and normalizes the selected affinities rather than applying a softmax over all experts.

0x04B Why The Two Normalized Forms Are Equivalent

Let the full-expert softmax denominator for token $t$ be:

$$D_t = \sum_{j=1}^{E}\exp(z_{t,j}).$$

Renormalizing one selected probability cancels that denominator:

$$\begin{aligned} g_{t,e} &= \frac{p_{t,e}}{\sum_{j\in\mathcal{E}_t}p_{t,j}} \\ &= \frac{\exp(z_{t,e})/D_t} {\sum_{j\in\mathcal{E}_t}\exp(z_{t,j})/D_t} \\ &= \frac{\exp(z_{t,e})} {\sum_{j\in\mathcal{E}_t}\exp(z_{t,j})}. \end{aligned}$$

So softmax(all E) -> top-k -> renormalize and top-k logits -> softmax(selected k) produce the same selected weights in exact arithmetic when they select the same experts. The cancellation is the reason, not a coincidental numerical match.

The unnormalized variant is a different operator. With logits:

[ln 8, ln 4, ln 2, ln 1]

the full softmax is:

[8/15, 4/15, 2/15, 1/15]

Top-2 retains mass 12/15 = 0.8. Renormalization produces [2/3, 1/3]; leaving the selected probabilities unchanged produces weights that sum to 0.8. If both selected experts returned the same vector u, the normalized mixture returns u, while the unnormalized mixture returns 0.8*u. This is a residual-scale change, not merely another way to write top-k.

Because exponentiation and division by a positive common denominator preserve strict ordering:

$$\operatorname{TopK}(z_t,k) = \operatorname{TopK}(\operatorname{softmax}(z_t),k)$$

for distinct exact logits. The qualification matters. Quantization, low-precision rounding, injected noise, or an exact tie can change the ordering contract even though the algebra is monotone.

0x04C Precision, Ties, And Jitter Are Router State

A stable reference softmax subtracts the row maximum before exponentiation:

$$m_t = \max_e z_{t,e}, \qquad p_{t,e}= \frac{\exp(z_{t,e}-m_t)} {\sum_j\exp(z_{t,j}-m_t)}.$$

The shift cancels algebraically and keeps exponentials in a representable range. It does not decide which experts win: subtracting one scalar from every expert logit preserves their order.

The pinned readable Mixtral implementation makes the following policy visible:

1. gate(hidden_states)                         -> router_logits [T,E]
2. softmax(router_logits, dim=1, dtype=float) -> FP32 probabilities over all E
3. topk(probabilities, k, dim=-1)             -> weights and expert IDs [T,k]
4. divide by selected weight sum              -> selected weights sum to one
5. cast selected weights to hidden dtype      -> expert-combine coefficients

This trace does not establish the internal accumulation mode of every router GEMM or softmax kernel. It establishes observable tensor dtypes and operations at the pinned Python implementation boundary.

Tie behavior is separate. The pinned path calls torch.topk; PyTorch documents that tied indices are not guaranteed to be stable. A bitwise route-reproduction record therefore cannot say only top-k=2. It must either prove that no selection-boundary tie occurs or pin an implementation/backend and accept its tie behavior. Inventing “lowest expert ID wins” would change the contract.

Jitter is also outside the learned gate matrix. The reference implementation can multiply hidden states by uniform noise during training before the router projection. The pinned Mixtral config sets router_jitter_noise=0.0, so this perturbation is inactive there. For a nonzero checkpoint, route reproduction would additionally require training/evaluation mode and random-state provenance.

0x04D Executable Policy Trace

The Loop 3 fixture stores no model activations. It uses explicit logit rows to test the router policy independently of expert computation:

State	Shape	Pinned or derived contract
stored router weight	[8,4096]	checkpoint metadata, BF16, no bias
router logits	[T,8]	gate output; fixture values are FP64 teaching inputs
full probabilities	[T,8]	stable reference softmax; pinned implementation requests FP32 output
selected IDs	[T,2]	top-k along the expert axis; baseline rows contain no boundary ties
selected probabilities	[T,2]	gathered from the full distribution
selected gate weights	[T,2]	renormalized in FP32, then cast to hidden dtype by the pinned path

For the clean row whose leading logits are [ln 8, ln 4, ln 2, 0] and whose four remaining expert logits are set far below the selection boundary, the oracle proves:

selected IDs                              [0, 1]
selected mass before renormalization      0.8
full-softmax then selected renormalize    [0.666666..., 0.333333...]
selected-logit softmax                    [0.666666..., 0.333333...]
maximum policy difference                 < 1e-12
top-k boundary margin                     ln 2

The boundary margin is the difference between the $k$-th selected logit and the best unselected logit. It is not a robustness guarantee, but it exposes when a small score perturbation can change an integer route.

The fixture rejects wrong stored axes, wrong expert counts, token-axis normalization, skipped selected renormalization, changed precision or cast policy, invented stable tie semantics, checkpoint jitter drift, wrong selected IDs, and false equivalence claims. Run the replayable gate with:

python3 docs/numinens-evidence/m6-1-routed-capacity/router_policy_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop3_router_policy.py

This makes non-selected experts inactive for that token. A21 shows the common core: compute scores, select routes, run the selected experts, weight their outputs, and combine. The exact score transform and normalization policy must travel with the checkpoint and implementation.

But this selection step is not the whole MoE layer. Top-k gives a route. The runtime still needs to make that route executable.

Selected-axis invariant:

softmax axis = k selected experts for one token
not axis = all tokens routed to one expert
not axis = flattened token-expert pairs

If weights[t] does not sum to one over that token’s selected experts, the combine step is no longer the intended MoE mixture. This is easy to get wrong when the implementation stores routed pairs in expert-major order.

0x05 Dispatch And Combine

Loop 4 ended with a list of accepted routes. Loop 5 begins there. It does not decide which route survives capacity, renormalize a dropped gate, or choose a fallback. Its job is narrower and more mechanical:

accepted token-major routes
    -> assign every route one expert-local row
    -> materialize or gather expert inputs
    -> run each expert on its rows
    -> multiply every route output by its own gate
    -> reduce all routes that originated from the same token

This distinction matters because a token is not a routed row. With T tokens and k accepted routes per token, the expert computation sees R = T*k rows before any overflow, while the residual stream still has only T rows.

0x05A The Route Row Is The Reversible Unit

For each accepted route, the dispatcher must preserve at least:

Field	Meaning	Why combine needs it
route_row	identity in token-major route order	checks route conservation
token_id	source row in H[T,d]	scatter destination
selected_slot	top-1, top-2, … slot for that token	selects the matching gate
expert_id	expert operator to execute	groups rows by expert
expert_local_row	row inside that expert’s local buffer	indexes the matching expert output
gate	weight attached to this route	scales before reduction

(token_id, selected_slot) is the stable logical key. expert_local_row is assigned only after admission and expert grouping, so it is not recoverable from token_id alone. A dropped route has no executable expert-local row and must remain in the overflow ledger rather than appearing in the packed input.

Suppose the accepted route table is:

route	token	slot	expert	gate
r0	t0	0	E1	0.75
r1	t0	1	E0	0.25
r2	t1	0	E0	0.60
r3	t1	1	E1	0.40
r4	t2	0	E1	0.20
r5	t2	1	E0	0.80

Sorting by (expert_id, token_id, selected_slot) produces this deterministic teaching layout:

packed row:        0   1   2 | 3   4   5
route id:         r1  r2  r5 | r0  r3  r4
expert:           E0  E0  E0 | E1  E1  E1
token:            t0  t1  t2 | t0  t1  t2
expert-local row:  0   1   2 | 0   1   2
expert offsets:   [0, 3, 6, 6]

The repeated final offsets say that E2 has zero rows. A runtime can choose another stable or unstable within-expert order; correctness requires the metadata and output order to agree, not this particular sorting rule.

0x05B Two Equivalent Logical Layouts

A fixed-capacity implementation can encode route placement with a Boolean dispatch tensor

$$D_{t,e,c} = \begin{cases} 1, & \text{if accepted route }(t,s)\text{ occupies slot }c\text{ of expert }e,\\ 0, & \text{otherwise,} \end{cases}$$

where D has shape [T,E,C]. Each accepted route contributes exactly one 1; every unused capacity slot remains zero. Gathering the token rows into fixed expert buffers is then

$$X^{expert}_{e,c,:} = \sum_{t=1}^{T}D_{t,e,c}H_{t,:},$$

which has shape [E,C,d]. This is the einsum("sec,sm->ecm", dispatch_mask, input) contract shown in A21’s DeepSpeed-style trace.

The combine tensor has the same sparse coordinates but stores gates:

$$G_{t,e,c} = g_{t,s} \quad\text{at the accepted route's }(t,e,c)\text{ coordinate,}$$

and zero elsewhere. If U[e,c,:] is the expert output, then

$$Y_{t,:} = \sum_{e=1}^{E}\sum_{c=1}^{C} G_{t,e,c}U_{e,c,:}.$$

That is A21’s einsum("sec,ecm->sm", combine_weights, expert_output) in indexed form. Padding is storage, not a route: a padded slot must have D=0, G=0, and cannot contribute to Y.

A packed implementation stores only accepted routes. Let a_r be the token id for route row r, b_r its expert id, and \pi the permutation that orders route rows by expert. Then

$$X^{packed}_{j,:}=H_{a_{\pi(j)},:}, \qquad j=1,\ldots,R_{accepted}.$$

Expert offsets partition X_packed into ragged expert slices. The result is combined with a weighted scatter-add:

$$Y_{t,:} = \sum_{j:\,a_{\pi(j)}=t} g_{\pi(j)}U^{packed}_{j,:}.$$

Fixed [E,C,d] buffers and packed [R_accepted,d] buffers are therefore two materializations of the same route relation. They are not identical runtime costs: padding, sorting, metadata, block formation, and kernel launch choices belong to Loop 7 and M6.6.

0x05C Why “Inverse Permutation” Is Shorthand

A21 describes the high-level computation as P · Experts · P^-1. That is intuitive for k=1, but it becomes misleading if read as ordinary matrix inversion when one token is duplicated into several route rows.

Define the token-to-route incidence matrix P with shape [R,T]:

$$P_{r,t}=1\quad\text{iff route row }r\text{ originated from token }t.$$

Every route row has one 1, but each token column can have several. For the six-route fixture, P is 6 x 3, not square, and

$$P^\top P = \operatorname{diag}(2,2,2),$$

not the identity. Let Q be the square permutation that groups route rows by expert and let \Gamma=\operatorname{diag}(g) hold route gates in token-major route order. The executable forward relation is

$$X^{packed}=QPH,$$

followed by expert computation and

$$Y=P^\top\Gamma Q^\top U^{packed}.$$

Q^T restores route order, while P^T reduces duplicate route rows into token rows. It is a weighted transpose/scatter-add, not P^-1. If capacity drops one route, the corresponding row is omitted from P, Q, and U; the accepted gate sum for that token may then be below one. Fixed-drop semantics do not silently renormalize it.

The A21 source corpus includes a more implementation-facing version of the same pipeline:

Source-local visual: A21 - MoE, adapted there from the MegaBlocks systems discussion. Read “inverse permutation” as the metadata-driven restore and reduction just derived. The pictured dropped route is one fixed-capacity policy, not an invariant of MoE; dropless grouped and block-sparse alternatives appear later in this article and in M6.6.

0x05D Executable Numeric Trace

Use three token rows:

H[t0] = [1, 2]
H[t1] = [3, 4]
H[t2] = [5, 6]

and two affine teaching experts, plus an unused third expert:

E0(x) =  2*x + [1, -1]
E1(x) = -1*x + [0.5, 2]
E2(x) =  x                 # zero routed rows in this fixture

The packed route trace is:

Scatter-add groups rows (0,3), (1,4), and (2,5) by token:

Y[t0] = [ 0.750, 0.750] + [-0.375, 0.000] = [0.375, 0.750]
Y[t1] = [ 4.200, 4.200] + [-1.000,-0.800] = [3.200, 3.400]
Y[t2] = [ 8.800, 8.800] + [-0.900,-0.800] = [7.900, 8.000]

The checked-in oracle obtains exactly the same matrix three ways: direct token-wise evaluation, fixed-capacity D/G tensors, and packed gather plus scatter-add. That equality is the representation-invariance test. It also runs a fixed-drop probe: the dropped route is absent from the expert buffers, its original gate is not redistributed, and the outer Transformer residual remains outside this MoE-output calculation.

0x05E Three Implementation Shapes, One Logical Contract

The source corpus exposes three useful implementation shapes:

1. Fixed-capacity mask/einsum. A21’s DeepSpeed-style path stores dispatch_mask[T,E,C], combine_weights[T,E,C], and expert buffers [E,C,d]. It is explicit and easy to audit, but it materializes padding.
2. Hit-expert gather and index_add_. The implementation pinned to this article constructs expert_mask[E,k,T]; where returns (selected_slot, token_id) for one expert; token rows are gathered; route gates are applied; and index_add_ accumulates all selected-expert contributions into [T,d]. This readable reference does not establish a production dispatch kernel.
3. Packed expert-major routes. A dispatcher can sort accepted routes, store prefix-sum offsets, run ragged or block-sparse expert work, and scatter-add from metadata. The exact sorting, block sizing, and transport are runtime decisions.

The pinned Mixtral-shaped reference can be reduced to:

expert_mask = one_hot(selected_experts, E).permute(2, 1, 0)  # [E,k,T]
output = zeros([T, d], dtype=hidden.dtype)

for expert_id in range(E):
    selected_slot, token_id = where(expert_mask[expert_id])
    expert_input = hidden[token_id]                             # gather
    route_output = expert[expert_id](expert_input)
    route_output *= gates[token_id, selected_slot, None]
    output.index_add_(0, token_id, route_output.to(hidden.dtype))

index_add_, not indexed assignment, is the essential semantic clue: with k>1, multiple rows target the same token. An overwrite can have correct shapes and still erase all but one expert contribution.

A metadata-explicit teaching version is:

def dispatch_accepted_routes(hidden, accepted_routes, num_experts):
    # Stable ordering is a teaching choice, not a universal runtime rule.
    packed_routes = sorted(
        accepted_routes,
        key=lambda r: (r.expert_id, r.token_id, r.selected_slot),
    )
    offsets = prefix_sum(count_by_expert(packed_routes, num_experts))
    packed_hidden = stack([hidden[r.token_id] for r in packed_routes])

    metadata = []
    for packed_row, route in enumerate(packed_routes):
        metadata.append({
            "route_row": route.route_row,
            "token_id": route.token_id,
            "selected_slot": route.selected_slot,
            "expert_id": route.expert_id,
            "expert_local_row": packed_row - offsets[route.expert_id],
            "gate": route.gate,
        })
    return packed_hidden, offsets, metadata


def combine_packed(expert_output, metadata, token_count, d_model):
    output = zeros([token_count, d_model], accumulator_dtype="declared")
    seen_routes = set()
    for row, meta in enumerate(metadata):
        key = (meta["token_id"], meta["selected_slot"])
        assert key not in seen_routes
        seen_routes.add(key)
        output[meta["token_id"]] += meta["gate"] * expert_output[row]
    return output

0x05F Tails, Dtypes, And Failure Closure

The layout contract must survive edge cases:

• A zero-hit expert has an empty packed interval: adjacent offsets are equal.
• A fixed-capacity expert with fewer than C routes has zero dispatch and combine entries in its tail slots.
• A token with one dropped route receives only its accepted route contributions under fixed-drop semantics. Gate renormalization requires a separately named policy.
• Route IDs and offsets are integer metadata. Hidden rows use the declared activation dtype. The pinned readable implementation computes router probabilities in FP32, casts selected gates to the hidden dtype, and casts route outputs before index_add_; that observation does not prove the kernel’s internal accumulation precision.
• The outer residual connection is not part of Y_moe and must not be smuggled into the combine oracle.

Failure lab:

Mutation	Shape may still look valid	Correctness failure
duplicate (token_id, selected_slot)	yes	one logical route executes twice
omit an accepted route	yes	route conservation fails
pack a dropped route	yes	capacity policy is violated
use non-contiguous expert-local rows	yes	output row no longer matches its metadata
pair a route with another token’s hidden row	yes	wrong expert input
pair a route with another slot’s gate	yes	wrong mixture coefficient
write output[token_id] = route_output	yes	later routes overwrite earlier routes
put 1 in a padded dispatch slot	yes	a phantom route enters expert compute
silently renormalize gates after fixed drop	yes	changes the declared overflow policy
call P^T an ordinary P^-1 for k>1	no algebraic basis	hides duplicate-route reduction

This is a logical single-device tensor trace. It does not claim a physical sort kernel, block-sparse schedule, all-to-all protocol, topology, bandwidth, latency, or throughput. Loop 7 owns logical buffer liveness and traffic; M6.6 owns the physical dispatcher and distributed runtime.

0x06 Expert Capacity And Overflow

Routing can be imbalanced. If every token wants the same expert, that expert becomes a bottleneck and other experts sit idle. Fixed-capacity routing turns that imbalance into an explicit allocation problem. The starting quantity is not the number of tokens but the number of requested routes in one routing group. If group $g$ contains $T_g$ token rows and every token requests $k$ experts, then:

$$R_g = T_g k$$

route assignments must be placed across $E$ expert buffers. A perfectly balanced assignment would give each expert the mean route load:

$$\mu_g = \frac{R_g}{E}=\frac{T_gk}{E}.$$

The capacity factor $\mathrm{CF}$ reserves slack above that balanced mean, and the ceiling converts the real-valued budget into an integer number of rows:

$$C_g = \left\lceil \mathrm{CF}\,\mu_g \right\rceil = \left\lceil \frac{\mathrm{CF}\,T_gk}{E} \right\rceil.$$

where:

R_g    requested routes in routing group g
T_g    token rows in routing group g
k      selected experts per token before overflow handling
E      experts visible to the routing group
mu_g   balanced mean routes per expert
CF     multiplicative capacity factor
C_g    reserved integer route slots per expert in group g

The fixed-shape buffer has $E C_g$ route slots. Its slack relative to requested routes is:

$$S_g = E C_g-R_g.$$

This slack is an allocation ledger, not a guarantee that every route fits. Capacity is partitioned by expert. Four unused slots on cold experts cannot receive a fifth route for a hot expert without changing the routing or execution representation.

If expert $e$ receives $n_{g,e}$ requested routes, its accepted and overflow counts are:

$$A_{g,e}=\min(n_{g,e},C_g), \qquad O_{g,e}=n_{g,e}-A_{g,e}=\max(0,n_{g,e}-C_g).$$

Summing over experts gives the fixed-capacity drop count:

$$D_g=\sum_{e=1}^{E}O_{g,e}.$$

When $E C_g<R_g$, at least $R_g-E C_g$ routes must overflow regardless of the score distribution. The converse is false: $E C_g\ge R_g$ does not imply zero overflow because reserved slots can be stranded on experts that received too few routes.

Capacity example using the toy trace above:

T = 8
k = 2
E = 4
CF = 1.25

R_g = 8 * 2 = 16 routes
mu_g = 16 / 4 = 4 routes/expert
C_g = ceil(1.25 * 4) = 5 routes/expert
E*C_g = 20 reserved slots
S_g = 20 - 16 = 4 slack slots

No expert exceeds C_g=5. If the capacity factor is 1.0:

C_g = ceil(1.0 * 8 * 2 / 4) = 4

Then expert E0 overflows:

n_g,E0 = 5
A_g,E0 = min(5, 4) = 4 accepted routes
O_g,E0 = max(0, 5 - 4) = 1 overflow route

The route may still have a high router score. It fails because the selected expert’s buffer is full.

This is the first place where the clean MoE story becomes operational:

top-k tells the token where it wants to go
capacity tells whether that route is available

M6.2 will expand this into load-balancing objectives, routing bias, device-level balancing, and DeepSeek-style routing constraints.

0x06A Capacity Belongs To A Routing Group

The symbol $T_g$ is not necessarily the entire global training batch. Implementations can split the global token batch into routing groups and arbitrate each group independently:

global token batch: [T_global,d]
    -> G routing groups of [T_group,d]
        -> capacity and dispatch computed per group

The group identity is therefore part of the operator contract. It must state which token rows compete for the same expert slots: one sequence, one device-local microbatch, one data-parallel shard, or another explicit partition. Smaller groups can reduce dense dispatch state and localize arbitration, but they expose capacity to more sampling variance. Larger groups smooth route counts, but increase metadata and may widen the later communication scope.

A global count can look perfectly balanced while every local group drops routes. Consider two groups with T_g=4, E=4, k=1, and CF=1, so each expert has C_g=1 slot:

Group	Requested expert IDs	Per-expert counts	Local overflow
A	E0,E0,E1,E1	[2,2,0,0]	2 routes
B	E2,E2,E3,E3	[0,0,2,2]	2 routes
global view	all eight routes	[2,2,2,2]	0 routes at global C=2

The global histogram cannot reconstruct the local outcome. Group A strands two slots on E2/E3; group B strands two slots on E0/E1. Grouped execution drops four routes even though the aggregate histogram is balanced.

Capacity is also a train/evaluation policy, not an immutable architecture constant. ST-MoE, for example, studies different training and evaluation capacity factors. Increasing evaluation capacity can reduce drops without retraining the router, at the cost of larger expert buffers and more work. That does not mean changing capacity is always quality-neutral: it changes which selected routes are actually executed.

Capacity arbitration also creates batch-composition dependence. With C_g=1, suppose focal token f requests E0 with gate score 0.80. In one group, its only E0 competitor scores 0.70; in another, a replacement competitor scores 0.90:

baseline group:  f:0.80, c_low:0.70  -> f accepted
mutated group:   f:0.80, c_high:0.90 -> f overflows

The focal token and its router score did not change. Its executed path changed because the competitor set changed. This is semantic dependence on the routing group, not merely nondeterminism from a kernel. The Soft MoE paper highlights batch dependence when contrasting capacity-based sparse routing with per-example soft routing. A reproducibility record therefore needs group construction and overflow arbitration, not only the router checkpoint.

Overflow Requires An Arbitration Order

Counts determine how many routes fit, but they do not determine which routes survive. A fixed-capacity implementation needs a priority key for every (token, selected_slot, expert) route. Possible contracts include source order, descending gate score, randomized order with a recorded seed, or an implementation-specific ordering. None follows automatically from the phrase “top-k routing.”

This article’s executable teaching oracle uses:

priority = descending gate score,
           then ascending token ID,
           then ascending selected slot

The last two fields are deterministic tie-breakers for the oracle, not a claim about GShard, Switch, or a production kernel. A real reproduction must pin its actual arbitration rule. Otherwise equal scores or equal quantized priorities can make accepted route IDs underspecified.

def arbitrate_fixed_capacity(routes, capacity):
    accepted, overflow = [], []

    for expert_id, expert_routes in group_by_expert(routes):
        ordered = sorted(
            expert_routes,
            key=lambda r: (-r.gate, r.token_id, r.selected_slot),
        )
        accepted.extend(ordered[:capacity])
        overflow.extend(ordered[capacity:])

    return accepted, overflow

This is a logical policy trace. It does not prescribe sorting kernels, prefix sums, device placement, collectives, or expert GEMM launch geometry; M6.6 owns that physical dispatcher.

0x06B The Capacity Design Space: Drop, Pad, Or Execute Ragged Work

Static expert buffers make dense kernels convenient:

expert input tensor: [E,C,d]

but create several distinct semantic and execution choices:

“Use the residual” needs care. In a pre-norm Transformer block, the outer residual connection already carries the block input around the MoE sublayer. Setting one overflowed expert contribution to zero is not the same as executing a special residual expert. An explicit fallback expert, dense FFN, or learned bypass is an additional operator and must be named in the model contract.

Rerouting also changes more than one expert ID. If a token originally selected experts {E0,E1} with normalized gates [0.7,0.3] and E0 overflows, replacing it with E2 requires a declared candidate order and a declared weighting policy. Reusing 0.7 for E2, renormalizing raw scores over {E1,E2}, or retaining only E1 produce different outputs.

MegaBlocks is an example of the dropless design. It reformulates expert computation as block-sparse operations so imbalanced expert assignments can execute without dropping tokens or padding every expert to the maximum load. The important lesson is not that one kernel solves every MoE deployment. It is that “capacity overflow” partly reflects the execution representation. A runtime with only dense fixed-shape expert kernels faces a different tradeoff from a runtime with efficient ragged or block-sparse kernels.

Source: MegaBlocks, Figure 3. The figure compares equal-capacity batched expert GEMMs with a block-sparse formulation that can execute unequal token loads without padding every expert to the same size. It is an execution representation, not a claim that every dropless MoE uses MegaBlocks kernels.

The Loop 4 oracle replays the grouped-overflow and focal-token cases, then rejects policy mutations that hide group identity, change the capacity rounding rule, omit deterministic teaching tie-breakers, confuse padding with admission, treat the outer residual as an unnamed expert, reroute without gate renormalization, or call a route-dropping implementation dropless:

python3 docs/numinens-evidence/m6-1-routed-capacity/capacity_policy_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop4_capacity_policy.py

0x06C Routing Families And Execution Alternatives

Token-choice top-k is common in autoregressive MoE language models, but it is one point in a larger design space. The first requirement is to draw the design space on two axes, not one.

0x06C.1 The Selection Axis And The Execution Axis Are Orthogonal

The selection axis defines the mathematical assignment:

token choice:
    each token selects k experts

Expert Choice:
    each expert selects C tokens

Soft MoE:
    every expert slot receives a weighted mixture of token rows

The execution axis defines how the resulting work is represented:

fixed capacity + drop:
    retain only routes that fit bounded expert buffers

fixed capacity + pad:
    reserve equal expert shapes and materialize unused rows

reroute or fallback:
    replace rejected routes under another declared policy

dropless ragged/block-sparse:
    retain the requested routes and represent unequal expert row counts

These axes are related but not interchangeable. Expert Choice changes who owns selection. MegaBlocks changes how an imbalanced route set is executed. A token-choice router can be paired with fixed dropping, max-load padding, or dropless block-sparse execution without changing its top-k scores. Conversely, changing token choice to Expert Choice changes the selected relation even if both later use the same expert kernel.

The key invariant is:

Selection semantics determine which contributions exist. Execution representation determines how those contributions are materialized. One cannot repair the other silently.

0x06C.2 Token Choice Fixes Routes Per Token, Not Load Per Expert

Let $S\in\mathbb{R}^{T\times E}$ contain routing affinities and let every token select exactly $k$ distinct experts:

$$\mathcal{E}_t=\operatorname{TopK}_{e}(S_{t,e},k).$$

The requested route count is therefore fixed before admission:

$$R_{TC}=Tk.$$

The expert-side count is derived from the selected relation:

$$n_e = \sum_{t=1}^{T}\mathbf{1}[e\in\mathcal{E}_t], \qquad \sum_{e=1}^{E}n_e=Tk.$$

The first equality makes $n_e$ data-dependent. The second conserves route rows. Token choice fixes every row degree of the token-to-expert bipartite graph to $k$; it does not fix the expert column degrees.

The Loop 6 fixture uses $T=8$, $E=4$, and $k=2$. Affinities are already normalized over experts so the example does not hide a score transform:

Token	$E_0$	$E_1$	$E_2$	$E_3$	Token-choice top-2
t0	0.31	0.30	0.29	0.10	E0,E1
t1	0.31	0.29	0.10	0.30	E0,E3
t2	0.30	0.10	0.31	0.29	E2,E0
t3	0.10	0.31	0.30	0.29	E1,E2
t4	0.27	0.26	0.25	0.22	E0,E1
t5	0.26	0.22	0.27	0.25	E2,E0
t6	0.22	0.27	0.21	0.30	E3,E1
t7	0.25	0.25	0.25	0.25	E0,E1 under the teaching tie-break

The resulting histogram is:

requested routes: 8 * 2 = 16
routes/token:     [2,2,2,2,2,2,2,2]
routes/expert:    [6,5,3,2]

The equal per-token contract coexists with strongly unequal expert loads. A capacity or execution policy acts after this route set exists.

0x06C.3 Expert Choice Fixes Columns, Not Rows

Source: Mixture-of-Experts with Expert Choice Routing, Figure 1. Conventional routing fixes selected experts per token and gives experts variable loads. Expert Choice transposes the assignment problem: each expert selects a bounded token bucket, fixing expert-side load while allowing variable experts per token.

The paper first computes token-to-expert affinities:

$$S=\operatorname{softmax}_{e}(XW_g), \qquad S\in\mathbb{R}^{T\times E}.$$

It then transposes the ranking axis. If expert $e$ owns bucket capacity $C$, it selects:

$$\mathcal{T}_e = \operatorname{TopK}_{t}(S_{t,e}, C).$$

The paper parameterizes the bucket with candidate-group token count $T$, expert count $E$, and capacity factor $c$:

$$C=\frac{Tc}{E}$$

up to the implementation’s required integer convention. This gives a selected route-slot count:

$$R_{EC}=EC.$$

Define each token’s selected-expert count:

$$m_t = \sum_{e=1}^{E}\mathbf{1}[t\in\mathcal{T}_e].$$

Double-counting the selected edges gives:

$$\sum_{t=1}^{T}m_t = \sum_{e=1}^{E}|\mathcal{T}_e| = EC.$$

Therefore the mean selected experts per token is:

$$\frac{1}{T}\sum_{t=1}^{T}m_t = \frac{EC}{T} = c.$$

The last equation constrains an average, not each token. It does not imply $m_t=c$, and it does not imply complete token coverage.

Same Mean Route Budget, Different Token Contract

Apply Expert Choice to the exact affinity matrix above with $C=4$. Both methods select 16 edges, so both average two expert contributions per token:

Token Choice: R_TC = T*k = 8*2 = 16
Expert Choice: R_EC = E*C = 4*4 = 16
mean route budget in both cases = 16/8 = 2

The assignments are nevertheless different:

Expert	Expert Choice top-4 tokens
E0	t0,t1,t2,t4
E1	t3,t0,t1,t6
E2	t2,t3,t0,t5
E3	t1,t6,t2,t3

Counting experts per token gives:

m_t = [3,3,3,3,1,1,2,0]

t7 receives no expert even though the average is exactly two. Expert Choice has perfect expert-side bucket counts [4,4,4,4]; it does not have fixed token-side work or guaranteed coverage.

The combine contract must account for variable $m_t$. In the paper notation, $I\in\mathbb{N}^{E\times C}$ stores selected token IDs, $G\in\mathbb{R}^{E\times C}$ stores selected gates, and $P\in\{0,1\}^{E\times C\times T}$ is the one-hot gather relation. If $U_{e,j,:}$ is expert output for bucket row $j$, then:

$$Y_{t,:} = \sum_{e=1}^{E}\sum_{j=1}^{C} P_{e,j,t}G_{e,j}U_{e,j,:}.$$

For $m_t=0$, this sum is zero at the MoE sublayer output. The outer Transformer residual may still carry the token state around the sublayer, but that is not the same as silently inventing an expert contribution. Any fallback expert or minimum-coverage rule is a different algorithm.

Candidate-Group State Is Part Of Expert Choice

An expert cannot know its top-$C$ tokens until its candidate set is known. The ranking group is therefore semantic state:

same focal token and affinity score
    + different competing token rows
        -> different Expert Choice membership

This is not an incidental kernel order. The Expert Choice paper explicitly lists autoregressive generation as a limitation because its implementation uses past and future tokens when performing top-k selection. An online decoder has no future candidate rows. Restricting ranking to a prefix, a request-local group, or a serving microbatch changes the selection distribution and can change the fixed-bucket interpretation. Expert Choice is not a causal-decoder drop-in merely because the final expert FFNs are token-local.

0x06C.4 Soft MoE Is A Dense Assignment Operator With Fixed Expert Rows

Soft MoE removes the discrete token-to-expert assignment and constructs a fixed number of expert input slots.

Source: From Sparse to Soft Mixtures of Experts, Figure 2. The same token-slot logits are normalized over tokens to construct dispatch weights and over slots to construct combine weights. This is the paper’s per-sequence vision operator, not a causal multi-request decoder diagram.

Let $E$ experts each own $S_e$ slots. Flatten expert and slot into one slot axis of size:

$$S=ES_e.$$

For token matrix $X\in\mathbb{R}^{T\times d}$ and learned slot parameters $\Phi\in\mathbb{R}^{d\times S}$, the score matrix is:

$$L=X\Phi, \qquad L\in\mathbb{R}^{T\times S}.$$

In indexed form, before flattening:

$$L_{t,e,s}=X_t^\top\Phi_{:,e,s}.$$

The same logits form two operators by normalizing different coordinates:

$$D_{t,j} = \frac{\exp L_{t,j}}{\sum_{u=1}^{T}\exp L_{u,j}}, \qquad \sum_{t=1}^{T}D_{t,j}=1,$$ $$C_{t,j} = \frac{\exp L_{t,j}}{\sum_{q=1}^{S}\exp L_{t,q}}, \qquad \sum_{j=1}^{S}C_{t,j}=1.$$

D normalizes each slot column across tokens. C normalizes each token row across slots. Swapping these axes preserves the [T,S] shape and changes the algorithm.

The dispatch result is:

$$X_{slots}=D^\top X, \qquad X_{slots}\in\mathbb{R}^{S\times d}.$$

Because $D_{t,j}\ge0$ and every column sums to one, each slot row is a convex combination of token rows. Apply the expert that owns each slot:

$$Y_{slots,j,:}=f_{owner(j)}(X_{slots,j,:}).$$

Then combine:

$$Y=CY_{slots}, \qquad Y\in\mathbb{R}^{T\times d}.$$

Because every row of $C$ is non-negative and sums to one, each output token is a convex combination of slot outputs. The two stochastic constraints prove different properties:

column-stochastic D:
    every slot receives one normalized token mixture

row-stochastic C:
    every token receives one normalized slot mixture

Exact Soft-Slot Tensor Trace

Use three token rows, two one-slot experts, and two slot parameters:

$$X= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}, \qquad \Phi= \begin{bmatrix} 1&0\\ 0&1 \end{bmatrix}.$$

Then:

$$L=X\Phi= \begin{bmatrix} 1&0\\ 0&1\\ 1&1 \end{bmatrix}.$$

The exact oracle obtains, rounded to six decimals:

D = softmax over tokens for each slot
    [[0.422319, 0.155362],
     [0.155362, 0.422319],
     [0.422319, 0.422319]]

column sums(D) = [1,1]

C = softmax over slots for each token
    [[0.731059, 0.268941],
     [0.268941, 0.731059],
     [0.500000, 0.500000]]

row sums(C) = [1,1,1]

The slot inputs are:

$$X_{slots} = D^\top X = \begin{bmatrix} 0.844638&0.577681\\ 0.577681&0.844638 \end{bmatrix}.$$

Let expert 0 compute $f_0(x)=2x$ and expert 1 compute $f_1(x)=-x$. Then:

Y_slots =
    [[ 1.689275,  1.155362],
     [-0.577681, -0.844638]]

Y = C @ Y_slots =
    [[ 1.079597,  0.617480],
     [ 0.031997, -0.306755],
     [ 0.555797,  0.155362]]

A minimal shape-faithful implementation is:

def soft_moe_layer(x, phi, experts, slots_per_expert):
    # x:   [T,d]
    # phi: [d,E,S_e]
    logits = einsum("td,des->tes", x, phi)       # [T,E,S_e]
    dispatch = softmax(logits, dim=0)             # token axis
    combine = softmax(logits.flatten(1), dim=1)   # flattened slot axis

    slot_x = einsum("td,tes->esd", x, dispatch)  # [E,S_e,d]
    slot_y = stack([
        experts[e](slot_x[e])
        for e in range(len(experts))
    ])                                             # [E,S_e,d]
    return einsum("ts,sd->td", combine, slot_y.flatten(0, 1))

The assignment object has $TS$ scalars. Expert FFN work executes $S$ rows. The paper’s complexity is consequently governed by slot count, not by expert count alone. Increasing experts while decreasing slots per expert can preserve $S$ and expert-row compute, but it still changes parameter ownership and distributed state.

The paper also L2-normalizes token and slot vectors and introduces a trainable scale before the score product to avoid undesirable softmax behavior as model width grows. That is a concrete implementation choice, not a generic property of every soft routing method.

Request And Causal Boundaries

Soft MoE’s convex mixtures are only valid over the declared token domain:

if T spans unrelated requests:
    a slot can mix hidden states from different users

if T spans all positions of a causal sequence:
    an early token output can depend on a slot containing future positions

The source paper evaluates visual recognition and explicitly lists autoregressive decoding as future work. A causal decoder needs an incremental or causality-preserving reformulation. A causal attention mask elsewhere in the Transformer does not repair token mixing inside D^T X.

0x06C.5 Dropless Is Not Padding-Free

Dropless execution does not define a new selection owner. It starts with a requested route relation, often token-choice top-k, and promises that every requested route executes.

For token-choice expert counts $n_e$, exact ragged route rows are:

$$R=\sum_{e=1}^{E}n_e=Tk.$$

A max-load padded representation reserves:

$$R_{max}=E\max_e n_e, \qquad S_{max}=R_{max}-R.$$

With block size $B$, a block-sparse representation can round each expert independently:

$$R_{block}=B\sum_{e=1}^{E}\left\lceil\frac{n_e}{B}\right\rceil.$$ $$S_{block}=R_{block}-R.$$

This keeps all $R$ route contributions while allowing some internal block-tail slack. It is dropless, but it is not padding-free.

For the fixture’s token-choice histogram [6,5,3,2]:

The $B=2$ value is a teaching fixture, not a MegaBlocks launch recommendation. It isolates the arithmetic of per-expert block tails.

The MegaBlocks Figure 3 shown in 0x06B makes the representation change visible: equal-capacity batched expert GEMMs become equal blocks in a block-diagonal matrix, while variable expert row counts become variable runs of smaller blocks. The published implementation makes the tail explicit. It constructs sparse topology from expert assignments, uses a padded gather, alternates block-sparse SDD and DSD operations for the two expert MLP projections, and unpermutes afterward. Its kernels require token counts per expert to be a multiple of the block size; the paper pads each expert group to the nearest multiple of 128 in its implementation.

indices, weights = router(x)       # selection semantics already decided
topology = make_topology(indices)  # block-sparse representation
x = padded_gather(x, indices)      # expert groups rounded to block boundary
x = sdd(x, w1, topology)
x = activation(x)
x = dsd(x, w2)
x = padded_scatter(x, indices)
return x * weights

That trace establishes three separate claims:

dropless:
    no requested route contribution is rejected for capacity

block sparse:
    unequal expert row counts are represented as sparse block runs

not padding-free:
    each expert's final partial block may still contain zero tail rows

M6.1 owns these logical row ledgers. M6.6 owns physical performance, sparse metadata formats, block size selection, sort/pack kernels, grouped GEMMs, collective exchange, topology, and measured throughput.

0x06C.6 Loop 6 Executable Contract

The checked-in oracle derives all four contracts rather than trusting prose:

python3 docs/numinens-evidence/m6-1-routed-capacity/routing_family_oracle.py --write-report
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop6_routing_families.py

It verifies:

Token Choice:
    exactly k distinct expert IDs per token before admission
    R_TC = T*k and data-dependent expert counts

Expert Choice:
    exactly C token IDs per expert
    R_EC = E*C, variable m_t, and explicit uncovered tokens

Soft MoE:
    dispatch columns sum to one
    combine rows sum to one
    slot and output tensors match the numeric oracle

dropless block execution:
    all requested routes execute
    block-tail rows remain visible in the representation ledger

Twenty one-field mutations reject collapsed axes, an implicit Expert Choice candidate group, unsupported causal-decoder claims, cross-request soft mixing, route-dropping described as dropless, zero-padding claims, hidden block tails, wrong Soft MoE normalization axes, and invented runtime measurements.

Selection and execution cannot be collapsed into one routing_type flag. A reproducible operator record needs at least:

selection owner and ranking axis
candidate-group identity
route or slot budget
normalization domains
token coverage/fallback behavior
admission policy
materialized expert-row representation
physical runtime owner

0x06C.7 Alternative-Method Failure Lab

Failure	Preserved shape	Broken contract
treat Expert Choice as topk(dim=0) only	often	candidate group, gates, variable token degree, and zero-coverage behavior omitted
infer $m_t=c$ from mean capacity factor $c$	yes	average edge count mistaken for a per-token guarantee
silently give uncovered Expert Choice tokens a dense expert	yes	adds an unrecorded fallback operator
rank a causal token against future rows	yes	route depends on unavailable information
normalize Soft MoE dispatch over slots	yes	slots stop being normalized token mixtures
normalize Soft MoE combine over tokens	yes	token outputs stop being normalized slot mixtures
mix unrelated requests in one Soft MoE token axis	yes	hidden-state information crosses request boundaries
call MegaBlocks padding-free	yes	block-tail padding and padded gather disappear from accounting
compare block rows with fixed capacity but omit dropped routes	yes	represented rows and executed semantic routes are conflated
attach a source-reported kernel speedup to the toy $B=2$ ledger	yes	physical measurement scope is fabricated

0x06D Router Training: Discrete Routes, Continuous Gradients, And Stability

The integer route and the continuous router are coupled, but they are not the same differentiable object. A training implementation normally combines at least four control signals:

task credit:
    quality signal through executed expert outputs and selected mixture weights

allocation pressure:
    auxiliary signal derived from route frequencies and router probabilities

numerical stabilization:
    a router-logit objective such as ST-MoE's z-loss

admission telemetry:
    selected, accepted, dropped, and executed route counts

Calling all four of these “load balancing” hides their gradients and can produce a shape-correct but behaviorally different training loop. This section derives them separately. It owns a single-device mathematical and pinned-implementation contract. Cross-rank expert/device/link balance and DeepSeek’s measured-load bias controller remain in M6.2; physical gradient exchange and overlap remain in M6.6.

0x06D.1 Top-k Gives A Piecewise-Smooth Training Problem

For valid token row $t$, let the router produce full-expert logits and probabilities:

$$z_{t,e}=h_t^\top w_{r,e}$$

The probability uses all $E$ experts:

$$p_{t,e}=\frac{\exp z_{t,e}}{\sum_{j=1}^{E}\exp z_{t,j}}$$

The ordered selected IDs are:

$$I_{t,:}=\operatorname{TopK}(p_{t,:},k)$$

The pinned Mixtral task path then renormalizes only those selected probabilities. Loop 8 derived the local selected-softmax Jacobian exactly. While $I_{t,:}$ remains fixed, the task graph is smooth. When two scores exchange order at the top-k boundary, the active graph changes discontinuously.

full_p = softmax(logits.float(), dim=-1)       # [N, E]
selected_p, selected_ids = topk(full_p, k)     # [N, k], [N, k]
task_gates = selected_p / selected_p.sum(-1, keepdim=True)
task_output = combine(run_selected(selected_ids, hidden), task_gates)

Autograd differentiates the graph selected by the forward pass. It does not differentiate an infinitesimal movement of an integer expert ID. Finite differences are meaningful only when the perturbation preserves selected IDs; the Loop 9 fixture records a minimum top-k boundary margin of 0.1 and checks that every perturbation stays inside that region.

0x06D.2 Task Credit Is Sparse, But “Dropped” Needs A Precise Axis

Loop 8 gives the task-loss router gradient for selected slot $s$:

$$\frac{\partial L_{task}}{\partial z_{t,I_{t,s}}} =g_{t,s}\left(q_{t,s}-\bar q_t\right)$$

Here $q_{t,s}$ is the gate derivative from the combined expert output and $\bar q_t$ is its selected-gate expectation. Unselected logits receive zero task-path gradient under this pinned selected-renormalization relation.

That does not imply every rejected selected route has zero router gradient. If admission happens after the selected gates are normalized, a rejected route contributes no expert output and receives no expert-parameter gradient, yet its selected score remains in the denominator. Loop 8’s exact fixture has precisely that case: the rejected route has zero expert-output and packed-input gradients but a selected-logit gradient of -0.0157495397.

The task exposure contract is therefore:

Object	Task-loss exposure under fixed IDs
executed expert parameters	accepted routes for that expert
selected gate logits	selected-gate Jacobian, including denominator effects
unselected expert path	no expert computation and no task gradient for that token
unselected router logits	zero for the pinned selected-renormalized task path
route IDs	integer state, locally constant

Sparse task credit creates a feedback risk: experts selected early receive more training signal, can become better on those rows, and can then become even more attractive to the router. Auxiliary objectives do not erase that mechanism; they add a second gradient field.

0x06D.3 Shazeer’s Importance And Expected Load Are Different Losses

Shazeer et al. introduced two distinct batch-level controls. For a nonnegative vector $v\in\mathbb{R}^{E}$, define:

$$\operatorname{CV}^2(v) = \frac{E^{-1}\sum_{e=1}^{E}(v_e-\bar v)^2}{\bar v^2}$$

The first vector is importance, the sum of sparse gate mass assigned to each expert over routing group $X$:

$$\operatorname{Importance}(X)_e =\sum_{x\in X}G(x)_e$$

Its penalty is:

$$L_{importance} =w_{importance}\operatorname{CV}^2(\operatorname{Importance}(X))$$

Equal gate mass does not guarantee equal row counts. One expert can receive a few high-weight rows while another receives many low-weight rows. The paper therefore defines a second vector, expected load, using noisy top-k selection probabilities:

$$H(x)_e=x^\top w_{g,e}+\epsilon_e\operatorname{Softplus}(x^\top w_{noise,e})$$

For Gaussian noise, the probability that expert $e$ enters the top-k set can be written using the standard-normal CDF $\Phi$ and the kth competing threshold. Summing those probabilities gives a differentiable expected load:

$$\operatorname{Load}(X)_e =\sum_{x\in X}\Pr\!\left(e\in\operatorname{TopK}(H(x),k)\right)$$

The corresponding penalty is:

$$L_{load} =w_{load}\operatorname{CV}^2(\operatorname{Load}(X))$$

These losses answer different questions:

importance CV^2:
    is sparse gate mass balanced?

expected-load CV^2:
    under the declared noisy gate, is expected expert row count balanced?

observed post-admission count:
    what actually executed after capacity, dropping, and rerouting?

The original expected-load construction depends on trainable noise and a probabilistic top-k threshold. Replacing it with an observed hard histogram is not an algebraic simplification; it changes the gradient estimator.

0x06D.4 GShard And Switch Replace Expected Load With A Hard-Frequency Proxy

GShard observes that the hard count produced by top-2 is not differentiable and multiplies a route-count fraction by a differentiable mean gate. Switch states the top-1 form particularly clearly.

Let $N$ be the number of valid router rows in the declared aggregation group. For top-1 routing, define the hard dispatch frequency:

$$f_e=\frac{1}{N}\sum_{t=1}^{N} \mathbf{1}[I_{t,1}=e]$$

Define the mean full-softmax probability:

$$P_e=\frac{1}{N}\sum_{t=1}^{N}p_{t,e}$$

The raw Switch proxy and its weighted contribution are:

$$\ell_{Switch}=E\sum_{e=1}^{E}f_eP_e$$

Then the objective adds $\alpha\ell_{Switch}$ to the task loss. The factor $E$ keeps the raw uniform reference at one: when $f_e=P_e=1/E$, $E\sum_e f_eP_e=1$.

The local derivative treats $f$ as stop-gradient because it came from hard IDs. For one valid row and expert $j$:

$$\frac{\partial\ell_{Switch}}{\partial z_{t,j}} =\frac{E}{N}p_{t,j} \left(f_j-\sum_{e=1}^{E}f_ep_{t,e}\right)$$

The derivation is the softmax Jacobian applied to a linear cost vector $f$:

$$\frac{\partial p_{t,e}}{\partial z_{t,j}} =p_{t,e}(\mathbf{1}[e=j]-p_{t,j})$$

Substituting that Jacobian into $E\sum_e f_eP_e$ yields the centered expression above. Gradient descent lowers probability on experts whose hard frequency is above the row’s probability-weighted average and raises it on experts below that average. The hard frequency changes only after a route boundary is crossed.

Two cautions follow:

1. The auxiliary loss uses pre-admission selected IDs unless an implementation explicitly says otherwise. A later drop histogram is a different statistic.
2. The objective encourages a frequency/probability relation; it does not directly minimize device imbalance, communication bytes, straggler time, or post-capacity dropping.

0x06D.5 The Pinned Mixtral Function Is A Top-2 Extension, Not Switch Equation 7 Verbatim

The pinned Transformers implementation receives one router-logit tensor per layer, concatenates them over the token-row axis, computes a full $E$-way softmax, takes top-k, and one-hot encodes the result:

Z = cat(router_logits_by_layer, dim=0)  # [L*B*S, E]
p = softmax(Z, dim=-1)                  # [L*B*S, E]
I = topk(p, k, dim=-1).indices          # [L*B*S, k]
M = one_hot(I, E)                       # [L*B*S, k, E]

Padding is expanded once over the slot/expert mask and once over the probability tensor. With $N$ valid rows across the concatenated layers, the implementation computes one frequency vector per slot:

$$f_{s,e}=\frac{1}{N}\sum_{t=1}^{N}\mathbf{1}[I_{t,s}=e]$$

It then sums the slot contributions implicitly. Define:

$$F_e=\sum_{s=1}^{k}f_{s,e}$$

The exact raw implementation objective is:

$$\ell_{HF\text{-}Mixtral} =E\sum_{e=1}^{E}F_eP_e$$

Each slot frequency sums to one, so $\sum_eF_e=k$. Under uniform top-k routing and uniform mean probability, the raw reference is therefore $k$, not one:

$$E\sum_{e=1}^{E}\frac{k}{E}\frac{1}{E}=k$$

This constant baseline does not affect gradients, but it matters when logs or coefficients from top-1 and top-k implementations are compared. The checkpoint configuration records router_aux_loss_coef=0.001, num_experts_per_tok=2, and output_router_logits=False by default. In this implementation, the coefficient alone does not activate the auxiliary path: router logits must be requested, and the auxiliary term is added to the task loss only when labels are present.

The exact Loop 9 fixture concatenates two layers, masks one padded position per layer, and leaves six valid rows:

Quantity	Expert 0	Expert 1	Expert 2	Sum
top-1 count	2	3	1	6
slot-0 frequency $f_{0,e}$	0.333333	0.5	0.166667	1
slot-1 frequency $f_{1,e}$	0.166667	0.166667	0.666667	1
aggregate $F_e$	0.5	0.666667	0.833333	2
mean probability $P_e$	0.314129	0.355880	0.329990	1

The raw Switch top-1 proxy is 1.0129450273. The raw pinned top-2 extension is 2.0079305630; after the checkpoint coefficient it contributes 0.0020079306. These numbers are exact fixture outputs, not checkpoint-training measurements.

0x06D.6 Auxiliary Gradients Reach A Different Set Of Logits Than Task Gradients

The top-2 implementation has the same local derivative as the Switch proxy after replacing $f$ by aggregate route frequency $F$:

$$\frac{\partial\ell_{HF\text{-}Mixtral}}{\partial z_{t,j}} =\frac{E}{N}p_{t,j} \left(F_j-\sum_{e=1}^{E}F_ep_{t,e}\right)$$

Because $p_{t,j}>0$ for every finite full-softmax logit, an unselected expert can receive an auxiliary router gradient even though it receives no task-path gradient for that token. The objective does not send that gradient into the unexecuted expert MLP; it updates the router path.

The distinction is easiest to audit as four vertical contracts:

Task loss. It consumes hard selected IDs and differentiable selected-$k$ gates. Under the pinned task relation, router-logit exposure is limited to the selected coordinates, while expert-parameter exposure is limited to accepted, executed routes.

Switch or pinned Mixtral balance proxy. It consumes hard IDs only through stop-gradient frequencies, but its differentiable domain is the full $E$-expert softmax. Potentially every valid router logit receives gradient; expert parameters receive none directly from this auxiliary scalar.

Router z-loss. It consumes no hard ID and differentiates the full log-partition. Every valid router logit receives gradient, while expert parameters again receive none directly.

Observed load or drop metric. It consumes hard route/admission state but has no differentiable probability domain unless a declared surrogate is attached. By itself it exposes neither router logits nor expert parameters to gradient.

For both balance proxies, each per-row analytic gradient sums to zero. Adding one constant to all logits leaves softmax probabilities unchanged, so these objectives cannot constrain that common-mode direction.

0x06D.7 Router Z-Loss Controls The Softmax-Invariant Common Mode

ST-MoE defines the per-row log-partition:

$$\zeta_t=\log\sum_{e=1}^{E}\exp z_{t,e}$$

Its router z-loss is:

$$L_z=\frac{1}{N}\sum_{t=1}^{N}\zeta_t^2$$

The derivative follows from $\partial\zeta_t/\partial z_{t,e}=p_{t,e}$:

$$\frac{\partial L_z}{\partial z_{t,e}} =\frac{2}{N}\zeta_t p_{t,e}$$

This gradient has a different geometry from task and balance gradients. Its row sum is $2\zeta_t/N$, not zero. If a constant $c$ is added to every expert logit, softmax and top-k order are unchanged but $\zeta_t$ becomes $\zeta_t+c$. The z-loss therefore anchors a direction that probability-only objectives cannot see.

That also sharpens the usual shorthand “z-loss makes logits small.” The mathematical target is a log-partition near zero, not independent elementwise clipping. Clipping occurs after large or poorly represented values have already been produced and introduces its own discontinuity; ST-MoE instead trains the router away from numerically extreme log-partitions. The paper reports the coefficient as a swept hyperparameter and uses 0.001 in its selected configuration. That source-reported choice is not a universal default.

In the Loop 9 fixture, raw $L_z$ is 2.0996682724. With a teaching-only coefficient 0.0005, its weighted value is 0.0010498341. The pinned Transformers Mixtral implementation does not implement router z-loss; the fixture computes it only to verify the formula and gradient.

0x06D.8 Selective Precision And Noise Are Separate Policies

Router exponentiation is a selective-precision boundary. ST-MoE computes router probabilities in float32 and explicitly keeps exponentiated tensors in float32 because large router values and low-precision roundoff can destabilize routing weights. The pinned Mixtral forward similarly asks softmax(..., dtype=torch.float) before top-k, then casts selected task gates back to hidden-state dtype.

Noise is a different policy family:

Shazeer noisy top-k: learned-scale Gaussian noise perturbs router logits as part of the training contract. It enables the paper’s differentiable expected-load estimator and supplies exploration.

Switch pseudocode: router jitter is a training-only exploration policy. Its exact distribution and runtime placement must be pinned before treating it as an implementation contract.

Pinned Transformers Mixtral: multiplicative uniform noise perturbs hidden rows before router projection when training and jitter_noise > 0. It can change both logits and selected routes.

Pinned checkpoint: router_jitter_noise=0.0, so stochastic router perturbation is inactive in the reconciled policy.

Changing noise type, location, schedule, random seed, or train/eval mode changes the route relation. Replaying routes during backward therefore requires either retained IDs/gates or enough deterministic RNG and kernel state to reproduce the exact forward choice.

0x06D.9 Training Integration Must Declare Which Objective Is Actually Live

A generic objective may be written:

$$L_{total} =L_{task}+\lambda_{aux}\ell_{aux}+\lambda_zL_z$$

But a formula in a design document does not prove that both branches execute. The pinned Mixtral integration is narrower:

output_router_logits = request_or_config_value
outputs = model(..., output_router_logits=output_router_logits)

loss = task_loss(logits, labels) if labels is not None else None
aux_loss = None

if output_router_logits:
    aux_loss = load_balancing_loss_func(
        outputs.router_logits, E, k, attention_mask
    )
    if labels is not None:
        loss = loss + router_aux_loss_coef * aux_loss

There is no z-loss branch in that pinned file. There is also no capacity/drop term in the auxiliary function. The function concatenates router rows across layers before taking means, so it controls the aggregate distribution; it does not guarantee that every individual layer is balanced. A per-layer loss would be a different objective even if the final scalar shape matched.

Padding is part of the objective contract. Without the attention mask, padded rows alter both $P_e$ and hard slot frequencies. With the mask, each valid token is repeated over the $k$ slot axis for the hard-frequency denominator, while the probability denominator has no slot axis. Mixing those denominators is a common top-k implementation error.

0x06D.10 Exposure Telemetry Is Not Automatically A Loss

Training should log enough independent quantities to diagnose whether quality, routing, admission, or runtime is failing:

$$n_e^{selected}=\sum_{t,s}\mathbf{1}[I_{t,s}=e]$$

After admission, executed exposure is:

$$n_e^{executed}=\sum_{t,s}A_{t,s}\mathbf{1}[I_{t,s}=e]$$

The selected gate mass is:

$$m_e^{gate}=\sum_{t,s:I_{t,s}=e}g_{t,s}$$

These should be kept beside, not collapsed into, gradient and runtime measurements:

per expert and per layer:
    selected routes, accepted routes, dropped routes
    selected gate mass and full-softmax probability mass
    task router-gradient norm and auxiliary router-gradient norm
    expert parameter-gradient norm and zero-hit steps

per routing group:
    CV^2, max/mean count, overflow rate, top-k margin distribution

physical owner in M6.6:
    packed rows, padded rows, bytes, kernel time, rank send/receive counts

The Loop 9 fixture has selected route counts [3, 4, 5], route-count $\operatorname{CV}^2=0.0416667$, and full-softmax gate masses [1.884775, 2.135282, 1.979942]. Those vectors are related but not interchangeable.

0x06D.11 Exact Oracle, Gradient Reconciliation, And Failure Closure

The evidence bundle computes all three objective gradients analytically and by central finite differences:

Objective	Scalar coordinates checked	Maximum absolute error	IDs stable?
Switch top-1 raw proxy	18	$1.71\times10^{-10}$	yes
pinned top-2 raw proxy	18	$2.38\times10^{-10}$	yes
router z-loss	18	$2.22\times10^{-10}$	yes

Two padded rows are excluded and receive zero objective gradient. The report also proves that full-softmax rows sum to one, each slot-frequency vector sums to one, aggregate top-k frequency sums to $k$, and each balance-gradient row sums to zero.

Every failure below can retain valid scalar or tensor shapes:

Selected-only softmax for $P_e$. Auxiliary exposure to unselected logits disappears.

Hard route counts differentiated as if smooth. The implementation invents a gradient through integer top-k IDs.

Padded rows included. Frequency and probability denominators describe different data.

The $[N,k,E]$ one-hot mask collapsed to “expert selected at least once.” Top-k slot multiplicity and the raw uniform reference change.

Top-2 raw uniform loss reported as one. Top-1 scaling is copied without the extra slot mass.

Per-layer means computed before averaging losses. The statistic differs from the pinned concatenated-row objective.

Z-loss called a balancing loss. Common-mode stability control is confused with allocation pressure.

Selected counts treated as executed counts. Capacity and dropping disappear from telemetry.

router_aux_loss_coef assumed to activate the branch. The pinned output_router_logits control path is omitted.

Pinned Mixtral claimed to use z-loss. The source-to-implementation trace is false.

Balanced experts equated with balanced devices or links. The claim crosses into M6.2 and M6.6 without placement evidence.

0x06D.12 What Training Can And Cannot Guarantee

Auxiliary pressure can improve exposure and hardware utilization, but too much pressure can interfere with task-driven specialization. ST-MoE reports that its chosen balance coefficient was large enough to encourage balance and small enough not to overwhelm cross-entropy; it also reports negative results for some added router entropy objectives. Those are source-scoped experiments, not proofs that one coefficient or regularizer transfers across model scale, data, routing group, or hardware.

Likewise, “experts become semantic specialists” is an empirical observation, not an architectural theorem. Routing creates conditional parameter paths. The resulting specialization depends on data order, router initialization, task and auxiliary gradients, admission policy, optimizer, noise, and training duration. M6.1 can derive the credit-assignment graph; only training evidence can establish what an expert learned.

0x06E Logical Traffic, Buffer Liveness, And The M6.6 Runtime Boundary

A route relation does not execute itself. Even on one device, an implementation may create router scores, selected IDs, selected gates, expert counts, prefix offsets, route maps, packed expert rows, expert outputs, and an output accumulator. Those tensors are easy to collapse into one phrase such as “dispatch overhead.” That phrase is too weak for either correctness or cost analysis.

Loop 7 separates three ledgers:

live payload:
    which declared tensors coexist at one schedule point?

reference materialization traffic:
    which logical tensor slices does a named teaching schedule read and write?

physical traffic:
    which cache lines, HBM transactions, workspace bytes, network packets,
    collectives, and kernel operands did one implementation actually move?

The first two belong here. The third belongs to M6.6 - MoE Runtime Systems and ultimately to profiler evidence. A sum of tensor payloads is not a memory peak; a memory peak is not traffic; logical traffic is not measured HBM traffic; and the payload of all routed rows is not automatically network traffic.

0x06E.1 One Declared Reference Schedule

Use a packed, no-drop token-choice route with:

T = 8,192 token rows
E = 8 logical experts
k = 2 selected experts per token
d = 4,096 hidden elements
R = T*k = 16,384 accepted route rows

BF16 hidden and expert rows: 2 bytes/scalar
FP32 router logits and selected gates: 4 bytes/scalar
INT32 IDs, counts, and offsets: 4 bytes/scalar

This is deliberately a logical single-device coordinate. It contains no rank, link, collective, destination device, replica, packet, kernel tile, or timing field. The reference stages are:

0 input
1 score
2 select
3 count + prefix
4 pack
5 expert forward
6 combine
7 return

A tensor payload follows directly from shape and element width:

$$B(U)=\left(\prod_{a\in\operatorname{shape}(U)}a\right)b_U$$

The declared buffers are:

Buffer	Shape	Bytes	Payload
input hidden rows $H$	$[T,d]$	2	64 MiB
router logits $Z$	$[T,E]$	4	0.25 MiB
selected expert IDs $I$	$[T,k]$	4	0.0625 MiB
selected gates $G$	$[T,k]$	4	0.0625 MiB
expert counts	$[E]$	4	32 B
expert offsets	$[E+1]$	4	36 B
route-to-token map	$[R]$	4	0.0625 MiB
route-to-choice map	$[R]$	4	0.0625 MiB
packed expert inputs $X^{packed}$	$[R,d]$	2	128 MiB
packed expert outputs $U^{packed}$	$[R,d]$	2	128 MiB
MoE output $Y$	$[T,d]$	2	64 MiB

Adding all rows in that table gives about 384.5001 MiB, but that number is not a peak. It includes external input and output tensors and tensors whose lifetimes do not overlap. The internal declared payload is about 256.5001 MiB, but that is also not a peak. Liveness decides which bytes coexist.

The two route maps are not decorative. route_to_token[j] identifies the output row reduced by packed route $j$; route_to_choice[j] recovers the corresponding gate coordinate in $G[t,s]$. An implementation may encode both in one packed integer, derive one from a permutation, or retain a richer route record. It may not discard their information and still claim a reversible weighted combine.

0x06E.2 Reference Traffic Is An Operation Ledger

For one operation $o$, define its reference logical traffic as the payload of the tensor slices that the declared materialization reads and writes:

$$Q_o^{ref} = \sum_{U\in\mathcal{R}_o}B(U_{read}) + \sum_{V\in\mathcal{W}_o}B(V_{write})$$

This is not a hardware lower bound. A fused operation can retain values in registers or shared memory; a naive implementation can reread them; a cache can serve a load without HBM; an atomic reduction can create more physical read-modify-write traffic than one logical output write. The reference ledger simply makes one schedule replayable.

Let $b_a,b_o,b_s,b_g,b_i,b_c,b_p$ be the element widths for input activations, output activations, router scores, gates, indices, counts, and prefix offsets. For packed no-drop routing, $R=Tk$ and represented expert rows $A=R$.

Score. Read the input rows and write all expert logits:

$$Q_{score}^{ref}=T d b_a+T E b_s$$

Select. Read the logits and write selected IDs and gates:

$$Q_{select}^{ref}=T E b_s+T k(b_i+b_g)$$

Count and prefix. Read one selected ID per route, write and read the expert histogram once logically, then write $E+1$ offsets:

$$Q_{count+prefix}^{ref} =Tk b_i+2E b_c+(E+1)b_p$$

The 2E b_c term means one logical histogram write and one prefix-scan read. It does not claim that a physical parallel histogram uses one transaction per counter.

Pack. Read one hidden row per accepted route, read one selected ID per route, write two route maps, and write $A$ represented expert rows:

$$Q_{pack}^{ref} =R d b_a+R b_i+2R b_i+A d b_a$$

The first term is $Rdb_a$, not $Tdb_a$, because this explicit gather reads a token row once for each route. A fused or cached implementation can change physical traffic without changing the logical duplication $R=Tk$.

Expert-output handoff. Excluding expert weights and the expert MLP’s internal activations, the expert stage writes:

$$Q_{expert\ handoff}^{ref}=A d b_o$$

Combine. Read executed expert outputs, gates, and both route maps, then write one logical output row per token:

$$Q_{combine}^{ref} =R d b_o+R(b_g+2b_i)+T d b_o$$

For the declared fixture, the exact report is:

Stage	Reference traffic
score	64.25 MiB
select	0.375 MiB
count + prefix	0.062595 MiB
pack	256.1875 MiB
expert-output handoff	128 MiB
combine	192.1875 MiB
declared total	641.062595 MiB

That total is useful only with its namespace:

reference logical activation-and-metadata reads/writes
for one explicit packed schedule
excluding expert weight and internal MLP traffic
excluding cache, fusion, allocation, transport, and network effects

Calling it “bytes transferred by MoE” would be wrong.

0x06E.3 Liveness, Not Addition, Determines The Reference Peak

For buffer $U$, let $\beta_U$ be its birth stage and $\lambda_U$ its last-use stage. Its live indicator at stage $s$ is:

$$\ell_U(s)= \mathbf{1}[\beta_U\le s\le\lambda_U]$$

The reference live payload and peak are:

$$M_{live}(s)=\sum_U\ell_U(s)B(U), \qquad M_{peak}^{ref}=\max_s M_{live}(s)$$

Under the inference schedule, buffers become dead as soon as their final declared reader completes:

Stage	Main live buffers	Live payload
input	$H$	64 MiB
score	$H,Z$	64.25 MiB
select	$H,Z,I,G$	64.375 MiB
count + prefix	$H,I,G,counts,offsets$	64.125065 MiB
pack	$H,I,G,counts,offsets,maps,X^{packed}$	192.250065 MiB
expert forward	$G,maps,X^{packed},U^{packed}$	256.1875 MiB
combine	$G,maps,U^{packed},Y$	192.1875 MiB
return	$Y$	64 MiB

The peak occurs while packed expert input and output rows coexist. It is not the sum of every tensor in the article. It is also not a full model peak: expert MLP intermediates, allocator fragmentation, framework state, residual-path state, and parameters are deliberately outside this routing envelope.

0x06E.4 Training Retention Is A Policy, Not A Free Extension

Backward needs enough state to reconstruct the exact forward relation. Loop 8 derives the gradients; this loop accounts for two outer-route retention choices.

Save outer route state. Retain:

hidden input
selected expert IDs and gates
expert offsets
route-to-token and route-to-choice maps
packed expert inputs and outputs

The declared retained payload is 320.250034 MiB, excluding expert-internal saved activations.

Rematerialize outer route state. Retain only:

hidden input
selected expert IDs
selected gate weights

Then replay count, prefix, route-map construction, and packing before the route backward. The retained payload is 64.125 MiB, avoiding 256.125034 MiB of outer-route state at the cost of replay work and temporary buffers.

Why retain IDs and gates rather than rerun the router? Because top-k is a policy boundary. Jitter, stochastic noise, equal-score tie policy, low-precision rounding, or a changed routing group can produce a different selected relation. Recomputing a different route is not activation checkpointing; it is a different forward pass. A more aggressive policy may save exact RNG and deterministic-kernel state, but then those objects become part of the manifest.

The packed input and output also have different overwrite rules:

inference:
    packed input may die after its expert's final read

training with saved expert state:
    packed input may be required by weight/input gradients
    packed output may be required by gate gradients

training with expert recomputation:
    both can die earlier, but the expert forward must be replayed exactly

Loop 8 owns the actual Jacobian and gradient path. Loop 7 establishes the state choices that make those gradients possible.

0x06E.5 Representation Changes Rows Before It Changes Hardware

The route histogram from Loop 6 is:

n = [6,5,3,2]
T = 8, E = 4, k = 2
requested routes = 16

Different execution representations expose different expert-row payloads before any kernel is chosen:

Representation	Represented rows	Semantic routes executed	Slack or loss
packed dropless	16	16	0
balanced fixed capacity, $C=4$	16	13	3 dropped and 3 unused slots
max-load fixed, $C=6$	24	16	8 slack rows
block rows, $B=2$	18	16	2 block-tail rows

For a general hard-route representation with $A$ expert rows:

$$M_{expert\ row\ pair}=A d(b_a+b_o)$$

That formula accounts for one materialized expert input/output pair. It does not say how many times a kernel reads those rows, whether input and output overlap, whether the output is accumulated in place, or whether padding participates in arithmetic.

Expert Choice has $A=EC$ selected expert slots before any implementation padding, but token degree is variable. Soft MoE has fixed slot rows $S$ and dense assignment state. If logits $L$, dispatch weights $D$, and combine weights $C$ are all materialized, their assignment payload is:

$$M_{Soft\ assignment} =TS(b_L+b_D+b_C)$$

and the slot input/output pair is:

$$M_{Soft\ slots}=Sd(b_a+b_o)$$

These formulas explain why “same number of experts” is not an equal-memory comparison across token choice, Expert Choice, and Soft MoE. Their route and assignment state have different axes.

0x06E.6 The Physical Runtime Firewall

The logical report stops at named tensor slices and schedule points. M6.6 must add the facts required to turn them into an execution claim:

Logical object fixed here	Physical question owned by M6.6
logical expert ID	which physical slot, replica, rank, and placement version?
accepted route rows	which are local, remote, padded, aligned, split, or replicated?
expert counts and offsets	which histogram, sort, prefix, packing, or fused kernel materializes them?
packed expert input/output	which workspace, allocator, lifetime fence, and dtype owns the storage?
represented rows $A$	padded batch, grouped GEMM, sparse blocks, or another backend?
route maps	local arrays, distributed dispatch handle, or recomputed metadata?
logical reads/writes	cache/HBM transactions, collectives, packetization, and measured time?

The often-used distributed payload expression

$$f_{remote}R d(b_a+b_o)$$

requires a physical placement and remote-route fraction $f_{remote}$. Neither exists in the M6.1 logical fixture. Even after they are supplied, the expression excludes count exchange, scale tensors, alignment, protocol headers, retransmission, topology imbalance, staging, and overlap. It is therefore an upper-level payload ledger, not a network trace.

M6.1 must not claim:

NCCL or DeepEP bytes
HBM bandwidth
sort or prefix-scan efficiency
grouped-GEMM or block-sparse occupancy
prefill/decode kernel choice
communication overlap
latency, throughput, or speedup

Those are not missing details. They are the explicit owner contract of M6.6.

0x06E.7 Executable Reconciliation And Failure Mutations

The checked-in Loop 7 oracle:

verifies R = T*k and every declared tensor shape
derives scalar payload from shape and dtype
replays stage-by-stage liveness and finds the peak
derives each reference traffic term independently
reconciles packed, fixed, max-load, and block-row counts
compares save-state and rematerialization retention
hashes the Loop 5 and Loop 6 upstream reports
rejects physical-runtime claims in the M6.1 owner namespace

Run it with:

python3 docs/numinens-evidence/m6-1-routed-capacity/logical_traffic_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop7_logical_traffic.py

Sixteen one-field mutations test route conservation, dtype widths, tensor shapes, unique buffer identities, lifetimes, route replay state, layout rows, block size, and the physical-runtime firewall.

0x06E.8 Traffic And Liveness Failure Lab

Failure	Why a numeric result can still look plausible	Broken contract
add every declared tensor and call it peak memory	all terms have byte units	non-overlapping lifetimes ignored
call the 641.062595 MiB reference ledger measured HBM traffic	operations are real	fusion, cache, tiling, and repeated physical reads omitted
call $Rdb$ network traffic	routed rows carry that payload	local routes and physical placement absent
count $Tdb$ for packed input at $k=2$	input has only $T$ unique rows	materialized route duplication omitted
count $Rdb$ for max-load fixed buffers	only $R$ semantic routes execute	represented slack rows omitted
call dropless padding-free	no semantic route is dropped	block-tail or alignment rows can remain
free gates before combine	output shape can still be allocated	weighted expert reduction becomes impossible
free route maps before combine	expert outputs retain valid shapes	source token and selected slot are lost
recompute top-k under different jitter	shapes and counts can match	backward follows a different route relation
overwrite packed input during training	forward can finish	saved-state backward may lose expert operands
include expert MLP intermediates in this oracle silently	raises the reported peak	owner and fixture scope changed without a manifest
attach a grouped-GEMM speedup to logical row slack	fewer represented rows can help	hardware and benchmark evidence absent

0x06F Single-Device Forward And Backward Integration Contract

The route tensors from the forward pass are also the coordinates of the backward pass. Autograd does not turn a sparse MoE into a dense expert update: the task-loss gradient follows accepted route rows, selected gates, and the exact expert computations that produced the output. At the same time, the router and expert paths meet again at the input hidden state, and the outer Transformer residual contributes a third gradient path.

This section derives that relation for one device and one task loss. It does not derive the auxiliary balancing or stability objectives; Loop 9 owns those. It also does not claim a distributed backward schedule, collective count, fused kernel, workspace, measured traffic, or throughput; M6.6 owns those physical contracts.

0x06F.1 Freeze The Forward Relation Before Differentiating It

Use PyTorch-style router storage $W_r\in\mathbb{R}^{E\times d}$ and hidden rows $H\in\mathbb{R}^{T\times d}$:

$$Z=H W_r^\top, \qquad Z\in\mathbb{R}^{T\times E}$$

For token $t$, top-k produces ordered integer expert IDs $I_{t,s}$ for slots $s\in\{1,\ldots,k\}$. The pinned readable Mixtral path first computes a full-expert softmax $p$, selects the top-k probabilities, and renormalizes those selected values:

$$p_{t,e}=\frac{\exp Z_{t,e}}{\sum_{j=1}^{E}\exp Z_{t,j}}, \qquad g_{t,s}=\frac{p_{t,I_{t,s}}}{\sum_{j=1}^{k}p_{t,I_{t,j}}}$$

For a fixed selected set, the full-softmax denominator cancels:

$$g_{t,s} = \frac{\exp Z_{t,I_{t,s}}} {\sum_{j=1}^{k}\exp Z_{t,I_{t,j}}}$$

This cancellation matters twice. It proves that the selected-renormalized path is a softmax over selected logits, and it proves that the task-loss path has zero derivative with respect to unselected logits while the integer IDs remain fixed. It does not make top-k differentiable. The operator is piecewise smooth between score-order boundaries.

Let $A_{t,s}\in\{0,1\}$ be the post-selection admission mask. The Loop 8 reference contract preserves pre-admission gates and assigns zero expert contribution to a rejected route; it does not renormalize after admission. For route coordinate $r=(t,s)$:

$$X_r=H_t, \qquad U_r=E_{I_{t,s}}(X_r), \qquad Y_t=\sum_{s=1}^{k}A_{t,s}g_{t,s}U_{t,s}$$

The surrounding Transformer block adds the residual outside the MoE operator:

$$R_t=H_t+Y_t$$

Changing any one of these choices changes the backward operator. In particular, post-admission renormalization, dense fallback, rerouting, residual fallback per rejected route, and a different selected set are not harmless implementation details.

0x06F.2 Combine Backward Is A Broadcast And A Dot Product

Let $D_t=\partial L/\partial R_t$. Because $R_t=H_t+Y_t$, the same upstream row $D_t$ enters the MoE branch and the identity residual branch. For an accepted route $r=(t,s)$, differentiating

$$Y_t=\sum_s A_{t,s}g_{t,s}U_{t,s}$$

gives:

$$\frac{\partial L}{\partial U_{t,s}} =A_{t,s}g_{t,s}D_t$$

and

$$q_{t,s} \equiv \frac{\partial L}{\partial g_{t,s}} =A_{t,s}\langle D_t,U_{t,s}\rangle$$

The vector gradient to an expert output is the token’s upstream row scaled by the route gate. The scalar gradient to the gate is a dot product between the upstream row and that route’s expert output. This explains Loop 7’s retention alternatives: a saved backward needs $U_{t,s}$ to form $q_{t,s}$, while an expert-rematerialized backward must reproduce the same $U_{t,s}$.

For a rejected route under the declared zero-output policy:

dU_route = 0
dg_route = 0
expert parameter gradient from that route = 0
packed-input gradient from that route = 0

That is not yet the complete router result because selected gates are coupled by their softmax denominator.

0x06F.3 Selected-Softmax Backward And The Dropped-Route Subtlety

For one token, write $q_s=\partial L/\partial g_s$ and define the gate-weighted mean:

$$\bar q_t=\sum_{j=1}^{k}g_{t,j}q_{t,j}$$

The selected-softmax Jacobian is:

$$\frac{\partial g_{t,j}}{\partial z_{t,s}} =g_{t,j}(\mathbf{1}[j=s]-g_{t,s})$$

Contracting that Jacobian with $q$ gives the selected-logit gradient:

$$\frac{\partial L}{\partial Z_{t,I_{t,s}}} =g_{t,s}(q_{t,s}-\bar q_t)$$

and, for the task-loss path with fixed IDs:

$$\frac{\partial L}{\partial Z_{t,e}}=0 \qquad e\notin\{I_{t,1},\ldots,I_{t,k}\}$$

Each selected-logit gradient row sums to zero:

$$\sum_{s=1}^{k}\frac{\partial L}{\partial Z_{t,I_{t,s}}}=0$$

That identity is a useful implementation assertion. Adding the same constant to every selected logit cannot change the selected softmax.

The rejected-route case is easy to implement incorrectly. A rejected route has $q_{t,s}=0$, yet its selected-logit gradient can be nonzero:

$$\frac{\partial L}{\partial Z_{t,I_{t,s}}} =-g_{t,s}\bar q_t \qquad \text{when }A_{t,s}=0$$

Why? The route still occupied one term in the pre-admission selected-softmax denominator. Changing its score changes the mass assigned to accepted routes even though its own expert did not run. If the implementation renormalizes only across accepted routes, this derivative changes. For a token with one accepted route, the accepted gate becomes one and the task loss supplies no relative router gradient at all. The admission and normalization order must therefore be part of the training manifest.

0x06F.4 Router Projection Backward

Scatter the selected gradients into $G_Z\in\mathbb{R}^{T\times E}$ at coordinates $(t,I_{t,s})$, leaving unselected coordinates zero. Since $Z=HW_r^\top$:

$$\frac{\partial L}{\partial W_r}=G_Z^\top H$$ $$\left.\frac{\partial L}{\partial H}\right|_{router}=G_ZW_r$$

The first equation reduces contributions from all token rows into the stored $[E,d]$ gate matrix. The second returns the router path to the hidden input. A code path that detaches selected gates before combine can still produce correctly shaped expert outputs, but $G_Z$, the router-weight gradient, and the router contribution to $H$ all disappear.

Top-k IDs are not ordinary floating-point operands:

selected values:
    differentiable inside the current active set

integer expert IDs:
    saved metadata, no ordinary infinitesimal gradient

score-order boundary:
    active set can jump; the local Jacobian changes discontinuously

A finite-difference test is valid only when its perturbations preserve the selected IDs. Loop 8 records the minimum score margin for exactly that reason.

0x06F.5 Bias-Free SwiGLU Expert Backward

The pinned Mixtral-style expert has three bias-free projections. For one packed route row $x\in\mathbb{R}^{d}$, first compute the gate and value projections:

$$a=W_1x, \qquad b=W_3x$$

Then apply the SiLU gate elementwise:

$$s=\operatorname{SiLU}(a), \qquad v=s\odot b$$

Finally project the gated intermediate back to model width:

$$u=W_2v$$

with $W_1,W_3\in\mathbb{R}^{d_{ff}\times d}$ and $W_2\in\mathbb{R}^{d\times d_{ff}}$. Given $g_u=\partial L/\partial u$:

$$g_v=W_2^\top g_u$$ $$g_a=(g_v\odot b)\odot\operatorname{SiLU}'(a), \qquad g_b=g_v\odot s$$

where

$$\operatorname{SiLU}'(a) =\sigma(a)\left[1+a(1-\sigma(a))\right]$$

The parameter and packed-input gradients are:

$$\frac{\partial L}{\partial W_2}=g_u v^\top, \qquad \frac{\partial L}{\partial W_1}=g_a x^\top, \qquad \frac{\partial L}{\partial W_3}=g_b x^\top$$ $$g_x=W_1^\top g_a+W_3^\top g_b$$

For multiple rows assigned to expert $e$, each matrix gradient is the sum of those route-row outer products. A zero-hit expert receives a zero task-loss gradient under this mathematical contract. A framework may represent that state as an explicit zero tensor or an absent/unused gradient depending on graph construction; optimizer and distributed reduction behavior must declare which representation is expected.

0x06F.6 Gather Backward Is Scatter-Add, Then The Residual Joins

Forward dispatch duplicates a token row when it has multiple accepted routes. Therefore its adjoint must add every routed input contribution back to the origin token:

$$\left.\frac{\partial L}{\partial H_t}\right|_{experts} = \sum_{r:\,token(r)=t} \frac{\partial L}{\partial X_r}$$

This is the backward counterpart of Loop 5’s rectangular incidence matrix. If $X^{packed}=QPH$, then:

$$\frac{\partial L}{\partial H} =P^\top Q^\top \frac{\partial L}{\partial X^{packed}}$$

The transpose is an additive reduction because one token can own multiple route rows. Indexed overwrite is wrong in backward for the same reason it was wrong in forward combine.

The complete block-input gradient is the sum of three paths:

$$\frac{\partial L}{\partial H} = \underbrace{D}_{residual} + \underbrace{G_ZW_r}_{router} + \underbrace{P^\top Q^\top G_X^{packed}}_{experts}$$

Omitting any term can leave all tensor shapes valid. The error appears in optimization, not necessarily in forward output.

0x06F.7 A Backward Trace That Preserves Route Identity

The following pseudocode is an algebraic trace, not a claim about a fused production kernel:

def moe_backward(d_block, saved, experts, router_weight):
    # saved IDs/gates/admission belong to the exact forward relation.
    ids = saved.selected_ids                 # [T,k], integer metadata
    gates = saved.selected_gates             # [T,k]
    admitted = saved.accepted_mask           # [T,k]
    route_meta = saved.route_metadata

    d_gate = zeros_like(gates)
    d_hidden_expert = zeros_like(saved.hidden)
    d_expert_weights = zeros_like_experts(experts)

    for route in route_meta:
        t, s, e, local_row = route
        if not admitted[t, s]:
            continue

        u = saved_or_recomputed_expert_output(e, local_row)
        x = saved_or_recomputed_expert_input(e, local_row)

        d_gate[t, s] = dot(d_block[t], u)
        d_u = gates[t, s] * d_block[t]
        d_x, d_theta = experts[e].backward(d_u, x)

        d_expert_weights[e] += d_theta
        d_hidden_expert[t] += d_x             # scatter-add, never overwrite

    centered = d_gate - sum(gates * d_gate, dim=-1, keepdim=True)
    d_selected_logits = gates * centered

    d_router_logits = zeros([T, E])
    scatter_selected(d_router_logits, ids, d_selected_logits)

    d_router_weight = d_router_logits.T @ saved.hidden
    d_hidden_router = d_router_logits @ router_weight

    d_hidden = d_block + d_hidden_router + d_hidden_expert
    return d_hidden, d_router_weight, d_expert_weights

saved_or_recomputed is a policy slot, not permission to rerun a different router. The safe rematerialization boundary from Loop 7 keeps selected IDs and gates fixed, reconstructs the same route maps and packed rows, and replays an expert with the same parameters, stochastic state, dtype policy, and activation semantics. If parameters have already been updated or offloaded to a different version before recomputation, the Jacobian no longer belongs to the recorded forward.

Mixed precision adds representation choices without changing the mathematical dependency graph:

forward hidden / expert rows:
    may use BF16 or FP16

router softmax and reductions:
    often promoted to FP32 for stability

saved gates:
    must declare whether backward uses FP32 probabilities or a cast copy

parameter-gradient accumulation:
    may use FP32 master/accumulator state

loss scaling:
    multiplies the upstream gradient and must be unscaled before optimizer checks

The Loop 8 oracle uses Python float64 to test algebra. Its tolerance is not a promised error bound for a mixed-precision training kernel.

0x06F.8 Exact Fixture And Finite-Difference Reconciliation

The checked-in fixture uses:

T = 3, d = 2, E = 3, k = 2, d_ff = 2
three bias-free SwiGLU experts
full softmax -> top-k -> selected renormalization
one rejected selected route, no post-admission renormalization
outer residual R = H + Y
fixed upstream gradient D

The selected IDs and gates are:

Token	Selected experts	Selected gates	Admission
0	[0,2]	[0.65475346, 0.34524654]	[accept, accept]
1	[1,2]	[0.55724785, 0.44275215]	[accept, reject]
2	[0,2]	[0.53991488, 0.46008512]	[accept, accept]

For token 1, the rejected expert-2 route has:

d_gate = 0
d_expert_output = [0,0]
d_packed_input = [0,0]
d(selected logit for expert 2) = -0.0157495397

The nonzero final line is the selected-softmax coupling derived above. The two selected-logit gradients for that token are equal and opposite, so their row sum is zero.

The oracle checks 48 scalar coordinates covering every hidden input element, every router-weight element, and every $W_1$, $W_3$, and $W_2$ element in all three experts. With central difference step $10^{-6}$:

all selected expert IDs remain unchanged
minimum top-k boundary margin = 0.23
maximum |analytic - numeric| = 1.23e-10

Run the exact replay and mutation suite with:

python3 docs/numinens-evidence/m6-1-routed-capacity/single_device_autograd_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop8_single_device_autograd.py

Eighteen one-field mutations reject changed router orientation, probability policy, integer-ID differentiation, post-admission renormalization, dense fallback, missing residual, wrong SwiGLU shapes, missing rejected-route coverage, unsafe finite-difference scale, and a distributed-runtime ownership claim.

0x06F.9 Single-Device Autograd Failure Lab

Failure	Why shapes can still pass	Broken gradient contract
detach selected gates before combine	expert path still produces $[T,d]$	router receives no task gradient
differentiate full-softmax probabilities but omit selected renormalization	selected IDs and output shape match	wrong gate values and wrong router Jacobian
send gradient to integer top-k IDs	metadata has numeric storage	no ordinary local derivative exists for index changes
give every expert every token’s gradient	expert matrices have compatible shapes	sparse task credit assignment destroyed
discard rejected gate from denominator without declaring renormalization	route does not execute	different forward and router gradient
update rejected expert parameters	expert ID was selected	admission mask ignored
scatter packed-input gradients by overwrite	destination row exists	one of multiple route contributions is lost
omit $G_ZW_r$ from $dH$	expert and residual paths remain	router dependence on input hidden state lost
omit residual $D$ from $dH$	MoE-local derivative remains valid	surrounding Transformer block differentiated incorrectly
reuse inference lifetimes in training	forward succeeds	expert/gate operands can die before backward
rerun top-k with different jitter during checkpoint replay	shapes and route counts may match	backward differentiates a different forward relation
finite-difference across a top-k tie	numeric slope is computable	local fixed-active-set Jacobian no longer applies
call a float64 oracle a BF16 error guarantee	equations reconcile	numerical implementation claim was never measured
call local scatter-add a distributed reduction schedule	algebra is correct	rank ownership, collectives, and topology remain unspecified

0x06G Paper-By-Contract Comparison: Similar Edge Counts, Different Operators

The history in 0x01A and the mechanism alternatives in 0x06C answer two different questions. The history explains which problem each paper attacked. The mechanism sections derive individual operators. Neither is enough to decide whether two named “MoE” methods are interchangeable.

A paper-level comparison needs a complete contract. For method $m$, define:

$$\mathcal{C}_m= (\mathcal{G}_m,\mathcal{S}_m,\mathcal{K}_m,\mathcal{A}_m, \mathcal{N}_m,\mathcal{O}_m,\mathcal{T}_m,\mathcal{X}_m)$$

where:

G: candidate group and dependency scope
S: scorer and selection owner
K: token-side or expert-side degree budget
A: admission, capacity, and overflow policy
N: normalization and combine operator
O: training objectives, noise, and numerical policy
T: logical tensors retained for forward and backward
X: execution representation

Two methods are contract-equivalent only if the axes needed by the claimed use are equal. Equal total parameters, equal $k$, equal mean expert rows, or even equal selected edges does not establish that relation.

0x06G.1 Jacobs Is A Dense Statistical Mixture, Not A Sparse Transformer Layer

Adaptive Mixtures of Local Experts supplies the division-of-labor ancestor. For one example, the gate produces positive weights that sum to one:

$$p_e(x)=\frac{\exp r_e(x)}{\sum_{j=1}^{E}\exp r_j(x)}$$

and all local experts define components of the mixture. In a simple output-space sketch one may write $\sum_e p_eE_e(x)$, but the paper’s learning rule is stronger than “average every regressor with MSE”: expert credit is governed by the expert’s posterior responsibility under the mixture likelihood. The gate learns to increase the probability of experts that explain that case well; experts compete instead of all receiving the same undifferentiated target error.

For the Loop 10 teaching group, evaluating every expert gives:

$$R_{Jacobs}=TE=8\cdot4=32\text{ expert rows}$$

There is no sparse top-k index, group capacity, or dropped-token contract in this ancestor. Reading the 1991 diagram as a modern sparse dispatch graph would erase the very algorithmic change introduced later.

0x06G.2 Shazeer Makes The Gate Sparse And The Load A Training Object

Shazeer et al. change the gate itself. Before softmax, they add learned-scale Gaussian noise and replace all but the largest $k$ coordinates by $-\infty$:

$$H_e(x)=xW_{g,:,e}+\epsilon_e\operatorname{Softplus}(xW_{noise,:,e})$$

The sparse gate is then:

$$G(x)=\operatorname{Softmax}\!\left(\operatorname{KeepTopK}(H(x),k)\right)$$

The selected relation is token-owned: every token asks for $k$ experts, while the number of rows received by one expert remains data-dependent. Importance CV$^2$ and expected-load CV$^2$ are two separate batch controls, as derived in 0x06D. The learned noise makes route replay depend on both matrices and the random draw.

The paper’s gating equation does not by itself define GShard’s later group-local fixed capacity $C$ or its probabilistic second route. The Loop 10 fixture records an explicit zero noise draw so that Shazeer and Mixtral can share the same 16 selected edges for one forward comparison:

$$R_{Shazeer}=Tk=8\cdot2=16$$

That equality isolates a useful result: two forward selections can coincide while their training objective, stochastic state, implementation, and overflow contracts remain different.

0x06G.3 GShard Top-2 Is Not Deterministic Dropless Top-2

GShard turns routing into a group-local bounded assignment. Let $C$ be expert capacity inside a declared group. After top-2 selection, normalize the selected gates:

$$(\tilde g_{t,1},\tilde g_{t,2}) = \frac{(g_{t,1},g_{t,2})}{g_{t,1}+g_{t,2}}$$

The first route is admitted when its expert buffer has space. The second route also requires a recorded uniform draw $u_t$:

$$A_{t,2} = \mathbf{1}[c_{e_{t,2}}<C]\, \mathbf{1}[u_t<2\tilde g_{t,2}]$$

The factor $2\tilde g_{t,2}$ makes low-weight second choices less likely to consume capacity. The algorithm processes first choices before second choices and advances expert decision counters in a declared order. Therefore group membership, token order, random state, and capacity are semantic inputs, not kernel trivia.

With $C=4$ and the fixture draw $u_t=0.25$ for every token:

requested top-2 edges:       16
fixed represented rows E*C: 16
accepted first routes:        8
accepted second routes:       5
semantic expert rows:        13

Three represented rows are slack. Two requested second routes to expert 0 and one late second route to expert 1 fail capacity. Calling this path “top-2, therefore 16 expert computations” is wrong even though its router returned two IDs per token.

0x06G.4 Switch Changes Both Degree And Failure Semantics

Switch Transformer chooses one expert:

$$e_t=\arg\max_e p_{t,e}, \qquad A_t=\mathbf{1}[c_{e_t}<C]$$

Top-1 removes the second expert branch; it does not remove capacity, imbalance, auxiliary pressure, selective router precision, or training jitter. An overflow token skips the expert contribution and continues through the Transformer’s outer residual path. That is not a hidden dense fallback and not a second-choice reroute.

For the same probabilities and $C=2$:

represented rows E*C: 8
accepted expert rows:  6
dropped token IDs:     [4,7]

Both dropped tokens preferred expert 0 after its two slots were occupied. Their output still has the correct Transformer shape, which is why an implementation can silently implement the wrong fallback without a shape error.

0x06G.5 Mixtral Pins A Different Top-2 Contract

Mixtral also gives two experts per token, but the pinned readable Transformers implementation used in this article is not GShard Algorithm 1. It computes a full-expert float32 softmax, takes top-2, renormalizes the selected probabilities, executes every selected route, and restores expert-major rows with index_add_:

$$g_{t,s} = \frac{p_{t,I_{t,s}}} {\sum_{j=1}^{2}p_{t,I_{t,j}}}, \qquad Y_t=\sum_{s=1}^{2}g_{t,s}E_{I_{t,s}}(H_t)$$

For the same fixture:

$$R_{Mixtral}=Tk=16$$

There is no fixed-$C$ rejection in that pinned reference. Its optional auxiliary function aggregates top-k slot frequencies, its checkpoint records zero jitter, and it does not implement ST-MoE z-loss. “Top-2” is therefore only one coordinate of its contract.

0x06G.6 Equal 16-Row Budgets Still Do Not Equalize Expert Choice Or Soft MoE

Expert Choice and Soft MoE expose why a scalar compute budget is not an assignment definition.

Expert Choice. With $E=4$ and $C=4$:

$$R_{EC}=EC=16, \qquad \bar{k}_{token}=\frac{EC}{T}=2$$

But the exact token degrees are [3,3,3,3,1,1,2,0], not eight copies of two. One token is uncovered. Expert load is fixed; token work and coverage are not. The complete candidate group is also required before each expert can rank its top-$C$ tokens.

Soft MoE. Set total slots $S=16$, so the expert MLPs also process 16 rows. The dispatch and combine relations are nevertheless dense matrices over $T\times S$ token-slot affinities:

$$D=\operatorname{softmax}_{token}(L), \qquad C=\operatorname{softmax}_{slot}(L)$$

Each matrix has:

$$TS=8\cdot16=128\text{ assignment scalars}$$

before accounting for logits or gradients. Every slot is a convex mixture of the declared token group, and every token combines all slot outputs fractionally. Sixteen hard edges, sixteen fixed expert buckets, and sixteen soft slots are three different operators.

0x06G.7 MegaBlocks Is An Execution Overlay, Not A Ninth Router

MegaBlocks begins after a hard route relation exists. It changes how unequal expert rows are represented so all requested routes can execute through block-sparse operations. It does not decide whether tokens or experts own selection, replace the router objective, or change selected gate weights.

For histogram $[6,5,3,2]$ and teaching block size $B=2$:

$$A_e=B\left\lceil\frac{n_e}{B}\right\rceil, \qquad A=[6,6,4,2]$$

so:

semantic token-choice routes: 16
represented block rows:       18
block-tail rows:                2

MegaBlocks can preserve the same 16 Mixtral-style selected edges while changing packing, padding, and kernels. That is execution equivalence at the semantic-route boundary, not paper equivalence across the whole contract.

0x06G.8 The Contract Matrix In Executable Form

A review should compare fields, not method labels:

@dataclass(frozen=True)
class MoEContract:
    candidate_group: str
    selection_owner: str
    degree_budget: str
    admission: str
    normalization_and_combine: str
    training_controls: tuple[str, ...]
    retained_state: tuple[str, ...]
    execution_representation: str


def equivalent_for_claim(left, right, required_axes):
    return all(getattr(left, axis) == getattr(right, axis)
               for axis in required_axes)

The required axes depend on the claim:

same selected relation:
    candidate_group + scorer + selection owner + degree + tie/noise state

same forward output:
    selected relation + admission + gate normalization + experts + combine

same backward:
    same forward + retained/rematerialized state + auxiliary objectives

same logical memory:
    represented rows + assignment tensors + metadata + liveness schedule

same runtime:
    all of the above + placement + kernels + topology + measured workload

Only the first four are partially owned here. Physical runtime equivalence remains an M6.6 measurement claim.

0x06G.9 Exact Replay And Non-Equivalence Checks

The Loop 10 oracle consumes one $[8,4]$ probability matrix and one deterministic tie policy. It derives token top-2, GShard’s two-pass admission, Switch top-1 overflow, expert-side top-$C$, Soft-MoE assignment shape, and block-tail rounding. It then proves six claims:

Shazeer, Mixtral, Expert Choice, and Soft MoE can each expose a scalar 16-row budget
GShard executes only 13 routes under the recorded capacity and random draws
Switch executes 6 rows and drops token IDs 4 and 7
Expert Choice leaves one token uncovered despite mean degree 2
Soft MoE needs 128 scalars in each [T,S] assignment matrix
MegaBlocks preserves 16 semantic routes while representing 18 block rows

Run it with:

python3 docs/numinens-evidence/m6-1-routed-capacity/variant_contract_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop10_variant_contracts.py

Sixteen one-field mutations reject an unspecified tie policy, an invented speedup, deterministic GShard second routing, Switch dense fallback, fixed token degree for Expert Choice, one-axis Soft-MoE normalization, cross-request soft assignment, MegaBlocks treated as a selector, padding-free block execution, and a false equal-budget reconciliation.

0x06G.10 Paper-Comparison Failure Closure

Top-2 used as a complete method name. GShard’s stochastic capacity-bounded second route and Mixtral’s dropless selected-renormalized route are collapsed.

Equal mean routes read as equal per-token work. Expert Choice’s degree vector and uncovered token disappear.

Equal expert rows read as equal assignment memory. Soft MoE’s two dense $[T,S]$ normalization domains disappear.

Dropless read as padding-free. MegaBlocks block tails disappear from represented-row accounting.

Training losses treated as interchangeable. Shazeer CV$^2$, GShard/Switch frequency-probability proxies, the pinned top-k extension, and z-loss lose their distinct statistics and gradients.

A forward coincidence promoted to implementation equivalence. Shazeer with a zero noise draw and Mixtral can select the same edges in this fixture while requiring different replay state and training controls.

An encoder/group operator called a causal serving drop-in. Expert Choice full-group ranking and Soft-MoE token mixing cross the dependency boundary derived in 0x06C.

A kernel paper promoted to a routing algorithm. MegaBlocks is assigned selection semantics it deliberately inherits from another router.

0x06H Sensitivity, Failure Closure, And Downstream Handoffs

The earlier sections derive individual contracts. They do not yet answer a practical question: which conclusion survives when one routing coordinate changes? A capacity factor, route histogram, routing-group size, or execution block size can each alter a different ledger. Sweeping all of them at once would produce numbers without an explanation.

Loop 11 therefore freezes the logical operator before varying one coordinate. The reference state is:

tokens T:             8
experts E:            4
selected experts k:   2
requested routes R:   16
expert route counts:  [6,5,3,2]
capacity rounding:    ceil
admission:            fixed drop
arbitration:          descending gate, then token ID, then selected slot

For a policy vector $\theta$, write the observable logical ledger as:

$$\mathcal{L}(\theta) =\left(C,A,D,U,\rho,P_B\right)$$

where $C$ is expert capacity, $A$ accepted routes, $D$ dropped routes, $U$ unused fixed slots, $\rho$ fixed-slot utilization, and $P_B$ block-tail rows. A one-factor sensitivity comparison changes one coordinate of $\theta$ while preserving the score matrix, selected relation, admission order, and every other coordinate needed by the claim.

This is not a quality or runtime experiment. It is an exact logical counterfactual over the route relation already replayed in Loops 4, 6, and 10.

0x06H.1 Capacity Slack Trades Drops For Reserved Rows

For requested route counts $n_e$ and capacity factor $\mathrm{CF}$:

$$C(\mathrm{CF}) =\left\lceil \mathrm{CF}\frac{R}{E}\right\rceil$$

Fixed-drop admission then gives:

$$A=\sum_{e=1}^{E}\min(n_e,C)$$ $$D=R-A$$

Reserved but unused rows are not the same quantity as drops:

$$U=EC-A$$

and fixed-slot utilization is:

$$\rho=\frac{A}{EC}$$

For the frozen [6,5,3,2] histogram:

The direction is not “larger is better.” Raising capacity monotonically removes drops for this frozen histogram, but also reserves more fixed rows and lowers utilization. Whether those rows become extra arithmetic, memory traffic, or only addressable capacity depends on the physical representation and kernel, which belongs to M6.6.

Changing $\mathrm{CF}$ can also change the computed function. At CF=1.0, three selected expert contributions are absent. At CF=1.5, all sixteen execute. A capacity sweep is therefore not a pure systems sweep unless the route policy is dropless or output equivalence is separately established.

0x06H.2 Equal Route Mass Does Not Equalize Skew

Capacity is computed from the balanced mean $\mu=R/E$, but overflow is caused by the complete count vector. Two useful shape statistics are count CV$^2$:

$$\operatorname{CV}^{2}(n) =\frac{\frac{1}{E}\sum_e(n_e-\mu)^2}{\mu^2}$$

and normalized count entropy, where $q_e=n_e/R$:

$$H_{norm}(n) =-\frac{\sum_e q_e\log q_e}{\log E}$$

With $R=16$, $E=4$, and CF=1.0, every scenario has $C=4$ and sixteen reserved rows:

The last two rows are count-level histograms, not route sets the frozen fixture can produce. Each of its eight tokens selects two distinct experts, so one expert can receive at most $T=8$ routes; [10,2,2,2] and [16,0,0,0] keep $R=16$ only by breaking that bound. They isolate how the count vector alone changes admission. The same formulas give two route sets the fixture can realize: [8,4,2,2] (CV$^2=0.375$, $H_{norm}=0.875$; 12 accepted, 4 dropped, 4 unused) and [8,8,0,0], where every token selects the same two experts (CV$^2=1.0$, $H_{norm}=0.5$; 8 accepted, 8 dropped, 8 unused). No realizable histogram has a larger CV$^2$, a lower entropy, or more drops than [8,8,0,0].

The same $R$, mean, capacity, and total reserved rows produce drop counts from zero to twelve across these count vectors, and from zero to eight across route sets the fixture can realize. A scalar “tokens per expert on average” cannot reconstruct admission. CV$^2$ and entropy summarize skew, but they also do not reconstruct which token lost which expert contribution. Exact replay still needs selected route identity and arbitration order.

These statistics have different uses. CV$^2$ is sensitive to large deviations and appears naturally in the Shazeer balance objectives derived in 0x06D. Entropy describes concentration on a bounded scale. Neither is a replacement for per-expert counts, device aggregation, or link traffic. M6.2 consumes the full statistics and decides which control signal is meaningful for its routing policy.

0x06H.3 Routing-Group Size Introduces A Ceiling Discontinuity

Even a perfectly balanced local count pattern can reserve extra rows because $C$ is integer. For group size $T_g$:

$$R_g=T_gk$$ $$S_{round}=EC-R_g$$

when no route overflows. With $E=4$, $k=2$, CF=1.0, and balanced-as-possible group counts:

The staircase comes from the ceiling, not from a changed router. Small groups can therefore have high fractional reservation overhead even under balanced routing. Conversely, merging groups may improve the capacity ledger while changing candidate competition, overflow arbitration, request isolation, and possibly causality. Loop 4’s globally balanced but locally overflowing example proves that an aggregate histogram cannot repair a group-local contract after the fact.

0x06H.4 Dropless Does Not Remove Granularity Sensitivity

A dropless execution can preserve all selected routes while its physical row representation still changes with block size $B$:

$$A_e(B)=B\left\lceil\frac{n_e}{B}\right\rceil$$ $$P_B=\sum_e A_e(B)-R$$

For the same [6,5,3,2] semantic routes:

All four rows preserve the same sixteen expert contributions. Only the representation changes. This is why Loop 6 treated MegaBlocks as an execution overlay and why M6.1 cannot infer speed from fewer logical drops. The block size also controls tile occupancy, metadata, launch shape, and kernel availability; M6.6 must measure those physical effects.

0x06H.5 A Sensitivity Result Needs Its Frozen Coordinates

A defensible sensitivity row records:

frozen:
    score/probability tensor revision
    valid-token mask and token order
    selected expert IDs and selected slots
    gate normalization and dtype
    tie, noise, and random state
    routing-group identity and visibility
    arbitration order and overflow action

varied:
    exactly one of capacity factor, count histogram,
    routing-group size, or representation block size

reported separately:
    requested routes, accepted routes, dropped routes,
    unused fixed slots, represented rows, and block tails

The route histogram cannot be called frozen while changing top-$k$, candidate scope, or the score matrix. Capacity-factor and group-size sweeps are especially easy to misread because both can change integer $C$ while only group construction changes which tokens compete.

0x06H.6 Handoff To M6.2: Preserve Training And Admission Statistics

M6.2 owns expert/device/communication balance and DeepSeek-style controllers. It cannot reconstruct those controls from packed expert rows alone. The M6.1 handoff therefore contains:

group identity + valid-token mask
full router probabilities [T,E]
selected expert IDs and selected slots [T,k]
selected gate weights [T,k]
admission mask [T,k]
selected and executed counts [E]
dropped route IDs
objective statistics
tie/noise/random state

Selected counts and executed counts must remain separate. The first describes router demand; the second includes admission. Full probabilities are required for probability-frequency balance losses even when only top-$k$ routes execute. M6.1 does not assign physical ranks, replicas, collectives, or kernels in this record.

0x06H.7 Handoff To M6.6: Preserve Route Identity And Representation

M6.6 starts after logical admission. Its minimum input is an accepted-route record:

token order
accepted route IDs
origin token IDs and selected slots
logical expert IDs and gate weights
route-metadata dtype
per-expert counts and prefix offsets
fixed, ragged, or block representation policy
represented rows, block size, and tail rows
saved or rematerialized backward state

That is sufficient to ask M6.6 to add physical expert placement, replica/version identity, transport, local kernel, overlap, and observed timing. It is deliberately insufficient to answer those questions inside M6.1. A record that already contains a device ID or kernel name would silently make a placement choice before the physical owner evaluates alternatives.

0x06H.8 Handoff To M6.8: Generalize The Choice Axis, Not The FFN Layout

M6.8 routes attention components, depth participation, or modality-conditioned paths. Reusing M6.1 means generalizing the semantic axes:

choice axis
candidate scope
selection owner
visibility scope
degree budget
admission policy
combine operator
state-growth axis
train/decode consistency

It does not mean forcing every method into [T,k] FFN expert buffers. A depth router may change which layers create K/V state. An attention router may select heads or projections. A modality router may have stage-specific candidate sets. The reusable contract is “what is selected, from which visible candidates, under which admission and combine rule,” not one expert-major tensor layout.

0x06H.9 Cross-Loop Failure Closure

Failure	Earliest evidence that rejects it	Downstream consequence if missed
total experts treated as active experts	Loop 2 parameter/residency oracle	model and memory ledgers are wrong before routing begins
full-softmax and selected-softmax weights conflated	Loop 3 router-policy oracle	forward scale and gradients drift despite identical IDs
routing group omitted	Loop 4 capacity oracle	overflow and batch dependence cannot be replayed
inverse permutation used instead of scatter-add	Loop 5 dispatch oracle	top-$k>1$ token contributions are lost
selection family collapsed with execution policy	Loop 6 routing-family oracle	Expert Choice, Soft MoE, and dropless work are misclassified
logical payload promoted to runtime traffic	Loop 7 traffic oracle	cache reuse, collectives, and kernel reads are fabricated
gate-gradient path detached	Loop 8 autograd oracle	router receives no task credit
z-loss called load balance	Loop 9 objective oracle	stability and allocation gradients are confused
paper name reduced to top-k	Loop 10 variant oracle	GShard, Switch, and Mixtral contracts appear interchangeable
capacity, skew, group, and block size varied together	Loop 11 sensitivity oracle	no causal explanation survives the comparison
one generic router object passed downstream	Loop 11 handoff manifest	M6.2 lacks probabilities, M6.6 lacks route identity, or M6.8 inherits an FFN-only schema

0x06H.10 Exact Replay And Negative Fixtures

The Loop 11 oracle derives all four sweeps and validates owner-specific handoff schemas:

python3 docs/numinens-evidence/m6-1-routed-capacity/sensitivity_handoff_oracle.py --fail-on-error
python3 -m unittest docs/numinens-evidence/m6-1-routed-capacity/test_loop11_sensitivity_handoffs.py

Its valid fixture proves:

capacity-factor sweep drop counts: [5,3,1,0]
capacity-factor fixed-slot utilization: [0.9167,0.8125,0.7500,0.6667]
same-mass skew sweep drop counts: [0,3,6,12]
small-group ceiling overhead: [2,0,2,0,0] rows
block-tail rows for B=[1,2,4,8]: [0,2,8,16]
handoff required-field counts: M6.2=13, M6.6=15, M6.8=10

Sixteen one-field mutations reject route-mass drift, unsupported rounding, false capacity/drop claims, negative counts, group-rounding drift, invalid block size, invented quality/runtime evidence, a missing handoff manifest, and an undeclared dropless policy.

The result closes M6.1’s sensitivity surface without choosing a DeepSeek controller, a distributed runtime, or a non-FFN routing method. Those are now explicit inputs to their owners rather than prose promises.

0x07 Minimal Capacity-Aware Pseudocode

The capacity check can be layered into the MoE forward path:

def dispatch_with_capacity(h, expert_idx, weights, capacity, overflow_policy):
    counts = [0 for _ in range(num_experts)]
    buffers = {e: [] for e in range(num_experts)}
    locations = []
    overflow = []

    # 0x06A priority: descending gate, then token ID, then selected slot
    routes = sorted(
        (
            (token_id, choice_id)
            for token_id in range(h.shape[0])
            for choice_id in range(expert_idx.shape[1])
        ),
        key=lambda r: (-weights[r[0], r[1]], r[0], r[1]),
    )

    for token_id, choice_id in routes:
        expert = expert_idx[token_id, choice_id]

        if counts[expert] < capacity:
            slot = counts[expert]
            counts[expert] += 1
            buffers[expert].append((token_id, h[token_id]))
            locations.append((token_id, choice_id, expert, slot))
        else:
            overflow.append((token_id, choice_id, expert))

    if overflow:
        overflow_policy.apply(overflow)

    return buffers, locations, overflow

The sort applies the arbitration order declared in 0x06A before any slot is claimed, so this sketch accepts the same routes as arbitrate_fixed_capacity. Visiting tokens and selected slots in source order instead would let early tokens fill an expert whatever their gates: that is the source-order contract, valid only when it is the pinned policy, and it can change which routes overflow.

The combine path must know:

which token produced this expert input
which selected-expert slot it represented
which gate weight should scale the output
where the final token row belongs

The invariant:

input rows:  [T,d]
output rows: [T,d]

The inside of the MoE layer may shuffle, group, drop, or reroute. The layer boundary still has to return one row per input token.

0x07A Implementation Assertions

A MoE layer can preserve shapes while still being wrong. Useful implementation assertions are:

assert logits.shape == [T, E]
assert expert_idx.shape == [T, k]
assert weights.shape == [T, k]
assert allclose(weights.sum(axis=-1), ones([T]))
assert output.shape == [T, d_model]

Capacity assertions:

for expert in range(E):
    assert routed_count[expert] <= capacity

for token_id in range(T):
    assert combine_has_output_row(token_id)

Debug trace fields:

token_id
selected_slot
expert_id
expert_local_row
gate_weight
overflow_policy_applied

Those fields are enough to audit a small route by hand. Without them, a bug can hide behind a correct [T,d] output shape.

0x08 Single-Device MoE vs Expert Parallelism

On one device, the logical layer can be taught as:

score -> top-k -> group -> run local experts -> scatter back

At scale, experts may live on different devices. The same accepted route then needs a second, physical identity:

logical route fixed by M6.1:
    token row, selected slot, logical expert, gate

physical plan added by M6.6:
    origin rank, physical replica, destination rank,
    send offset, receive offset, local kernel row,
    placement version, completion state

The route conservation invariant still matters, but it is not a transport measurement:

accepted logical routes
    = locally executed routes + remotely executed routes

returned route outputs
    = exactly one output for every executed route

A21’s expert-parallel section and the DeepSpeed-MoE paper verify the need to move selected activations to expert owners and return their outputs. They do not make T*k*d*bytes a universal network trace. The physical payload needs placement, local-route fraction, dtype, padding, metadata, and transport policy. Section 0x06E now fixes the exact logical namespace; M6.2 owns balancing and DeepSeek route control, while M6.6 owns dispatch collectives, kernels, topology, prefill/decode behavior, and measured performance.

0x09 What MoE Changes And What It Does Not

MoE changes the FFN capacity contract:

dense:
    every token uses the same FFN parameters

MoE:
    each token uses selected expert parameters

MoE does not remove the rest of the Transformer:

attention:
    still runs unless separately changed

residual stream:
    still expects [B,S,d_model] rows

KV cache:
    still exists in autoregressive serving

non-expert weights:
    still exist and are active

It also does not automatically reduce memory:

serving process:
    must have access to all required experts

training process:
    must store/update expert parameters and route gradients

distributed runtime:
    may shard experts, but then pays communication

The right summary:

MoE trades dense universal compute for sparse routed compute plus routing, capacity, and communication machinery.

0x0A Failure Lab

0x0B Red Flags

Misconception	Correction
”MoE reduces total parameters.”	Usually the opposite: MoE increases total parameters while reducing active expert parameters per token.
”MoE is free capacity.”	The new costs are router computation, load balancing, capacity buffers, dispatch/combine, and communication.
”Routing is just top-k.”	Top-k only selects candidates. The system still needs capacity, overflow policy, dispatch, expert compute, and combine.
”All top-k routers use the same gate formula.”	Full-expert softmax, selected-axis normalization, sigmoid affinities, and post-selection renormalization are distinct policies.
”Token dropping is inherent to MoE.”	It is common with fixed-capacity dense buffers, but dropless block-sparse or ragged execution is another design point.
”Expert Choice gives every token the capacity factor’s expert count.”	Capacity factor fixes the average selected edges per token; individual tokens can receive many experts or none.
”Soft MoE is top-k without indices.”	It is a dense token-slot assignment with different dispatch and combine normalization axes.
”Dropless means no padding anywhere.”	Dropless preserves requested routes; a block implementation can still round each expert to a block boundary.
”Experts are guaranteed to become interpretable specialists.”	Specialization is learned and empirical; it varies by layer, objective, data, and routing policy.
”Dropped tokens are harmless.”	Dropping or skipping expert routes changes computation and can hurt training or output quality.
”Expert parallelism removes the memory problem.”	It shards experts, but introduces all-to-all communication and synchronization.
”MoE changes attention.”	In the standard Transformer MoE path, the MoE layer changes the FFN side. Attention remains a separate mechanism.
”Active parameters equal served memory.”	Serving memory must account for resident experts, non-expert weights, buffers, and routing metadata.
”Adding all buffer sizes gives peak memory.”	Peak is the maximum sum of simultaneously live buffers under a declared schedule.
”Logical bytes are measured traffic.”	Logical reads/writes, physical HBM traffic, and network payload require different manifests and evidence.
”Activation checkpointing can rerun any router.”	Route replay must preserve selected IDs, gates, group membership, tie policy, and stochastic state.
”A rejected expert route has no router gradient.”	Its expert gradient is zero under the declared policy, but its selected logit can still receive gradient through the pre-admission softmax denominator.
”Top-k is differentiable because selected gates have gradients.”	Selected values are differentiable inside a fixed active set; integer route IDs and active-set changes are not ordinary local derivatives.
”Dispatch backward restores one row by assignment.”	Multiple route rows can originate from one token, so gather backward is a scatter-add reduction.
”The MoE input gradient is only the expert gradient.”	The full block input receives residual, router, and routed-expert contributions.
”All auxiliary MoE losses are equivalent.”	Shazeer importance/load CV losses, GShard/Switch hard-frequency proxies, and router z-loss have different statistics and gradients.
”The balance loss only updates selected logits.”	The pinned proxy uses a full-expert softmax, so unselected router logits can receive auxiliary gradient.
”Observed load count is differentiable.”	A hard selected or executed count is telemetry unless a declared surrogate or stochastic estimator supplies a gradient.
”Top-k and top-1 balance losses have the same uniform scalar.”	The pinned slot-summed top-k extension has raw uniform reference $k$, while Switch top-1 has reference one.
”Router z-loss is another load-balance loss.”	It controls the log-partition and the softmax-invariant common-mode logit direction; it does not count routes.
”The checkpoint coefficient proves the auxiliary loss trained the model.”	In the pinned implementation, router logits must be requested for the branch to execute; this article does not reproduce checkpoint training.

0x0C What M6.1 Hands To M6.2, M6.6, And M6.3

M6.1 gives M6.2 the basic routed layer:

tokens -> router -> selected experts -> capacity -> dispatch -> combine

M6.2 asks what happens when this is scaled:

load balancing:
    how do we prevent expert collapse?

DeepSeek MoE:
    what do fine-grained experts, shared experts, and device-limited routing change?

M6.1 gives M6.6 an immutable logical execution input:

accepted route relation and route identity
represented expert-row count
logical tensor shapes and dtypes
reference buffer births and last uses
reference activation and metadata reads/writes
fixed-active-set forward/backward relation
route-row, selected-gate, router, expert, and residual adjoints
full-softmax auxiliary probability axis and hard-frequency stop-gradient contract
pre-admission selected, post-admission executed, and gate-mass telemetry namespaces
router z-loss as a separate common-mode stability objective

M6.6 adds what cannot be inferred from those objects:

physical expert placement and replicas
local versus remote routes
collectives and transport handles
grouped, padded, or block-sparse kernels
workspaces, completion fences, and overlap
measured bytes, latency, and throughput
distributed backward exchange and expert-gradient reductions

M6.1 also gives M6.3 a contrast:

MoE:
    changes capacity and active routing in the base architecture

LoRA/PEFT:
    freezes base weights and adds small trainable adapter state

Those should not be merged into one topic. MoE is routed capacity. LoRA is parameter-efficient adaptation.

0x0D Publish Gate

Check	Status
Historical lineage, capacity, sparsity, expert, and routing prerequisites precede mechanics	pass
Activation sparsity, dense overwork, representation interference, and finite budgets are separated as four motivations	pass
Division-of-labor intuition is followed by explicit limits on semantic specialization	pass
Local experts, sparse gating, GShard, Switch, and modern execution are presented as a problem-response lineage	pass
Dense FFNs, activation sparsity, dropout, ensembles, soft mixtures, static sparsity, and sparse MoE are contrasted	pass
Dense FFN baseline is explicit	pass
Dense FFN FLOP ledger is explicit	pass
MoE is framed as conditional FFN capacity	pass
Router/top-k/gate weights are defined	pass
Dispatch/combine are treated as first-class operations	pass
Capacity and overflow are explicit	pass
Toy dispatch and capacity examples are included	pass
Selected-axis softmax and scatter metadata invariants are explicit	pass
Full-softmax, selected normalization, and alternative score policies are distinguished	pass
Stored router axes, runtime transpose semantics, and [T,E] logits are reconciled	pass
Normalized top-k equivalence is derived and the unnormalized counterexample is explicit	pass
Router precision, unstable ties, and train/evaluation jitter are part of the policy contract	pass
The pinned Mixtral router policy is replayed by an executable oracle and negative mutations	pass
Routing-group and train/evaluation capacity semantics are explicit	pass
Token-choice, Expert Choice, Soft MoE, and dropless execution are compared	pass
Routing-selection semantics and route-execution representation are separate axes	pass
Equal 16-route token-choice and Expert Choice fixtures prove different token-degree and coverage contracts	pass
Expert Choice bucket, edge-count, mean-capacity, combine, candidate-group, and autoregressive boundaries are derived	pass
Soft MoE dispatch/combine axes, slot-cost object, and causal boundary are explicit	pass
Soft MoE convex-mixture properties and a complete numeric tensor trace are replayed by an exact oracle	pass
Fixed drop, max-load padding, and dropless block-tail ledgers are reconciled over one route histogram	pass
MegaBlocks is described as dropless but not padding-free, with physical kernels and measurement deferred to M6.6	pass
Buffer payload is derived from shape and dtype, not prose labels	pass
Sum of declared buffers, live peak, reference traffic, and network payload are separate ledgers	pass
The packed reference schedule derives score, select, count/prefix, pack, expert-handoff, and combine traffic	pass
Stage-by-stage liveness derives the 256.1875 MiB reference inference peak	pass
Save-state and rematerialize policies retain enough state to preserve the selected relation	pass
Packed, balanced fixed, max-load fixed, and block-row payloads reconcile one route histogram	pass
Expert Choice and Soft MoE expose their distinct route/assignment memory axes	pass
Sixteen invalid fixtures close shape, dtype, route, lifetime, replay, layout, and owner-boundary failures	pass
M6.1 logical bytes are explicitly prevented from becoming M6.6 physical performance claims	pass
Full-softmax selected renormalization reduces to selected-logit softmax under a frozen active set	pass
Combine adjoints derive both gate scalars and expert-output vectors	pass
Selected-softmax, router-weight, and router-input gradients are derived with explicit axes	pass
Rejected-route expert gradients and selected-logit gradients are distinguished	pass
Bias-free SwiGLU expert parameter and packed-input adjoints are derived	pass
Gather backward is an additive route-to-token reduction	pass
Residual, router, and expert input-gradient paths are reconciled	pass
Forty-eight scalar gradients pass fixed-active-set central finite differences	pass
Router gradient and numerical-stability paths are explicit	pass
Shazeer importance and expected-load CV losses are derived as different objectives	pass
GShard/Switch hard-frequency and mean-probability axes are explicit	pass
The Switch proxy gradient is derived under fixed hard IDs and stop-gradient frequencies	pass
The pinned Mixtral function is traced as [L*B*S,E] -> [N,k,E] -> [k,E] with padding masks	pass
Top-1 and top-k raw uniform references are distinguished as one and $k$	pass
Full-softmax auxiliary gradients and selected-gate task gradients have different exposure	pass
Router z-loss, its derivative, and its nonzero common-mode row sum are derived	pass
ST-MoE z-loss is not attributed to the pinned Mixtral implementation	pass
Checkpoint coefficient, router-logit output switch, labels, and loss addition are traced	pass
Selected, executed, dropped, gate-mass, and gradient telemetry remain separate	pass
Two padded rows are excluded from the exact objective fixture	pass
Fifty-four scalar objective gradients reconcile under fixed top-k IDs	pass
Sixteen invalid router-training mutations close axis, denominator, mask, source, and claim failures	pass
Jacobs, Shazeer, GShard, Switch, Mixtral, Expert Choice, and Soft MoE are compared by complete operator contracts	pass
GShard stochastic second-route admission is distinguished from deterministic dropless Mixtral top-2	pass
Equal 16-row budgets are shown not to equalize hard edges, token coverage, dense assignment state, or represented rows	pass
Switch top-1 overflow is traced to the outer residual rather than an invented dense fallback	pass
MegaBlocks is treated as an execution overlay that preserves routing semantics but retains block-tail rows	pass
The paper-contract oracle replays GShard, Switch, Expert Choice, Soft-MoE accounting, and block rounding on one score matrix	pass
Sixteen invalid variant mutations close paper-name, axis, causality, padding, and physical-performance conflations	pass
Capacity factor, route skew, routing-group size, and block size are swept one coordinate at a time	pass
Accepted routes, dropped routes, unused fixed slots, and block-tail rows remain separate ledgers	pass
Equal route mass is shown not to equalize CV-squared, entropy, overflow, or token-level route identity	pass
Routing-group ceiling discontinuities are derived without hiding candidate-group dependence	pass
M6.2 receives probability, selection, admission, objective, and replay state without physical placement	pass
M6.6 receives accepted-route identity, metadata, representation, and replay state without invented kernels or measurements	pass
M6.8 receives generalized choice, scope, visibility, admission, combine, and state-growth axes without an FFN-only layout	pass
Sixteen invalid sensitivity mutations close mass, rounding, claim, block, and handoff drift	pass
DeepSeek measured-load bias control remains M6.2 ownership	pass
Distributed gradients, bytes, kernels, and timing remain M6.6 ownership	pass
Implementation assertions can catch shape-correct routing bugs	pass
Total vs active vs served-memory ledgers are separated	pass
Distributed all-to-all is introduced but deferred to M6.2 and M6.6	pass
LoRA/PEFT is deferred to M6.3	pass
M4/M5 content is reused only as boundary context	pass

0xFF Reference

• Transformer Series A21 - MoE
• Transformer System M4.1 - Transformer Cost Model
• Transformer System M5.1 - Autoregressive Serving State
• Jacobs et al., Adaptive Mixtures of Local Experts
• Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• MoEfication: Transformer Feed-forward Layers are Mixtures of Experts
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Mixtral of Experts
• Mixtral-8x7B-v0.1 checkpoint, pinned revision
• Hugging Face Transformers Mixtral implementation, pinned commit
• PyTorch torch.topk documentation
• DeepSpeed-MoE
• ST-MoE: Designing Stable and Transferable Sparse Expert Models
• Mixture-of-Experts with Expert Choice Routing
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
• From Sparse to Soft Mixtures of Experts