M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing

0x00 Overview: When Sparse Routing Becomes A Distributed System

M6.1 ended with a complete logical MoE layer:

tokens -> router -> selected experts -> dispatch -> expert compute -> combine

The owner question for this article is:

How do routing objectives, expert structure, and placement constraints co-evolve when a sparse MoE layer is trained and distributed at scale?

That pipeline can look like a clean way to buy more parameters without paying for every expert on every token. It hides the harder question: what must remain true for conditional computation to work after the router, experts, and hardware begin learning and operating together?

A29 introduces the problem with a software team. One option is to hire one generalist who writes the interface, service, database, and deployment code. That resembles a dense FFN: the same parameter block processes every token. A second option is to hire specialists and a manager who assigns each task. That resembles MoE: experts provide different transformations, and a router decides which expert receives each token.

The specialist organization only works if two contracts hold:

expert contract:
    each specialist develops useful capability,
    and the specialists collectively cover the work

router contract:
    the manager understands the specialists,
    sends each task to useful people,
    and does not overload one person while others wait

The analogy is useful, but its limits matter. Neural experts are not assigned human-readable job descriptions. Their specializations emerge from data, routing, and optimization. A token can use several experts, shared experts can run for every token, and a physical accelerator can own many logical experts. The manager is also trained by gradients produced by the organization it is managing. Routing mistakes therefore change which experts learn, and what the experts learn changes later routing decisions.

This creates a feedback system rather than a static lookup:

small routing preference
    -> one expert receives more examples
    -> that expert gets more optimization signal
    -> its output becomes more useful for those examples
    -> the router has more reason to select it again

The loop can produce useful specialization, but it can also produce collapse. A few experts become hot and well trained; the remaining experts become cold and undertrained. The model still reports top-k sparse activation, yet the promised capacity is not being used evenly and the slowest hot expert controls the layer time.

0x00A Dynamic Routing Creates Six Coupled Pressures

A29’s opening difficulty inventory is broader than load balance alone. Dynamic routing and sparse activation create at least six coupled pressures:

Pressure	Model-level symptom	Runtime symptom	Why a local top-k equation is insufficient
expert load imbalance	a few experts dominate while others receive little training	hot expert batches, idle kernels, stragglers	top-k constrains routes per token, not routes per expert
router degradation	early preferences become self-reinforcing and exploration shrinks	increasingly skewed route histograms	the router learns from consequences of its own earlier choices
parameter and memory growth	total capacity expands much faster than activated capacity	all expert weights or shards still need storage and residency	inactive arithmetic is not absent model state
communication bottleneck	selected experts may live away from token-origin ranks	all-to-all traffic and topology hot spots dominate latency	an expert ID is also a network destination
fragmented expert batches	each expert receives a smaller, variable microbatch	poor matrix utilization, padding, or ragged-kernel overhead	global batch size does not equal per-expert batch size
drop-versus-pad tradeoff	overflow routes lose expert computation or buffers contain slack	quality/runtime mismatch, wasted bandwidth, or OOM pressure	bounded hardware shapes conflict with stochastic route counts

Consider ten tokens and six experts. If five tokens choose one expert while the other five choose five different experts, the global batch is ten but expert batch sizes are [5,1,1,1,1,1]. The hot expert performs a useful matrix multiplication, while the five cold experts execute tiny matrices or occupy padded capacity slots. Equal active FLOPs on paper do not imply equal kernel efficiency or equal completion time.

Gradient flow is uneven too. An overloaded expert accumulates gradients from more examples. A cold expert sees fewer examples and may fail to discover a useful region. The router and expert parameters are jointly optimized, so load imbalance is not merely an after-training deployment issue. It changes the training trajectory that produced the model.

0x00B A Route Has Semantic And Physical Meaning

The logical layer description assumes an expert can be called without asking where it lives. At large scale, that assumption fails. The model may contain dozens or hundreds of experts, while one accelerator owns only a shard of them. A token’s route now determines both a neural computation and a network destination.

logical question:
    which expert should transform this token?

physical question:
    which rank owns that expert, and how does the token get there?

This coupling is the central difficulty of distributed MoE. Dynamic routing and sparse activation create capacity, but the same decisions can concentrate tokens on a few experts, devices, or links. The result can be sparse in arithmetic yet slow in wall-clock time because one expert is overloaded or one collective waits on a straggling rank.

The Transformer Series A29 article begins from these difficulties before introducing DeepSeek. It names load imbalance, router degradation, parameter and memory pressure, communication bottlenecks, and the tension between dropping overflow tokens and padding fixed expert buffers. M6.2 follows the same teaching order, but turns each problem into an explicit tensor, placement, or optimization contract.

0x00C DeepSeek Is A Problem-Response Sequence

DeepSeekMoE, DeepSeek-V2, and DeepSeek-V3 are not a list of model names to memorize. They are a sequence of responses to different bottlenecks:

DeepSeekMoE / V1 lineage:
    conventional experts mix unrelated knowledge and duplicate common knowledge
    -> finer routed experts plus isolated shared experts

DeepSeek-V2:
    many fine-grained experts increase destination fanout
    -> device-limited routing, communication balance, device-level dropping

DeepSeek-V3:
    strong auxiliary losses can interfere with the language-model objective
    -> sigmoid affinities, group-limited routing, and load-control bias

deployment:
    balanced logical routes still need executable placement and collectives
    -> runtime telemetry, replication, dispatch, and topology policy

That progression fixes the ownership boundary for the rest of the article:

Contract	M6.2 responsibility
consumes	M6.1’s routing-group identity, valid-token mask, full router probabilities, selected expert/slot IDs, selected gate weights, admission mask, route counts, dropped-route IDs, objective statistics, and tie/noise state
owns	expert-, device-, node-, and communication-load definitions; differentiable balance objectives; DeepSeek fine-grained/shared expert structure; V2 locality and dropping; V3 affinity, selection-bias, and gate semantics
hands forward	logical destinations, route multiplicity, semantic locality restrictions, route loads, and controller state needed to realize the layer physically
defers	rank/replica placement, collective algorithms, grouped kernels, overlap, graph capture, and measured latency to M6.6
defers	Expert Choice, conditional attention/depth, MoA/MoD, and multimodal routing to M6.8
defers	MLA cache and absorbed-kernel mechanics to M5.8, routed adapter composition to M6.9, and quantized deployment to M8

The boundary is deliberately asymmetric. M6.2 may derive a logical communication payload, but it cannot claim a physical collective, topology, kernel, or latency result. Conversely, M6.6 may optimize execution, but it cannot silently change selected experts, gate weights, load-control state, or drop semantics.

0x00D Historical And Evidence Ladder

The lineage is easier to understand as a sequence of contracts that became insufficient:

Step	Contract introduced	Limitation exposed for M6.2
adaptive mixtures of local experts	a learned gate assigns examples to competing expert predictors	useful specialization can concentrate assignments and learning signal
sparsely gated MoE	a large expert bank is activated sparsely per example	sparse arithmetic introduces load, capacity, and communication problems
GShard	expert routing is sharded across a distributed mesh with explicit capacity and auxiliary balance	bounded buffers can drop work and distributed completion follows the slowest shard
Switch Transformer	top-1 routing simplifies the selected path and studies selective router precision and stability	fewer routes per token do not remove hot experts, undertraining, or communication skew
DeepSeekMoE / V1	fine-grained routed experts and isolated shared experts attack knowledge mixing and redundancy	more logical experts increase routing and destination-management pressure
DeepSeek-V2	device-limited routing and multi-scope balance couple specialization to communication locality	locality constraints and device-level dropping become part of model semantics
DeepSeek-V3	group-limited sigmoid affinity and measured-load selection bias reduce dependence on a strong batch-level auxiliary loss	controller state, selection scores, combine weights, and production placement must remain separate

No one source proves that whole ladder. This article uses four evidence classes with different authority:

A29 source-local explanation:
    teaching order, terminology, analogies, and links into the original corpus

primary papers and technical reports:
    stated algorithm, training objective, architecture, and source-reported results

pinned checkpoint configs and reference code:
    persistent dimensions, stored state, and one executable forward contract

production runtime:
    placement, collectives, kernels, overlap, and measured behavior; owned by M6.6

An equation or intuition inherited from A29 is not promoted to a universal fact merely because it is pedagogically useful. When a report, released checkpoint, and public inference file expose different slices of the system, the article keeps those slices separate rather than filling the gaps by inference.

0x01 Prerequisites: Load, Capacity, Placement, And Collectives

0x01A A Route Is Logical; An Expert Has Physical Ownership

Let a local rank start with token rows:

H_r: [T_r,d]

The router produces selected logical expert IDs:

expert_ids_r: [T_r,k]
gate_weights_r: [T_r,k]

A placement table resolves each expert ID to an owner:

owner[e] -> (node_id, rank_id, local_expert_slot)

This mapping is independent of the router’s semantic score. Two experts with similar routing behavior may reside on the same rank and create a device hotspot; a balanced expert histogram can still create uneven network traffic if placement groups hot experts poorly.

Keep three levels distinct:

Level	Unit	Typical failure
expert load	tokens or routes per expert	one expert is overloaded or starved
device load	aggregate routes for experts owned by a rank	one accelerator becomes the step limiter
communication load	bytes exchanged between ranks or nodes	one link or destination hot-spots

DeepSeek-V2’s separate expert-, device-, and communication-level objectives make sense only after these are treated as different random variables.

0x01B Why Routers Naturally Become Imbalanced

Suppose $n_e$ is the number of selected routes assigned to expert $e$ in one balancing scope. Ideal uniform token-choice routing would give approximately:

$$n_e^* = \frac{Tk}{E}$$

where $T$ tokens each create $k$ routes over $E$ experts. Real routers do not remain uniform by default. Small early score differences can create a feedback loop:

expert receives more tokens
    -> expert receives more gradient examples
    -> expert improves faster for those examples
    -> router assigns it even more tokens

The overloaded expert gets a larger local batch and more training signal; underused experts get neither. This is simultaneously:

an optimization problem:
    experts learn at different rates and specialization can collapse

a systems problem:
    the slowest expert/device determines synchronization time

Balance should therefore be measured rather than inferred from average top-k sparsity. Useful telemetry includes route counts $n_e$, mean router affinity $p_e$, overflow count, per-rank token count, all-to-all send/receive bytes, and expert compute time.

0x01C Capacity Converts Dynamic Routing Into Bounded Buffers

Static accelerator programs and efficient grouped kernels often need a bounded expert batch. A common capacity approximation is:

$$C = \left\lceil c_f\frac{Tk}{E}\right\rceil$$

where $c_f$ is a capacity factor. Capacity turns an unbounded random route count into a buffer shape:

expert_input: [E,C,d]

But it creates two opposing inefficiencies:

C too small:
    overflow routes must be dropped, rerouted, or handled by a fallback

C too large:
    underfilled expert slots are padded or reserved but do no useful work

Ragged or block-sparse expert kernels can reduce padding, but they do not remove imbalance. A large expert batch still takes longer, and distributed ranks still exchange unequal traffic. Capacity is a runtime contract, not a complete balancing strategy.

0x01D Expert Parallelism Is A Distributed Transpose

With expert parallelism, token-origin ranks and expert-owner ranks are different ownership layouts. The forward pass changes layout twice:

token-major rows on origin ranks
    -> pack by destination expert
    -> all-to-all
expert-major rows on owner ranks
    -> expert FFNs
    -> all-to-all
token-major outputs on origin ranks

If each selected route carries a hidden row of width $d$ and element size $b$, the logical routed activation volume before locality is approximately:

$$M_{routes} \approx T_{global} k d b$$

Only routes whose expert owner differs from the token’s current rank cross the network, so actual link traffic is this volume multiplied by the remote-route fraction, then shaped by topology, padding, alignment, and collective implementation. The combine exchange moves a remote output payload of similar order. Training adds reverse-path activation-gradient exchanges. Expert gradients are reduced across any data-parallel replicas of the same expert, not indiscriminately across all expert-parallel ranks. This is why sparse expert arithmetic does not imply cheap distributed execution.

An all-to-all is also synchronization-sensitive. The collective cannot finish until its participants provide and receive their partitions. One overloaded rank, one slow expert, or one congested inter-node path can set layer latency.

0x01E Balance Is A Multi-Objective Constraint

The main language-model loss asks the network to predict tokens. It does not directly ask routes to use hardware evenly. Auxiliary balancing losses add that pressure, but the coefficient creates a tradeoff:

too weak:
    hot experts, starvation, overflow, stragglers

too strong:
    routing is pushed toward uniformity even when specialization prefers otherwise

This explains the evolution from classic auxiliary losses to DeepSeek-V3’s auxiliary-loss-free control bias. “Loss-free” does not mean “balance-free.” It means the balancing control is moved out of the main differentiable objective: expert selection scores receive an adjustable bias for routing, while the affinity used to weight selected expert outputs remains a separate quantity.

0x01F DeepSeek Vocabulary Before The Equations

The rest of the article uses several related but non-interchangeable ideas:

Term	Local meaning
routed expert	selected conditionally for each token
shared expert	always active path intended to capture common computation
fine-grained expert	smaller expert unit created by segmenting a wider FFN capacity
device-limited routing	token routes may touch only a bounded set of devices
node-limited routing	routes are constrained at node granularity before expert selection
affinity score	learned token-expert compatibility
control bias	non-gradient routing adjustment used to influence load
selected gate	weight used to combine outputs of selected experts

The methods solve different problems. Fine-grained segmentation changes specialization granularity; shared experts change which computation is unconditional; locality constraints change communication topology; balancing losses or biases change route distribution.

0x01G Claim Status And Scope

Claim	Status	Evidence	Rewrite-safe wording
Scaling MoE creates load imbalance, communication bottlenecks, and memory pressure.	Source-local and externally verified	A29, DeepSpeed-MoE, Switch	”Sparse activation reduces active expert compute, but distributed routing adds load and communication constraints.”
DeepSeekMoE uses fine-grained expert segmentation and shared expert isolation.	Source-local and externally verified	A29, DeepSeekMoE paper	”DeepSeekMoE changes both the granularity of routed experts and the always-on shared path.”
DeepSeek-V2 and DeepSeek-V3 separate total and activated parameters.	Externally verified	DeepSeek-V2, DeepSeek-V3 reports	”Parameter counts are source-reported and must not be turned into general MoE guarantees.”
Auxiliary-loss-free balancing is still a balancing mechanism.	Source-local and externally verified	A29, Loss-Free Balancing paper, DeepSeek-V3	”It removes auxiliary-loss gradients, not the need to control expert load.”
Expert parallelism uses all-to-all style token exchange.	Source-local and externally verified	A21, DeepSpeed-MoE	”All-to-all is the communication price of placing experts on different ranks.”
DeepSeek checkpoint and reference implementations expose version-specific score, grouping, and scaling rules.	Externally verified, artifact-specific	Official DeepSeekMoE, V2, and V3 code/config artifacts	”Pin the artifact before turning a paper-level router into code.”
Llama 4 Maverick alternates dense/MoE layers and uses a shared expert plus one of 128 routed experts.	Externally verified, source-reported	Official Meta Llama 4 overview	”Use this as a routing-contract comparison, not a quality conclusion.”

0x01H Immutable Route Manifest: Stored State Before Algorithm

Before comparing balance losses or locality rules, freeze what each released artifact actually stores. Let $T$ be the number of valid token rows in one explicit routing scope, $d$ the residual width, $E$ the routed-expert count, $K$ the selected routed experts per token, $G$ the number of logical expert groups, and $G_k$ the selected logical groups. The learned router has stored axes:

$$W_r\in\mathbb{R}^{E\times d}$$

and maps a token-major input to an expert-major score axis:

$$H\in\mathbb{R}^{T\times d}, \qquad Z=HW_r^\top\in\mathbb{R}^{T\times E}.$$

When grouping is enabled and $E$ is divisible by $G$, the released implementations reshape the expert axis as:

$$Z_{grouped}\in\mathbb{R}^{T\times G\times(E/G)}, \qquad group(e)=\left\lfloor\frac{e}{E/G}\right\rfloor.$$

That is a logical contiguous-ID grouping rule. It is not, by itself, a proof that group $g$ is device $g$ or node $g$. A report can describe device- or node-limited intent while a config stores only n_group and topk_group. A physical manifest must still bind logical expert IDs to ranks, replicas, devices, and nodes; M6.6 owns that binding.

The pinned public artifacts establish this cross-version manifest:

Field	DeepSeekMoE 16B / V1 lineage	DeepSeek-V2	DeepSeek-V3 671B reference
layer schedule	28 layers; layer 0 dense; 27 MoE layers	60 layers; layer 0 dense; 59 MoE layers	61 layers; first 3 dense; 58 MoE layers
residual width $d$	2,048	5,120	7,168
routed / shared experts	$E=64$, $S=2$	$E=160$, $S=2$	$E=256$, $S=1$
selected routed experts	$K=6$	$K=6$	$K=8$
routed expert width	1,408	1,536	2,048
logical groups	no grouped selection field; treat as $G=1$	$G=8$, $G_k=3$, 20 experts/group	$G=8$, $G_k=4$, 32 experts/group
score family	softmax	softmax	sigmoid
selection state	softmax probabilities	softmax probabilities after group eligibility	sigmoid affinity plus FP32 correction bias after group eligibility
combine-weight source	selected softmax values; no selected renormalization	selected softmax values multiplied by 16; no selected renormalization	original sigmoid affinities, selected-axis renormalization, then multiply by 2.5
router tensor	[64,2048]	[160,5120]	weight [256,7168], correction bias [256]
released config dtype	BF16	BF16	FP8 weight configuration with FP32 scale state; reference default GEMM path dequantizes to BF16

The table has three deliberate qualifications.

First, norm_topk_prob=false means different things in the V1/V2 code and the V3 code. V1 leaves the selected values at their original softmax mass. V2 multiplies those values by routed_scaling_factor=16. V3 explicitly renormalizes selected original sigmoid affinities and then applies route_scale=2.5. A shared Boolean name does not create a shared numerical contract.

Second, the V3 correction bias is selection state, not expert-output weight state. The public forward adds the FP32 bias before group and expert top-k, then gathers combine weights from the uncorrected sigmoid affinities. Replacing the latter with corrected scores changes the model even though all tensor shapes remain valid.

Third, a model-level dtype label is not an instruction to cast every tensor to that type:

V1 checkpoint declaration:
    base/router/expert storage declared BF16;
    released router code has no explicit FP32 score cast

V2 checkpoint declaration:
    base/router/expert storage declared BF16;
    released router explicitly casts hidden rows and router weight to FP32,
    computes softmax in FP32, and later casts expert output back

V3 reference declaration:
    linear weights may be FP8 with FP32 block scales;
    default reference GEMM path dequantizes for BF16 compute;
    correction bias is FP32;
    selected weights are returned in the input activation dtype

Treating dtype=fp8 as “scores, bias, normalization, and accumulation are all FP8” would erase the precision boundaries that make the released forward meaningful.

For a bias-free SwiGLU expert with intermediate width $d_e$, each routed expert stores:

$$P_{expert}=d_ed+dd_e+d_ed=3dd_e.$$

The shared path in these released implementations is one wider SwiGLU MLP whose width is $S d_e$, so its parameter count equals $S P_{expert}$. This gives the per-MoE-layer structural ledger:

Artifact	one routed expert	routed bank $E P_{expert}$	active routed path $K P_{expert}$	always-on shared path $S P_{expert}$	router/control parameters
V1 16B	8,650,752	553,648,128	51,904,512	17,301,504	131,072
V2	23,592,960	3,774,873,600	141,557,760	47,185,920	819,200
V3 reference	44,040,192	11,274,289,152	352,321,536	44,040,192	1,835,008 router weights + 256 control-bias values

These are structural MoE-layer counts. They exclude attention, norms, embeddings, dense FFNs, quantization scales, tensor-parallel storage effects, replicas, optimizer state, and transient route buffers. They also do not convert model-level “activated parameters” reported by a paper into this local expert-only ledger.

Capacity needs the same evidence discipline:

Evidence surface	V1	V2	V3
released config	no fixed capacity or drop field	no fixed capacity or drop field	no fixed capacity or drop field
released reference forward	ragged route counts; no fixed-capacity admission step	ragged counts and optional expert parallel exchange; no report-style device-capacity drop step	local-expert loop plus reduction; no fixed-capacity admission step
report-level statement	do not infer a capacity policy from the config	training describes device-level capacity and lowest-affinity dropping	report states no token dropping and describes separate deployment load-balance strategies

Therefore capacity = none is not a safe cross-version conclusion. The artifact-level statement is narrower: these pinned public configs do not persist a fixed-capacity field, and these reference inference paths do not implement the report’s complete training-time capacity controller. Loop 6 will derive V2 admission and dropping from its report contract; Loop 7 will keep V3’s no-drop system statement separate from the public inference loop.

0x01I Expert, Group, Device, Node, And Communication Load Are Different Measurements

“Load” is often introduced as the number of tokens sent to an expert. That is only the first projection of the route relation. A complete measurement contract must say which token rows are valid, whether it measures requested or admitted routes, whether it counts rows or gate mass, which aggregation map is applied, and whether local traffic is excluded. Without those coordinates, two dashboards can display different values under the same label and both look plausible.

Start from a scope, not a denominator shortcut. Let a balancing scope contain $T$ token rows, a valid-row mask $v_t\in\{0,1\}$, and $K_t$ selected slots for valid token $t$. $K_t$ is written as a variable even when a model uses fixed top-$K$: padding, conditional routes, and admission policy can make T * K the wrong observed denominator.

For selected expert ID $e_{t,j}$ and admission bit $a_{t,j}$, define two route indicators:

$$S_{t,j,e} = v_t\,\mathbf{1}[e_{t,j}=e], \qquad X_{t,j,e} = S_{t,j,e}\,a_{t,j}.$$

$S$ records router demand before capacity, dropping, or fallback. $X$ records the routed expert rows that actually execute. Their expert histograms are:

$$n^{sel}_e=\sum_{t,j}S_{t,j,e}, \qquad n^{exec}_e=\sum_{t,j}X_{t,j,e}, \qquad d_e=n^{sel}_e-n^{exec}_e.$$

The conservation laws expose off-by-one and padding errors:

$$R_{sel}=\sum_e n^{sel}_e=\sum_t v_tK_t, \qquad R_{exec}=\sum_e n^{exec}_e=\sum_{t,j,e}X_{t,j,e}, \qquad R_{drop}=R_{sel}-R_{exec}=\sum_e d_e.$$

The corresponding route fractions use their own observed mass:

$$q^{sel}_e=\frac{n^{sel}_e}{R_{sel}}, \qquad q^{exec}_e=\frac{n^{exec}_e}{R_{exec}}.$$

These quantities are undefined when their denominator is zero. Reporting an empty scope as a perfectly balanced all-zero vector would make “no work” look like ideal routing. The machine-readable oracle therefore returns undefined for nonlinear metrics on an empty scope rather than fabricating zeros.

Gate mass is semantic influence, not row load. If $g_{t,j}$ is the selected combine weight, define:

$$m^{sel}_e=\sum_{t,j}S_{t,j,e}g_{t,j}, \qquad m^{exec}_e=\sum_{t,j}X_{t,j,e}g_{t,j}.$$

One route contributes one expert input row regardless of whether its gate is 0.9 or 0.1. The two routes can have very different influence on the output, but they still create two expert rows. Conversely, V2’s selected weights are scaled by 16, while V3 renormalizes selected sigmoid affinities and then scales by 2.5; raw gate mass is not a cross-version compute metric. Count load and gate mass therefore remain separate telemetry families.

The always-on shared path is separate again. With $S_{shared}$ shared expert units and $T_v=\sum_t v_t$ valid tokens, its logical row work is $T_vS_{shared}$. It is not inserted into the routed-expert histogram because it has no top-$K$ competition and no routed-load denominator.

Aggregation is a typed matrix projection. Let $B^{group}\in\{0,1\}^{G\times E}$ map each expert to one logical expert-ID group. Then:

$$n^{group}=B^{group}n^{expert}.$$

For the pinned V2 and V3 artifacts, this logical grouping is derived from contiguous expert IDs. It is useful for group eligibility and group-local statistics, but it remains physically unbound.

Only after an explicit placement manifest exists may another incidence matrix $B^{device}\in\{0,1\}^{D\times E}$ aggregate expert rows onto devices:

$$n^{device}=B^{device}n^{expert}.$$

A device-to-node incidence matrix $B^{node}\in\{0,1\}^{N\times D}$ then gives:

$$n^{node}=B^{node}n^{device}.$$

This derivation exposes a critical non-equivalence:

$$B^{group}\ne B^{device}$$

unless a versioned placement artifact explicitly proves equality. A checkpoint field named n_group=8 is not evidence for eight devices or one group per node. The Loop 3 fixture deliberately maps experts from one logical group across two devices so an implementation that reuses group IDs as device IDs fails.

Replication needs one more distinction. If several devices can execute the same logical expert, an expert-to-device ownership table is not sufficient to reconstruct load. The accepted route must already contain the resolved physical destination or replica ID. Replica choice, placement search, and physical dispatch belong to M6.6; M6.2 only defines how to aggregate a supplied mapping.

Communication load is a source-destination relation. Let $o_t$ be the origin device for token $t$, and let $\delta(e)$ be the resolved destination for expert $e$. The executed route-count matrix is:

$$C_{u,v} = \sum_{t,j,e} X_{t,j,e}\, \mathbf{1}[o_t=u]\, \mathbf{1}[\delta(e)=v].$$

Its row and column sums answer different questions:

$$s_u=\sum_v C_{u,v}, \qquad r_v=\sum_u C_{u,v}.$$

$s_u$ is the number of executed route rows produced by source device $u$; $r_v$ is the number received for experts on destination device $v$. The diagonal $C_{u,u}$ is local expert work, not network traffic. Define the remote matrix:

$$C^{remote}_{u,v}=C_{u,v}\mathbf{1}[u\ne v].$$

Device-limited routing constrains each token’s number of distinct destinations,

$$\phi_t= \left| \left\{\delta(e_{t,j}):a_{t,j}=1\right\} \right|,$$

but a small $\phi_t$ does not imply balanced receive columns. Every source can use only one remote destination and still overload the same receiver. Likewise, balanced node compute $n^{node}$ does not imply balanced inter-node traffic: the same total work can flow primarily in one direction.

This matrix counts logical routed rows. Turning a count into bytes requires an explicit payload schema, metadata schema, dtype, local/remote split, padding or ragged representation, and forward/backward phase. Loop 9 derives that logical traffic handoff, and M6.6 chooses and measures the physical collective. Loop 3 does not call $C$ measured link bandwidth or latency.

Skew summaries require the complete axis and the correct order of operations. For a non-empty nonnegative count vector $n\in\mathbb{R}^{U}$ with $R=\sum_u n_u$ and $\mu=R/U$, useful summaries include:

$$\operatorname{CV}^{2}(n) = \frac{\frac{1}{U}\sum_u(n_u-\mu)^2}{\mu^2}, \qquad \rho_{max}(n)=\frac{\max_u n_u}{\mu}.$$

With $q_u=n_u/R$ and the convention $0\log0=0$:

$$H_{norm}(n) = -\frac{\sum_u q_u\log q_u}{\log U}, \qquad E_{eff}(n)=\exp\!\left(-\sum_uq_u\log q_u\right).$$

CV$^2$ emphasizes large deviations, $\rho_{max}$ exposes the worst owner, normalized entropy lies between zero and one when $U>1$, and $E_{eff}$ reports the effective number of occupied units. None identifies which routes were dropped or which source-destination pair is hot, so the full count vectors and traffic matrices remain primary evidence.

Distributed scopes must aggregate linear counts before computing nonlinear summaries:

$$n^{global}=\sum_r n^{(r)}, \qquad \operatorname{CV}^{2}(n^{global}) \ne \frac{1}{R}\sum_r\operatorname{CV}^{2}(n^{(r)}) \quad\text{in general}.$$

Averaging per-rank CV$^2$, entropy, or max/mean values changes the statistic. The scope ID, valid-token mask, aggregation group, route stage, and reduction order are therefore part of telemetry identity, not logging metadata.

One route relation, five different conclusions. The executable fixture uses five valid tokens, eight experts, top-2 selection, four logical groups, four devices, and two nodes. Two selected routes are dropped. Its expert counts are:

selected expert rows: [2,1,1,1,2,1,1,1]   R_sel  = 10
executed expert rows: [1,1,1,1,2,0,1,1]   R_exec =  8
dropped expert rows:  [1,0,0,0,0,1,0,0]   R_drop =  2

The same executed vector projects differently:

Axis	Executed counts	CV$^2$	What it says
expert	[1,1,1,1,2,0,1,1]	0.250	one hot and one starved routed expert
logical group	[2,2,2,2]	0.000	contiguous ID groups are exactly balanced
physical device	[2,2,3,1]	0.125	explicit placement is device-imbalanced
physical node	[4,4]	0.000	aggregate node compute is balanced
remote receive by device	[0,2,2,0]	1.000	network receive work is concentrated despite balanced nodes

The executed device traffic matrix is:

              destination device
source          0  1  2  3
device 0       [2, 2, 0, 0]
device 1       [0, 0, 1, 0]
device 2       [0, 0, 1, 0]
device 3       [0, 0, 1, 1]

Four of eight executed rows are device-remote, but only one crosses a node boundary. The node compute vector is balanced while the inter-node transfer is directional from node 0 to node 1. No scalar “load balance” score preserves all of those facts.

The telemetry path should preserve typed primitives and derive summaries later:

def measure_route_load(route, placement):
    valid = route.valid_token_mask[:, None]
    selected = valid & route.selected_slot_mask
    executed = selected & route.admission_mask

    selected_expert = bincount(route.expert_ids[selected], minlength=route.E)
    executed_expert = bincount(route.expert_ids[executed], minlength=route.E)
    selected_gate_mass = weighted_bincount(
        route.expert_ids[selected], route.gate_weights[selected], minlength=route.E
    )

    logical_group = aggregate(executed_expert, route.expert_to_logical_group)

    # Physical metrics require an explicit placement or resolved destination.
    device = aggregate(executed_expert, placement.expert_to_device)
    node = aggregate(device, placement.device_to_node)
    traffic = count_pairs(
        route.origin_device_for_executed_rows,
        placement.destination_device_for_executed_rows,
    )

    return {
        "scope_id": route.scope_id,
        "selected_expert_counts": selected_expert,
        "executed_expert_counts": executed_expert,
        "dropped_expert_counts": selected_expert - executed_expert,
        "selected_gate_mass": selected_gate_mass,
        "logical_group_counts": logical_group,
        "device_counts": device,
        "node_counts": node,
        "route_matrix": traffic,
        "remote_route_matrix": zero_diagonal(traffic),
    }

The article-local oracle also rejects same-shape failures: padding rows entering the denominator, selected counts copied into executed telemetry, gate mass called row load, logical groups bound to devices without placement evidence, local rows called network traffic, nonlinear per-rank summaries averaged instead of counts being reduced first, empty scopes called balanced, and latency inferred from a logical count matrix.

0x02 The Load Problem

Top-k routing gives each token a sparse active path:

E_t = TopK(router(h_t), k)

But the router does not automatically produce a balanced batch. If 60 percent of tokens choose one expert, that expert becomes the step limiter. Other experts may be idle, undertrained, or padded with zeros.

There are three separate imbalances:

Imbalance	Object	Symptom
expert imbalance	one expert receives too many tokens	overflow, padding, slow expert compute
device imbalance	one rank owns too many selected experts/tokens	synchronization waits on that rank
communication imbalance	one link or all-to-all partner gets too much traffic	latency spikes and poor scaling

This matters during training and serving. In training, underused experts may not learn enough. In serving, the slowest expert shard controls the MoE layer latency.

The Switch-style auxiliary load-balancing sketch is:

$$L_{aux} = \alpha E \sum_{e=1}^{E} f_e p_e$$

Renderer-safe fallback:

L_aux ~= alpha * E * sum_e f_e * p_e

where:

E       number of experts
f_e     fraction of tokens actually routed to expert e
p_e     mean router probability assigned to expert e
alpha   loss weight

The exact formula differs across systems, but the contract is stable: the training objective is pressured to avoid concentrating tokens into too few experts.

The three ledgers are related but not identical:

expert balance:
    n_e should be close to target tokens per expert

device balance:
    sum of tokens for experts on one rank should not dominate

communication balance:
    send and receive traffic should not hot-spot one link or rank

That separation matters because an apparently balanced expert distribution can still be device-imbalanced if the hot experts live on the same rank.

0x02A Why Hard Route Counts Cannot Train The Router Directly

The load problem is easy to state using the assignment indicator

$$A_{t,e}= \mathbf{1}[e\in\operatorname{TopK}(p_t,k)].$$

Two frequency conventions appear in MoE papers and implementations:

$$F_e=\frac{1}{Tk}\sum_{t=1}^{T}A_{t,e}, \qquad f_e=\frac{1}{T}\sum_{t=1}^{T}A_{t,e}=kF_e.$$

F_e is the fraction of all routed assignments received by expert $e$, so $\sum_e F_e=1$. f_e is selections per token for expert $e$, so $\sum_e f_e=k$. Both are valid. A derivation becomes wrong when it silently takes a formula written for one convention and substitutes statistics from the other.

With route-fraction convention $F$, uniform load is

$$Q=\left(\frac{1}{E},\ldots,\frac{1}{E}\right),$$

and the most direct penalty would be

$$L_{hard}=\frac{1}{2}\lVert F-Q\rVert_2^2.$$

This objective correctly measures imbalance in the forward pass. It does not by itself train the router because TopK and the indicator $A_{t,e}$ are discrete. Small changes to a non-boundary logit leave the selected expert IDs unchanged, so the derivative of the hard count is zero almost everywhere and undefined at selection boundaries.

The router does expose a differentiable object before top-k. Let

$$P_e=\frac{1}{T}\sum_{t=1}^{T}p_{t,e}$$

be the mean pre-selection probability or affinity proxy for expert $e$. For a softmax router, $P$ sums to one. $P$ is not the actual load: an expert can receive moderate probability on many tokens and still lose every top-k competition. It is useful because changing router logits changes $P$ smoothly.

A general straight-through construction makes this distinction explicit:

$$\widehat{F}=P+\operatorname{sg}(F-P),$$

where sg is stop-gradient. In the forward pass, $\widehat{F}=F$. During backward, the stop-gradient term contributes no derivative, so gradients flow as if $\widehat{F}=P$. This does not make top-k differentiable in a mathematical sense; it defines a surrogate gradient that pushes the differentiable score distribution in a direction associated with the measured hard load.

hard = route_histogram(topk_idx, E) / (T * k)  # measured load
prob = router_prob.mean(dim=0)                  # differentiable proxy
load_with_proxy_grad = prob + (hard - prob).detach()
loss = 0.5 * ((load_with_proxy_grad - 1 / E) ** 2).sum()

0x02B How The Auxiliary-Loss Family Evolved

A29’s loss history is valuable because each paper fixes a different defect in the previous measurement.

Adaptive local experts separated cooperation from competition. A loss on the error of the weighted sum lets experts compensate for one another: several experts can participate in every example and remain strongly coupled. A weighted sum of per-expert errors instead lets the gate reward an expert when that expert performs better than the mixture average on an example. This encourages responsibility regions. It establishes specialization, but it does not impose sparse execution or hardware balance.

Sparsely-gated MoE separated importance from load. Shazeer et al. defined expert importance as the sum of gate values over a batch. Equal importance is not equal token count. One expert could receive a few high-weight routes while another receives many low-weight routes. The paper therefore introduced both an importance penalty and a load penalty, using the probability that noisy top-k admits a token as a differentiable load estimate. The distinction is operational: output weighting affects model semantics, while route count determines expert batch shape and memory pressure.

For a batch $X$, the paper’s importance vector is

$$I_e(X)=\sum_{x\in X}G_e(x),$$

and its penalty is

$$L_{importance}(X) = w_{importance}\operatorname{CV}^{2}(I(X)).$$

Writing $S_I=\sum_e I_e$ makes the geometry explicit:

$$\operatorname{CV}^{2}(I) = \frac{\frac{1}{E}\sum_e(I_e-\bar I)^2}{\bar I^2} = \frac{E\sum_e I_e^2}{S_I^2}-1, \qquad \bar I=\frac{S_I}{E}.$$

If normalized gates keep $S_I$ fixed, minimizing CV$^2$ is equivalent to minimizing the squared concentration $\sum_e I_e^2$. This is an importance statement. It still does not reveal how many token rows each expert must execute.

The paper’s second estimator uses the noise distribution. Let

$$\mu_{t,e}=(x_tW_g)_e, \qquad \sigma_{t,e}=\operatorname{Softplus}((x_tW_{noise})_e),$$

and let $\tau_{t,e}$ be the sampled $k$-th largest noisy score after excluding expert $e$. Conditional on the sampled noise of the other experts, the probability that expert $e$ enters noisy top-k is

$$\pi_{t,e}^{admit} = \Phi\!\left(\frac{\mu_{t,e}-\tau_{t,e}}{\sigma_{t,e}}\right),$$

so the smooth expected-load vector and its penalty are

$$\widehat{Load}_e(X)=\sum_{t\in X}\pi_{t,e}^{admit}, \qquad L_{load}(X)=w_{load}\operatorname{CV}^{2}(\widehat{Load}(X)).$$

This is not the same surrogate as mean softmax probability. It estimates a noisy admission event relative to a top-k threshold. With $z_{t,e}=(\mu_{t,e}-\tau_{t,e})/\sigma_{t,e}$ and standard-normal density $\phi$, its direct score derivative is

$$\frac{\partial\pi_{t,e}^{admit}}{\partial\mu_{t,e}} = \frac{\phi(z_{t,e})}{\sigma_{t,e}}.$$

The derivative becomes largest near the admission boundary and shrinks for experts that are already almost surely selected or rejected. The conditional threshold also depends on the sampled competing scores; treating it as a global constant outside the paper’s estimator changes the algorithm.

GShard retained measured occupancy but needed a differentiable factor. If $c_e$ is the hard number of tokens received by expert $e$ out of $S$, a penalty involving only $(c_e/S)^2$ has no router gradient. GShard pairs a hard occupancy factor with a smooth mean gate factor $m_e$. The hard side tells the loss what actually happened; the smooth side supplies a path back to router parameters.

Algorithm 1 is group-local. For one routing group of $S$ tokens,

$$m_e=\frac{1}{S}\sum_{s=1}^{S}g_{s,e}, \qquad \ell_{aux}^{GShard} = \frac{1}{E}\sum_{e=1}^{E}\frac{c_e}{S}m_e.$$

The counter $c_e$ in that algorithm is incremented for the first-choice expert before the second-choice random-routing pass. Capacity admission changes the combine tensor, but the first-choice decision counter still advances when that expert is full. Those details make this a specific group-level top-2 objective, not a generic top-k loss. Its outer training objective is $L=L_{nll}+\kappa\ell_{aux}$; therefore its absolute numerical scale cannot be compared with Switch by looking only at the displayed inner expression.

Switch simplified the common form. For top-1 routing, a widely used sketch is

$$L_{Switch}=\alpha E\sum_{e=1}^{E}f_eP_e,$$

where both $f_e$ and $P_e$ have balanced value $1/E$. At balance,

$$E\sum_e\frac{1}{E}\frac{1}{E}=1,$$

so the leading factor keeps the balanced reference independent of expert count. If both hard routes and mean probability collapse onto one expert, the unweighted expression rises from $1/E$ to $1$, and the normalized expression rises from $1$ to $E$.

The dot product is not a generic distance between arbitrary vectors. It works as a training pressure because $f$ and $P$ are correlated views of the same router: the hard selections identify overloaded experts, while the differentiable probabilities provide the gradient path.

0x02B1 The Dot Product Has An Exact Router-Logit Gradient

Freeze the hard route fraction $f$ during backward, as top-k implementations do, and write a whole family of balance terms as

$$L_{bal}=\sum_{e=1}^{E}q_eP_e, \qquad P_e=\frac{1}{T}\sum_{t=1}^{T}p_{t,e}.$$

The coefficient $q_e$ contains the hard statistic and its loss weight. For Switch top-1, $q_e=\alpha E f_e$. Because

$$\frac{\partial L_{bal}}{\partial p_{t,e}} = \frac{q_e}{T},$$

and the softmax Jacobian is

$$\frac{\partial p_{t,e}}{\partial z_{t,j}} = p_{t,e}(\mathbf{1}[e=j]-p_{t,j}),$$

the exact logit gradient under the stopped hard assignment is

$$\boxed{ \frac{\partial L_{bal}}{\partial z_{t,j}} = \frac{p_{t,j}}{T} \left(q_j-\sum_{e=1}^{E}p_{t,e}q_e\right) }.$$

This formula explains the update rather than merely naming it. For token $t$, expert $j$ is pushed down when its hard-load coefficient $q_j$ is above the token’s probability-weighted average coefficient, and pushed up when it is below. A constant component in every $q_e$ cancels through the softmax Jacobian. Therefore a perfectly uniform hard-load vector produces zero balance gradient even though $\partial L/\partial P$ is not itself zero.

# hard statistics are measured in the forward pass and stopped in backward
q = build_weighted_hard_load_coefficients(route_ids).detach()  # [E]
P = router_prob[valid_token_mask].mean(dim=0)                   # [E]
loss = (q * P).sum()

# analytical per-token router-logit gradient
center = (router_prob * q).sum(dim=-1, keepdim=True)            # [T,1]
grad_logits = router_prob * (q - center) / valid_token_count

Two same-shape implementations are wrong:

differentiate through a fabricated continuous top-k histogram:
    claims a gradient the actual selection operator does not provide

detach P instead of the hard histogram:
    preserves the scalar forward loss but removes the router gradient

The lineage can be read as a measurement table:

Method	What it measures	Differentiable path	Remaining design choice
local-expert competition	per-expert task error	gate-weighted expert losses	no sparse execution guarantee
Shazeer importance loss	sum of gate values	gate values	equal importance can hide unequal counts
Shazeer load loss	expected noisy top-k admissions	probability of admission	depends on noise and threshold approximation
GShard	hard token fraction times mean gate	mean gate factor	capacity and group policy still matter
Switch	hard selected fraction times mean probability	mean probability	coefficient trades balance against task loss
DeepSeek V1/V2	expert, device, then communication aggregates	grouped affinity statistics	more objectives and coefficients to tune
DeepSeek V3	measured load controls selection bias	external feedback state	controller rate, saturation, and topology still matter

0x02C A Numeric Router-Loss Trace

Take $T=8$, $E=4$, and top-1 routing. Suppose hard route counts are [4,2,1,1]:

F = [0.500, 0.250, 0.125, 0.125]
Q = [0.250, 0.250, 0.250, 0.250]

and the router’s mean soft probabilities are

P = [0.400, 0.300, 0.200, 0.100].

The normalized Switch-style term, without $\alpha$, is

$$4\,(0.5\cdot0.4+0.25\cdot0.3+0.125\cdot0.2+0.125\cdot0.1)=1.25.$$

The balanced reference is 1.0. Expert 0 contributes the largest product because it is high in both measured load and differentiable preference. Gradient descent lowers probability pressure on that expert relative to colder experts. The loss does not move already-dispatched rows between buffers; it changes router parameters so later batches are more likely to produce a different distribution.

Three implementation assertions catch common mistakes:

assert topk_idx.shape == (T, k)
assert hard_counts.sum() == T * k
assert torch.isclose(route_fraction.sum(), torch.tensor(1.0))

For top-k routing, check the normalization convention before comparing a logged loss with a paper. A statistic that sums to $k$ needs a different leading factor from one that sums to one.

0x02D From V1 Two Scopes To V2 Three Exact Objectives

Let $N_r$ be the number of routed experts, $K_r$ the number selected per token, and $T$ the number of valid tokens in the balancing scope. DeepSeekMoE and DeepSeek-V2 normalize the hard expert frequency inside its definition:

$$f_i = \frac{N_r}{K_rT}\sum_{t=1}^{T} \mathbf{1}[i\in\mathcal{E}_t]$$

and use the mean pre-selection affinity

$$P_i=\frac{1}{T}\sum_{t=1}^{T}s_{i,t}.$$

Because every token contributes $K_r$ expert assignments,

$$\sum_i f_i=N_r, \qquad f_i^{balanced}=1, \qquad P_i^{balanced}=\frac{1}{N_r}.$$

The exact expert-level paper form is therefore

$$\mathcal{L}_{ExpBal} = \alpha_1\sum_{i=1}^{N_r}f_iP_i,$$

whose balanced value is $\alpha_1$. This is algebraically equivalent to placing $N_r/K_r$ outside a raw selections-per-token frequency. Mixing the normalized $f_i$ above with a second external $N_r/K_r$ factor double-normalizes the objective.

Expert balance alone is insufficient after experts are placed. Let $\mathcal{G}_d$ be the experts owned by device $d$. Device aggregates are constructed from expert statistics:

$$f_d^{\prime}=\frac{1}{|\mathcal{G}_d|}\sum_{i\in\mathcal{G}_d}f_i, \qquad P_d^{\prime}=\sum_{i\in\mathcal{G}_d}P_i,$$

and the paper’s device-level term is

$$\mathcal{L}_{DevBal} = \alpha_2\sum_{d=1}^{D}f_d^{\prime}P_d^{\prime}.$$

The average in $f_d^{\prime}$ is essential. Under equal-size device groups and uniform expert load, $f_d^{\prime}=1$, $P_d^{\prime}=1/D$, and the objective again has balanced value $\alpha_2$. A sum instead of an average changes the scale with experts per device. This device binding is explicit in the DeepSeekMoE/V2 paper’s training contract; it must not be inferred from an arbitrary checkpoint group ID in another model.

Communication balance is different again. Device-limited routing bounds each token to at most $M$ destination devices, but it does not guarantee equal receive-side traffic. The V2 term counts a token once for a destination even if it selects several experts on that destination:

$$f_d^{\prime\prime} = \frac{D}{MT}\sum_{t=1}^{T} \mathbf{1}[\text{token }t\text{ is sent to device }d], \qquad P_d^{\prime\prime}=\sum_{i\in\mathcal{G}_d}P_i,$$

with

$$\mathcal{L}_{CommBal} = \alpha_3\sum_{d=1}^{D}f_d^{\prime\prime}P_d^{\prime\prime}.$$

If every token reaches exactly $M$ devices and receive load is uniform, $f_d^{\prime\prime}=1$ and the balanced value is $\alpha_3$. Counting expert rows instead of the per-token destination indicator overweights tokens that choose two experts on the same device and silently changes the communication objective into another compute-load statistic.

The three losses can share one backward expression. Define

$$q_e = \alpha_1f_e +\alpha_2f_{d(e)}^{\prime} +\alpha_3f_{d(e)}^{\prime\prime}.$$

Then

$$\mathcal{L}_{ExpBal}+\mathcal{L}_{DevBal}+\mathcal{L}_{CommBal} = \sum_e q_eP_e,$$

and the boxed softmax-logit gradient from Section 0x02B1 applies unchanged. The terms differ in the hard coefficient projected onto each expert, not in the final softmax Jacobian.

A deterministic example makes the three projections auditable. Take four valid tokens, six experts, top-2 routing, three devices with two experts each, and at most two destination devices per token:

selected expert IDs:
    t0 [0,2]    t1 [0,3]    t2 [4,1]    t3 [0,5]

expert -> device:
    [0,0,1,1,2,2]

loss weights:
    alpha1 = 0.01    alpha2 = 0.02    alpha3 = 0.03

The exact ledgers are:

Projection	Hard statistic	Smooth probability aggregate	Weighted loss
expert	counts [3,1,1,1,1,1]; normalized $f=[2.25,.75,.75,.75,.75,.75]$	$P=[.2625,.1325,.1375,.125,.18,.1625]$	0.0114375
device compute	$f'=[1.50,.75,.75]$	$P'=[.395,.2625,.3425]$	0.0209250
destination communication	destination-token counts [4,2,2]; $f''=[1.50,.75,.75]$	$P''=P'$	0.0313875

The total balance loss is 0.06375, and the combined per-expert coefficient is

$$q=[0.0975,0.0825,0.045,0.045,0.045,0.045].$$

The executable oracle evaluates all 24 logit derivatives and checks them against central finite differences. Its maximum absolute error is $3.75\times10^{-7}$, and every token’s gradient row sums to zero as required by softmax shift invariance.

The derivation exposes why three terms exist:

expert objective:
    prevents routed experts from starving or dominating

device objective:
    balances aggregate expert compute on accelerator owners

communication objective:
    balances what ranks receive during token exchange

Collapsing them into one generic “load loss” hides different failure signals and different placement dependence.

0x02D1 Reduction Scope Is Part Of The Objective

The symbol $T$ is not permission to flatten every available token. It names the valid-token set over which the paper or implementation computes one statistic. A sequence-local loss, a microbatch-local loss, and a data-parallel global loss can receive identical tensors and produce different gradients.

Consider two sequences with top-1 routing over two experts:

sequence 0 hard routes: [expert 0, expert 0]
sequence 1 hard routes: [expert 1, expert 1]

each sequence:
    maximally imbalanced

flattened batch:
    exactly balanced

If the corresponding mean router probabilities are [0.85,0.15] and [0.15,0.85], the normalized Switch-style loss is 1.70 for each sequence but 1.00 after flattening both sequences. Neither number is an arithmetic bug. They answer different questions.

The pinned DeepSeek-V2 checkpoint configuration declares seq_aux=true. Its reference modeling file computes a normalized hard histogram [B,E], multiplies it by mean scores [B,E] for each sequence, sums experts, then averages over the batch. Its alternative seq_aux=false branch flattens tokens before constructing the statistic. The same file injects the scalar loss through a custom autograd identity:

forward(x, aux_loss):
    return x

backward(grad_output):
    return grad_output, ones_like(aux_loss)

The MoE activation is unchanged in the forward pass, while backward receives one unit of sensitivity for the auxiliary scalar. This is runtime integration of a training objective, not an inference-time route operation.

Every implementation must record:

token scope:
    sequence, microbatch, routing group, or globally reduced batch

mask:
    padding and invalid rows excluded before both hard and soft statistics

distributed order:
    aggregate linear counts/probability sums first; form nonlinear loss second

hard-state policy:
    selected routes, admitted routes, or expected noisy admissions

gradient policy:
    stopped hard statistic plus named differentiable proxy

0x02E Balance By Objective, Capacity, Or Selection Rule

Auxiliary loss is not the only design choice:

Intervention	What it controls directly	Cost or limitation
noisy token-choice routing	exploration in token-to-expert selection	does not guarantee balanced expert batches
auxiliary loss	expected future route distribution	coefficient can interfere with task specialization
capacity and dropping	maximum executable work per expert/device	discards selected computation or creates train/inference mismatch
Expert Choice	expert batch size because experts choose tokens	assignment count per token varies; autoregressive use needs a causality-safe construction
Soft MoE	continuous token-expert slot mixing	changes the routing and execution semantics rather than fixing sparse top-k in place
external bias controller	selection pressure from measured load	adds non-gradient state and controller dynamics

M6.1 compares token choice, Expert Choice, and Soft MoE at the local-layer level. M6.8 develops alternative conditional-compute axes. M6.2 keeps its main line on token-choice DeepSeek MoE so those alternatives remain genuine design branches rather than unexplained names.

0x03 Expert Parallel All-To-All

The phrase expert parallel all-to-all compresses two different contracts:

1. a logical transpose from token-major accepted routes to expert-major segments and back; and
2. a physical execution that binds logical experts to replicas, ranks, links, collectives, kernels, streams, and completion events.

M6.2 owns the first contract. M6.6 owns the second. Keeping them separate matters because the logical transpose is well-defined before any expert has a device address, while the realized network traffic is not.

Source-local lineage from A21. The figure establishes the two-transpose structure. It does not identify the physical collective, topology, placement, or kernel used by a production runtime.

0x03A The Unit Of Work Is An Accepted Route, Not A Token

After selection and capacity admission, let $R$ be the number of accepted routes. With top-$k$, one token can contribute several rows, so token identity alone cannot identify a route. M6.2 gives every accepted row this semantic record:

route_id
execution_scope_id
origin_partition_id
origin_token_row
selected_slot_id
logical_expert_id
combine_gate
hidden[d]

The pair (origin_partition_id, origin_token_row) identifies the token within the distributed input. selected_slot_id distinguishes that token’s multiple expert choices. route_id is the dense identity used by the permutation and its inverse. None of these fields says where the expert will execute.

The Loop 9 fixture has four token rows, hidden width three, four experts, and top-2 routing:

Route	Origin (partition,row,slot)	Expert	Gate	Hidden row
0	(0,0,0)	2	0.70	[1,2,3]
1	(0,0,1)	0	0.30	[1,2,3]
2	(0,1,0)	1	0.60	[4,5,6]
3	(0,1,1)	2	0.40	[4,5,6]
4	(1,0,0)	0	0.55	[7,8,9]
5	(1,0,1)	3	0.45	[7,8,9]
6	(1,1,0)	1	0.25	[10,11,12]
7	(1,1,1)	3	0.75	[10,11,12]

Rows 0 and 2 both have local token row zero, but they belong to different origin partitions. A combine path keyed only by origin_token_row would silently merge different tokens.

0x03B Count, Prefix, Permute

For logical expert $e$, count accepted routes:

$$c_e=\sum_{r=0}^{R-1}\mathbf{1}[e_r=e].$$

Exclusive prefix sums define expert segment boundaries:

$$o_0=0,\qquad o_{e+1}=o_e+c_e.$$

A stable permutation $\pi$ orders route IDs by (logical_expert_id, route_id). Its inverse satisfies

$$\pi^{-1}[\pi[j]]=j.$$

Renderer-safe fallback:

c[e] = count accepted routes whose logical_expert_id is e
o[0] = 0; o[e+1] = o[e] + c[e]
pi = stable_argsort(route_id, key=(logical_expert_id, route_id))
inverse_pi[pi[j]] = j

The toy has equal counts and therefore simple offsets, but a nontrivial permutation:

c          = [2,2,2,2]
o          = [0,2,4,6,8]
pi         = [1,4,2,6,0,3,5,7]
inverse_pi = [4,0,2,5,1,6,3,7]

expert-major segments:
E0: routes [1,4]
E1: routes [2,6]
E2: routes [0,3]
E3: routes [5,7]

The count vector sizes the segments; it does not replace the permutation. Equal counts do not preserve route order, and offsets do not recover origin identity.

0x03C Forward Transpose Identity

Let the teaching experts be scalar operators $E_0(h)=h$, $E_1(h)=2h$, $E_2(h)=3h$, and $E_3(h)=4h$. The logical execution is

$$Z_j=E_{e_{\pi[j]}}(H_{\pi[j]}),$$

followed by inverse permutation and weighted scatter-add:

$$Y_t=\sum_{r:\,origin(r)=t}g_r Z_{\pi^{-1}[r]}.$$

For token zero, routes 0 and 1 give

$$Y_0=0.7\,(3[1,2,3])+0.3\,(1[1,2,3])=[2.4,4.8,7.2].$$

The complete oracle result is:

Y0 = [ 2.40,  4.80,  7.20]
Y1 = [ 9.60, 12.00, 14.40]
Y2 = [16.45, 18.80, 21.15]
Y3 = [35.00, 38.50, 42.00]

It exactly matches direct token-major evaluation. The proof obligation is not that the buffers look similar; it is that each accepted route executes once, returns once, and is scattered with its own gate to its own origin.

0x03D Backward Is The Reverse Semantic Transpose

With upstream gradient $\bar Y_t$, route $r$ receives

$$\bar Z_r=g_r\bar Y_{origin(r)},$$

and the gate gradient is

$$\bar g_r=\langle\bar Y_{origin(r)},Z_r\rangle.$$

Expert backward produces route-input gradients and expert-parameter gradients. The inverse ownership path then sums route-input gradients by origin token. For an all-ones upstream gradient, the toy produces:

gate gradients by route = [18,6,30,45,24,96,66,132]
expert scalar gradients = [15,17.25,10.2,35.55]
token-gradient factors  = [2.4,2.4,2.35,3.5]

The direct and transposed token gradients agree exactly. A forward-only dispatch explanation is insufficient for training: the route permutation, origin record, gates, and expert segmentation are also backward state, whether saved or deterministically rematerialized.

0x03E Logical Bytes Are Three Ledgers

If the accepted hidden row has width $d$ and element size $b_h$, the logical activation payload over all accepted routes is

$$M_{act}=Rdb_h.$$

If expert outputs use element size $b_o$, the return payload is

$$M_{out}=Rdb_o.$$

For metadata fields $f$ with encoded widths $b_f$:

$$M_{meta}=R\sum_f b_f.$$

Renderer-safe fallback:

M_act  = accepted_routes * hidden_width * hidden_bytes
M_out  = accepted_routes * hidden_width * output_bytes
M_meta = accepted_routes * sum(metadata_field_bytes)

The fixture uses 20 metadata bytes per route and yields:

Ledger	All eight logical routes	Two remote rows under the teaching owner binding
activation rows	48 B	12 B
output rows	48 B	12 B
route metadata	160 B	40 B

The all-route ledger exists before placement. A remote-only ledger requires an external binding from logical experts to owner keys:

$$R_{remote}=\sum_r\mathbf{1}[origin\_owner(r)\ne bound\_owner(e_r)].$$

The teaching binding maps experts [0,1,2,3] to abstract owners [0,1,0,1] and gives route matrix [[3,1],[1,3]]. It exercises conservation and remote-row accounting, but those owner keys are not claims about devices or ranks.

For the earlier realistic example, $N_{tokens}=4096$, $k=2$, $d=4096$, and two-byte elements give 64 MiB of all-route activation rows and about 128 MiB for activation plus output rows. That is not automatically network traffic: local routes may not cross a link, while padding, metadata, count exchange, alignment, protocol headers, topology, contention, and retransmission can add other costs. Logical bytes cannot predict latency.

0x03F Packed, Fixed-Capacity, And Blocked Representations

The transpose identity does not mandate one physical layout:

Representation	Logical shape	Advantage	Cost or constraint
fixed capacity	[E,C,d] plus mask	regular addresses and static maxima	padding or dropped capacity slots
packed ragged	[R,d] plus counts/offsets	stores accepted rows only	prefix sums, metadata, variable segments
block-sparse	expert-aligned blocks plus occupancy	can match block kernels	block rounding and fragmentation
replicated local path	selected experts may have several replicas	can reduce remote pressure	requires versioned replica selection and combine identity

M6.2 chooses packed_accepted_routes only for the executable semantic oracle. M6.6 decides whether a runtime realizes that relation with padded collectives, variable exchange, grouped GEMM, block-sparse kernels, replication, or a hybrid.

0x03G Executable Logical Transpose

def logical_expert_transpose(hidden, accepted_routes, experts):
    # Route identity is frozen before any physical owner is selected.
    records = enumerate_routes(
        hidden,
        accepted_routes,
        fields=("route_id", "origin_partition", "origin_row",
                "selected_slot", "logical_expert", "gate"),
    )

    counts = bincount(records.logical_expert, minlength=num_experts)
    offsets = exclusive_prefix_sum(counts)
    permutation = stable_argsort(records, key=("logical_expert", "route_id"))
    inverse = invert_bijection(permutation)

    packed_hidden = gather(records.hidden, permutation)
    packed_expert = gather(records.logical_expert, permutation)
    packed_output = run_logical_expert_segments(
        packed_hidden, packed_expert, offsets, experts
    )

    route_output = gather(packed_output, inverse)
    output = weighted_scatter_add(
        route_output,
        records.combine_gate,
        key=(records.origin_partition, records.origin_row),
    )
    return output, records, counts, offsets, permutation, inverse

The runtime must reject a non-bijective permutation, count/offset mismatch, duplicate route identity, nonfinite gate, missing origin partition, or a combine record that no longer names the route that produced its output.

0x03H The Typed M6.6 Handoff

M6.2 emits semantic state; M6.6 enriches it with physical execution state:

M6.2 must provide	M6.6 may add	M6.2 must not claim
contract and execution-scope versions	physical slot and replica ID	that a logical expert is one physical rank
route/origin/slot/expert identities	destination rank and placement version	which transport or collective runs
gate, dtypes, and metadata schema	send counts and offsets	which grouped kernel is selected
expert counts, offsets, permutation, inverse	transport, kernel, completion handle	latency or tokens/s from logical counts
representation and saved/rematerialized state	measured bytes and time	that replication preserves identity without a version fence

The placement version must remain stable from dispatch through combine. Every semantic route must bind to one compatible replica, send and receive counts must conserve accepted rows, and return metadata must recover the original route. Replication changes physical execution multiplicity, not logical expert semantics.

The checked-in Loop 9 oracle executes the eight-route forward and backward identities, verifies the expert and owner permutations, reconciles all-route and remote-only byte ledgers, and rejects physical runtime fields on the M6.2 side. It proves the semantic transpose. It does not benchmark a collective or choose a deployment topology.

0x04 DeepSeekMoE: Fine-Grained And Shared Experts

DeepSeekMoE does not begin from communication. Its first question is whether conventional experts are specializing well enough to justify their parameter cost.

0x04A The V1 Motivation: Knowledge Mixing Versus Redundancy

A conventional top-k MoE can fail in two opposite directions.

Knowledge mixing occurs when one routed expert must serve too many unrelated token regions. If there are only a few large experts, a selected expert can receive code syntax, factual names, multilingual morphology, and general reasoning features. The expert is conditionally active, but its internal parameters still represent a broad mixture rather than a narrow reusable capability.

Knowledge redundancy occurs when different routed experts independently relearn common transformations. Every domain needs some syntax, token composition, and general language processing. If those common components are copied into many routed experts, total parameters grow without providing equally distinct capacity.

These are not contradictions. One expert can mix several domains while several experts duplicate the same general feature. The design problem has two axes:

specialization:
    routed experts should acquire focused, non-identical capability

sharing:
    common computation should not be redundantly relearned in every route

Return to A29’s software-team analogy. A team with only “front end” and “back end” roles may still be too coarse: the back-end expert handles storage, stream processing, authentication, and distributed systems. Splitting that role into narrower specialists increases compositional choice. But all specialists may still need database access conventions. A shared database role can centralize that common work instead of requiring each specialist to recreate it.

The analogy does not prove that a neural expert learns a named skill. It explains the architectural pressure: increase the number of smaller routed units to create more combinations, then isolate an always-on path that can absorb common transformations.

0x04B Fine-Grained Segmentation Preserves A Cost Ledger

Let a conventional routed bank contain $N$ experts with intermediate width $d_{ff}$ and activate $K$ experts. Ignoring biases, each SwiGLU-style expert has parameter order

$$P_{expert}\approx 3d\,d_{ff}.$$

Split every expert into $m$ narrower experts, each with width $d_{ff}/m$:

$$N_{fine}=mN, \qquad d_{ff,fine}=\frac{d_{ff}}{m}.$$

The routed-bank parameter count remains approximately constant:

$$mN\left(3d\frac{d_{ff}}{m}\right) =N(3dd_{ff}).$$

If the token activates $mK$ fine experts, active routed parameters also remain approximately constant:

$$mK\left(3d\frac{d_{ff}}{m}\right) =K(3dd_{ff}).$$

The change is therefore not “more experts for free.” It exchanges expert width for expert count while attempting to hold total routed parameters and selected routed compute near the baseline. Router logits, expert metadata, dispatch fanout, and kernel granularity do not remain constant.

The combinatorial choice set grows from

$$\binom{N}{K} \quad\text{to}\quad \binom{mN}{mK}.$$

That count describes possible expert subsets, not guaranteed learned skills. Many combinations may be rare, correlated, or functionally redundant. The architectural hypothesis is that a larger composition space gives optimization more ways to assemble narrow transformations for a token.

The paper makes the combinatorial change concrete with $N=16$, $K=2$, and $m=4$:

$$\binom{16}{2}=120, \qquad \binom{64}{8}=4{,}426{,}165{,}368.$$

The second number is not a count of learned capabilities. It is the number of possible eight-expert subsets before shared-expert isolation. Optimization, correlated scores, placement constraints, and finite data can make the realized route family much smaller.

ordinary:
    N wider routed experts, select K

fine-grained:
    mN narrower routed experts, select mK

approximately preserved:
    routed expert parameters
    selected routed FFN width

increased:
    router output width
    possible expert combinations
    route metadata and potential placement fanout

There are now two coordinate systems, and they must not be silently merged:

Coordinate system	Stored quantities	What it answers
paper construction	coarse $(N,K,d_{ff})$, split factor $m$, isolated units $K_s$	how one conceptual MoE bank is transformed while preserving its expert ledger
released checkpoint	routed count $N_r$, routed top-k $K_r$, shared count $N_s$, expert width $d_e$	what tensors and route axes actually exist in one artifact

Under the paper construction, the bridge after isolation is

$$N_r=mN-K_s, \qquad K_r=mK-K_s, \qquad N_s=K_s.$$

The released V1 config stores $(N_r,K_r,N_s)=(64,6,2)$. If the bridge is applied algebraically, it implies $mN=66$ and $mK=8$, but it does not identify $m$, $N$, or $K$ separately. The config contains no counterfactual coarse checkpoint from which those factors could be recovered. Figure 2’s $16\rightarrow64$ teaching construction and the released $64$ routed plus $2$ shared units are therefore related ideas, not interchangeable manifests.

0x04C Shared Expert Isolation Changes What Must Be Routed

DeepSeekMoE then isolates $K_s$ fine-grained expert units as shared experts. In the paper’s construction, those units are always active, while the router chooses the remaining active units from indices $K_s+1$ through $mN$. Retaining the paper’s layer residual, the exact structural form is

$$h_t^l = u_t^l +\sum_{s=1}^{K_s} \operatorname{FFN}_s(u_t^l) +\sum_{i=K_s+1}^{mN}g_{i,t}\operatorname{FFN}_i(u_t^l),$$

with

$$g_{i,t}= \begin{cases} s_{i,t}, & s_{i,t}\in\operatorname{TopK} \left(\{s_{j,t}\}_{j=K_s+1}^{mN},\,mK-K_s\right),\\ 0, & \text{otherwise}. \end{cases}$$

The released software graph uses a different boundary. DeepseekMoE.forward returns only the expert contribution

$$M(u_t)=E^{shared}_{fused}(u_t) +\sum_{e\in R_t}g_{t,e}E_e^{routed}(u_t), \qquad |R_t|=K_r,$$

and DeepseekDecoderLayer.forward later computes $h_t=residual_t+M(u_t)$. The paper equation and code are consistent only when this module boundary is retained. Adding a residual inside the MoE module while also preserving the decoder residual would double it.

The total number of active fine-grained units remains $mK$ when the shared and routed paths are counted together. Shared experts are not free residual helpers. They consume parameters and active FLOPs for every token.

The intended training effect is a division of representational labor:

shared path:
    common transformations useful across many route regions

routed path:
    residual specialization that varies by token

Nothing in the architecture guarantees that the decomposition occurs cleanly. If the shared path is too strong, routed experts may contribute little. If it is too weak, routed experts may continue duplicating common knowledge. Balance coefficients, data, expert width, initialization, and optimization determine the learned outcome.

Active expert work must therefore include both terms:

active routed path:
    |R_t| * P_fine_expert

always-on shared path:
    K_s * P_fine_expert

active expert work:
    (|R_t| + K_s) * P_fine_expert

If shared experts are omitted from the active ledger, per-token work is underestimated.

0x04C1 Why Several Shared Experts Can Be One Fused SwiGLU

The V1 reference code does not instantiate two separate shared DeepseekMLP objects. It multiplies the expert intermediate width by n_shared_experts and creates one wider MLP. This is an exact representation of a sum of independent bias-free SwiGLU experts, not an approximation.

For shared unit $s$, write

$$E_s(x)=D_s\left[\operatorname{SiLU}(G_sx)\odot U_sx\right],$$

where $G_s,U_s\in\mathbb{R}^{d_e\times d}$ and $D_s\in\mathbb{R}^{d\times d_e}$. Stack the gate and up matrices by rows and the down matrices by columns:

$$G_{fused}=\begin{bmatrix}G_1\\ \vdots\\G_{N_s}\end{bmatrix}, \quad U_{fused}=\begin{bmatrix}U_1\\ \vdots\\U_{N_s}\end{bmatrix}, \quad D_{fused}=\begin{bmatrix}D_1&\cdots&D_{N_s}\end{bmatrix}.$$

Elementwise activation and multiplication preserve the channel blocks, while the final horizontal concatenation performs their sum:

$$D_{fused}\left[\operatorname{SiLU}(G_{fused}x)\odot U_{fused}x\right] =\sum_{s=1}^{N_s}E_s(x).$$

The Loop 5 executable oracle evaluates both sides on two vectors and obtains maximum absolute error $6.94\times10^{-18}$. This equivalence depends on independent channel blocks and the stated bias-free SwiGLU form. It does not justify fusing experts that have cross-expert normalization, interaction, or distinct nonlinear state.

Source: DeepSeekMoE, Figure 2. The panels hold expert parameter count and activated compute approximately fixed while moving from conventional top-2 routing to finer routed experts and then shared-expert isolation. The figure establishes the architecture construction, not a universal guarantee that experts become semantically clean specialists.

0x04C2 What The Paper Measures, And What It Does Not Prove

The architecture’s motivation is specialization; its evidence is empirical. In controlled ablations with equal total and activated expert parameters, the paper reports that isolating a shared expert usually improves over its GShard baseline and that splitting to 32 and then 64 total experts improves normalized aggregate performance. It also reports Pile losses of $1.808$, $1.806$, and $1.811$ for one, two, and four shared experts, respectively. Those close values motivated a $1{:}3$ shared-to-activated-routed ratio when scaling, rather than a claim that one count is universally optimal.

Source: DeepSeekMoE, Figure 3, recovered from the paper source bundle and rasterized from the original vector PDF. Values are normalized independently per benchmark by the best compared result. The figure supports the paper’s ablation claim under its training setup; it does not establish semantic labels for experts or guarantee monotonic gains at other scales.

The paper supplies two further interventions: disabling top routed experts causes a larger degradation than in a matched GShard comparison, and replacing the shared expert with one additional routed expert changes Pile loss from $1.808$ to $2.414$ at the same stated compute. These are source-reported sensitivity tests. They support lower redundancy and a useful shared path in that trained model, not a theorem that routed experts always specialize or shared experts always learn only universal knowledge.

0x04D A Released DeepSeekMoE Checkpoint Trace

A29 includes a module-tree and code walkthrough. To make the implementation contract reproducible, this section pins the public deepseek-ai/deepseek-moe-16b-base checkpoint and its remote modeling file rather than treating every DeepSeek version as one codebase.

The published checkpoint configuration reports:

hidden_size             = 2048
moe_intermediate_size   = 1408
n_routed_experts        = 64
n_shared_experts        = 2
num_experts_per_tok     = 6
first_k_dense_replace   = 1
moe_layer_freq          = 1
scoring_func            = softmax
norm_topk_prob          = false

This means the first decoder layer remains dense, and subsequent FFN positions use the MoE module at the configured frequency. The two shared expert units are represented in code as one DeepseekMLP whose intermediate width is 2 * 1408 = 2816; they are not routed through the 64-way gate.

The persistent matrix shapes for one MoE layer are therefore:

router.weight                         [64, 2048]

each routed expert:
    gate_proj.weight                  [1408, 2048]
    up_proj.weight                    [1408, 2048]
    down_proj.weight                  [2048, 1408]

fused shared path:
    gate_proj.weight                  [2816, 2048]
    up_proj.weight                    [2816, 2048]
    down_proj.weight                  [2048, 2816]

Ignoring biases because these projections are bias-free, the exact per-layer parameter ledger is:

State	Derivation	Parameters
one routed expert	$3\cdot2048\cdot1408$	$8{,}650{,}752$
resident 64-expert routed bank	$64\cdot8{,}650{,}752$	$553{,}648{,}128$
six active routed experts per token	$6\cdot8{,}650{,}752$	$51{,}904{,}512$
two-unit always-on shared path	$2\cdot8{,}650{,}752$	$17{,}301{,}504$
resident routed plus shared expert state	previous resident terms	$570{,}949{,}632$
active routed plus shared expert path	previous active terms	$69{,}206{,}016$
router weight	$64\cdot2048$	$131{,}072$

The checkpoint’s active expert width is $(6+2)\cdot1408=11{,}264$, close to its dense-layer intermediate width $10{,}944$ by a ratio of $1.0292$. This is a checkpoint-specific width comparison. It is not the same baseline as the paper’s conceptual conventional expert of width $m d_e$, and it excludes attention, normalization, embeddings, residual traffic, and routing overhead.

For hidden states H: [B,S,2048], the gate path is:

flatten H                         -> [T,2048], T = B*S
linear router weight [64,2048]   -> logits [T,64]
softmax                           -> scores [T,64]
topk(k=6)                         -> expert_ids [T,6], weights [T,6]
hard one-hot histogram           -> expert occupancy [64]
mean soft scores                 -> probability proxy [64]

The released configuration sets norm_topk_prob=false, so one must not silently impose selected-axis normalization when reproducing this checkpoint. Other DeepSeek versions use different score and scaling contracts. “Top-k gate” is not enough information to reconstruct numerical behavior.

For the concrete fixture $B=1$, $S=3$, $T=3$, the training branch has the following tensor trace:

hidden_states                       [1,3,2048]
flattened hidden                    [3,2048]
router logits / softmax scores      [3,64]
top-k IDs and selected weights      [3,6]
repeat_interleave route rows        [18,2048]
expert outputs before combine       [3,6,2048]
weighted routed reduction           [3,2048]
fused shared output                 [3,2048]
MoE module output                   [1,3,2048]
decoder residual result             [1,3,2048]

The public training branch makes the route expansion explicit:

# Structural sketch of the released 16B modeling file.
topk_idx, topk_weight, aux_loss = gate(hidden_states)
x = hidden_states.reshape(T, d)
flat_topk_idx = topk_idx.reshape(T * k)

routed_x = x.repeat_interleave(k, dim=0)       # [T*k,d]
routed_y = empty_like(routed_x)

for expert_id, expert in enumerate(experts):
    mask = flat_topk_idx == expert_id
    routed_y[mask] = expert(routed_x[mask])

routed_y = routed_y.view(T, k, d)
y_route = (routed_y * topk_weight[..., None]).sum(dim=1)
y_moe = y_route.view(B, S, d) + shared_experts(identity)

# DeepseekDecoderLayer owns the residual, outside DeepseekMoE.
y_block = residual + y_moe

The inference branch uses a different data movement strategy: sort flattened expert IDs, derive token indices, run each expert on its contiguous token group, multiply by the corresponding gate weights, then scatter-reduce outputs back to token order. The mathematics is the same mixture; the memory layout is different.

The executable tiny oracle evaluates token-major training layout and expert-major grouped layout independently. Their routed outputs agree exactly in the fixture, while selected weight sums remain $0.70$ and $0.75$ to catch accidental renormalization. Layout equivalence is an output invariant; it does not imply identical allocation, launch, communication, or accumulation order.

The visual deliberately leaves physical expert placement unbound. It proves the stored and logical forward contract only.

Four assertions distinguish semantic from layout correctness:

assert expert_ids.shape == (T, 6)
assert gate_weights.shape == (T, 6)
assert routed_x.shape == (T * 6, 2048)
assert y_route.shape == (T, 2048)
assert shared_experts.intermediate_size == 2 * 1408
assert decoder_output.shape == hidden_states.shape

The released single-model modeling path is not a production expert-parallel training implementation. Repeating rows and looping over experts explains ownership and combination, but a distributed runtime must replace that local organization with pack, all-to-all, grouped expert execution, return exchange, and route-aware reduction. M6.6 owns that physical realization.

0x04E What Changes From V1 To V3

Version / family	Primary problem being addressed	Router and balance contract	Capacity / communication contract
DeepSeekMoE / V1 lineage	knowledge mixing and redundancy in conventional experts	softmax-style affinity plus expert- and device-level auxiliary balancing	fine-grained routed experts plus always-on shared expert isolation
DeepSeek-V2	fine-grained experts can fan each token across too many devices	adds communication-level balance to expert/device objectives	at most $M$ devices per token; device-level token dropping during training
DeepSeek-V3	conventional auxiliary pressure can damage task optimization	sigmoid affinity; top-k uses affinity plus control bias; original selected affinities form output gates	group/node-limited routing; report states no token dropping and adds deployment balancing strategies

The progression has three different owners:

V1 changes the representation units.
V2 changes route locality and receiving-side balance.
V3 changes affinity, group selection, and the main load controller.

This is a routing/load-balancing article, not a full DeepSeek architecture article. MLA remains in M5.8.

0x05 Restricted Routing: V2 Devices And V3 Nodes

Fine-grained routing improves choice, but it can increase device fanout. If one token selects experts across many devices, communication grows and synchronization becomes harder.

The hard part is not writing topk twice. It is identifying what the first top-k ranges over, how that logical unit is bound to hardware, and which unit later consumes capacity. DeepSeek-V2 has three evidence surfaces:

Evidence surface	What it establishes	What it does not establish
V2 report	target experts span at most $M$ devices; select $K$ experts within those devices; training uses device capacity factor $1.0$, lowest-affinity dropping, and an approximately 10% no-drop sequence path	complete tie handling, a serialized placement manifest, capacity rounding, or private trainer code
released checkpoint and modeling file	contiguous logical expert groups, max-per-group eligibility, top-k expert selection, selected gate scaling, and tensor shapes	that group $g$ is physical device $g$, or that inference code reproduces report-level dropping
production runtime	expert-to-rank placement, packing, collectives, replication, overlap, and kernels	not recoverable from the report or checkpoint alone; M6.6 owns it

That separation is not editorial caution. It changes whether the same code actually bounds physical fanout.

0x05A The Report Contract: Restrict Devices Before Expert Top-k

Let $p(e)$ map routed expert $e$ to a physical device. The V2 report says that, for each token, the router first retains $M$ devices containing high-affinity experts and then selects $K$ experts from that restricted domain. The report does not give a complete executable device aggregation and tie rule. The following teaching realization uses the maximum affinity on each device, matching the released checkpoint’s later max-per-logical-group rule without claiming this is the only valid report implementation:

$$q_{t,d}=\max_{e:p(e)=d}s_{t,e}$$ $$\mathcal{D}_t=\operatorname{TopM}_{d}(q_{t,d}), \qquad \mathcal{A}_t=\{e:p(e)\in\mathcal{D}_t\}$$ $$\mathcal{R}_t=\operatorname{TopK}_{e\in\mathcal{A}_t}(s_{t,e}), \qquad |\{p(e):e\in\mathcal{R}_t\}|\le M$$

The last inequality follows from construction: every selected expert belongs to one of the $M$ eligible devices. It does not follow from taking global top-k and measuring its fanout afterward.

def device_limited_topk(scores, expert_to_device, k, max_devices):
    # Explicit teaching choice: score each device by its best local expert.
    device_score = {
        d: max(scores[e] for e in experts_on(d))
        for d in unique(expert_to_device)
    }
    allowed_devices = topk(device_score, max_devices)
    eligible_experts = [
        e for e in experts
        if expert_to_device[e] in allowed_devices
    ]
    return topk(scores, k, candidates=eligible_experts)

For the Loop 6 oracle, token t2 has these highest candidates:

Expert	Score	Device	Global top-3	Device-limited top-3
E0	0.50	D0	yes	yes
E2	0.49	D1	yes	yes
E4	0.48	D2	yes	no: third device
E1	0.10	D0	no	yes

Global top-3 has score sum $1.47$ and touches three devices. With $M=2$, the restricted route has score sum $1.09$, touches two devices, and accepts E1 instead of the higher-scoring E4. The $0.38$ gap is not a universal quality loss; it is a concrete proof that locality restriction and global score maximization are different objectives.

DeepSeek-V2 reports $D=8$ expert-parallel devices, $M=3$ destination devices per token, and $K=6$ activated routed experts. It reports that $M\ge3$ performed roughly in line with unrestricted top-k in its experiments. That is source-reported empirical evidence under its setup, not a guarantee for other expert counts, placements, or workloads.

0x05B The Released Checkpoint Implements Logical Group-Limited Greedy

The public deepseek-ai/DeepSeek-V2 checkpoint configuration makes the full-model routing dimensions concrete:

hidden_size            = 5120
moe_intermediate_size  = 1536
n_routed_experts       = 160
n_shared_experts       = 2
num_experts_per_tok    = 6
n_group                = 8
topk_group             = 3
topk_method            = group_limited_greedy
scoring_func           = softmax
routed_scaling_factor  = 16.0
norm_topk_prob         = false
seq_aux                = true
aux_loss_alpha         = 0.001

For each token, the released modeling file computes router logits in FP32, applies softmax over 160 routed experts, reshapes scores into eight groups, takes the maximum score inside each group, retains three groups, masks experts in the other five groups, and finally takes six experts from the surviving candidate set.

For $T$ flattened tokens, the tensor trace is:

State	Shape	Ownership
router logits and FP32 softmax scores	$[T,160]$	one score per routed expert
grouped scores	$[T,8,20]$	contiguous logical groups, not proven devices
group maxima	$[T,8]$	eligibility statistic
selected group IDs	$[T,3]$	logical candidate restriction
eligible-expert mask	$[T,160]$	masks five groups
selected expert IDs and weights	$[T,6]$ each	routed demand and combine coefficients

# Pinned structural trace of the released V2 modeling file.
scores = softmax(linear(x.float(), router_weight.float()))  # [T,160]

grouped = scores.view(T, 8, 20)
group_score = grouped.max(dim=-1).values                    # [T,8]
selected_group = topk(group_score, 3).indices               # [T,3]

eligible = expand_group_mask(selected_group, groups=8, experts=160)
eligible_score = scores.masked_fill(~eligible, 0.0)
topk_weight, topk_idx = topk(eligible_score, 6)             # [T,6]

# This checkpoint does not normalize selected weights.
topk_weight = topk_weight * 16.0

This trace sharpens three distinctions:

1. Group restriction precedes expert top-k. It is not equivalent to taking global top-6 and checking how many devices were touched afterward.
2. Group score and expert score have different jobs. The group maximum chooses eligible groups; original expert scores choose experts inside them.
3. Selection and combination still need the checkpoint’s scaling rule. With norm_topk_prob=false, selected weights are scaled by 16.0 rather than normalized to sum to one.

There is a fourth invariant: logical locality is not physical locality without a binding. In the oracle’s aligned placement, each two-expert logical group maps to one device, so selecting two groups bounds fanout to two devices. Under a crossed placement, the same logical selected experts touch three physical devices for token t1. Tensor shapes and selected IDs remain valid while the intended communication bound fails.

logical group rule:
    expert_id -> contiguous group_id

physical locality rule:
    expert_id -> device_id from a placement manifest

equivalence requires:
    every retained logical group maps to the intended bounded hardware unit

The config does not persist that placement relation. Loop 6 therefore executes the public group rule and an illustrative physical placement as separate functions.

The public inference model proves group selection, not the private pretraining admission controller.

0x05C Device Capacity Needs An Admission Unit

After locality selection there are at least two useful counts. Let $X_{t,e}=1$ when token $t$ selects expert $e$, and let $Z_{t,d}=1$ when at least one selected expert for token $t$ lives on device $d$:

$$Z_{t,d}=\mathbb{1}\!\left[\sum_{e:p(e)=d}X_{t,e}>0\right]$$

Then device $d$ receives

$$H_d=\sum_t Z_{t,d} \quad\text{hidden-state destination rows, while}\quad R_d=\sum_t\sum_{e:p(e)=d}X_{t,e} \quad\text{local expert routes.}$$

$H_d$ and $R_d$ are not interchangeable. One hidden row can be transmitted once to a device and then feed several selected local experts. In the executable toy:

selected destination-token rows = 8  -> average 2 per device
selected expert-route rows      = 12 -> average 3 per device

The V2 report states that each device receives an average computational budget with capacity factor $1.0$, then loses lowest-affinity tokens until it reaches that budget. It does not publish enough trainer detail to prove whether the stored admission unit is a destination-token row, an expert route, a padded work slot, or how a non-integral mean is rounded. A faithful implementation must name those choices.

The Loop 6 oracle uses a deliberately explicit, non-canonical teaching policy:

$$C_{\mathrm{teach}}=1.0\times\frac{\sum_{t,d}Z_{t,d}}{D}=\frac{8}{4}=2$$

The fixture is exactly divisible so it does not smuggle in a rounding convention. For each selected destination row, priority is the maximum selected local expert affinity:

$$a_{t,d}=\max_{e:p(e)=d,\,X_{t,e}=1}s_{t,e}.$$

Protected-sequence rows are admitted first; remaining rows are ordered by $a_{t,d}$ and admitted up to capacity. If a destination row is dropped, every selected routed expert contribution for that token on that device is removed. Other devices’ routed contributions, the always-on shared path, and the decoder residual remain.

def teaching_device_capacity(selected_routes, sequence_id, protected, D):
    destination_rows = coalesce_by_token_and_device(selected_routes)
    # Exact only for this divisible fixture. A real runtime must freeze rounding.
    capacity = len(destination_rows) // D

    admitted = []
    dropped = []
    for device in range(D):
        rows = rows_for_device(destination_rows, device)
        protected_rows = [r for r in rows if sequence_id[r.token] in protected]
        ordinary_rows = [r for r in rows if r not in protected_rows]
        ordinary_rows.sort(key=lambda r: r.max_local_affinity, reverse=True)

        slots = max(0, capacity - len(protected_rows))
        admitted += protected_rows + ordinary_rows[:slots]
        dropped += ordinary_rows[slots:]

    return admitted, dropped

The report does not specify what happens if protected rows alone exceed capacity. The oracle’s fixture avoids that case and rejects any claim that it established a universal answer.

0x05D A Full Admission Trace

The selected destination counts are [2,3,2,1], so only D1 exceeds the teaching capacity of two:

Device	Destination row	Priority	Protected?	Decision
D0	t0 -> {E0,E1}	0.30	no	admit
D0	t2 -> {E0,E1}	0.50	yes	admit
D1	t0 -> {E2}	0.24	no	drop
D1	t1 -> {E2}	0.40	no	admit
D1	t2 -> {E2}	0.49	yes	admit
D2	t1 -> {E4,E5}	0.39	no	admit
D2	t3 -> {E4,E5}	0.60	yes	admit
D3	t3 -> {E6}	0.58	yes	admit

The training trace executes 11 of 12 selected expert routes. Token t0 keeps its D0 routes but loses E2 on D1; because selected gates are not silently renormalized, its routed gate mass changes from $0.79$ to $0.55$. The token itself is not deleted:

$$M_{\mathrm{drop}}(u_{t0})= E^{\mathrm{shared}}_{\mathrm{fused}}(u_{t0}) +0.30E_0(u_{t0})+0.25E_1(u_{t0}),$$

and the decoder still adds the residual $u_{t0}$. This is why “drop token” is dangerously imprecise: the policy drops selected routed computation on one receiving device, not the token row from the Transformer.

Approximately 10% of V2 training sequences are source-reported as never dropped. The toy protects one of two sequences only to exercise that branch; its 50% toy fraction is not a model hyperparameter. The V2 report also states that training uses dropping for acceleration while evaluation does not drop tokens. Evaluation therefore executes all 12 selected routes in this toy.

0x05E Why Affinity Priority Appears Here

DeepSeek-V2 cites Scaling Vision with Sparse Mixture of Experts for its priority idea. That paper’s Batch Prioritized Routing sorts vision-patch tokens by a router-derived priority before attempting bounded expert-buffer allocation. Its primary Figure 4 shows that decreasing capacity retains a progressively smaller subset of patches.

The figure is useful lineage evidence, not a DeepSeek implementation diagram. V-MoE uses per-expert buffers and defines token priority from routing weights; V2 states a per-device budget and does not publish the same full allocation loop. The transferable design choice is sort meaningful work before bounded admission, not “copy the V-MoE buffer algorithm unchanged.”

0x05F Report, Checkpoint, And Trainer Must Remain Separate

The model file also exposes sequence-level auxiliary-loss code, but it is not the complete distributed pretraining system described in the V2 report. The report’s expert-, device-, and communication-level objectives and device token-dropping policy are training-system contracts. Their absence from a checkpoint-oriented modeling file should not be interpreted as evidence that the paper did not use them. Conversely, copying a paper equation into this inference model does not reproduce the original trainer.

paper/report evidence:
    training objectives, device capacity, dropping policy, topology intent

checkpoint config/modeling evidence:
    tensor dimensions, group-limited selection, gate scaling,
    shared/routed module structure

production runtime evidence:
    dispatch implementation, expert placement, collectives, replication

Keeping those evidence layers separate prevents a common source-audit error: merging paper pseudocode, Hugging Face remote code, and a production serving engine into one imaginary canonical implementation.

0x05G V3 Changes The Restricted Unit

DeepSeek-V3 changes the placement unit from individual devices to nodes. Its report describes a node score derived from the sum of leading affinity scores among experts placed on that node. A teaching form is:

$$q_{t,n}=\sum_{e\in\operatorname{TopExpertsOnNode}(t,n)}s_{t,e}$$ $$\mathcal{N}_t=\operatorname{TopNodes}(q_t,M_n), \qquad \mathcal{E}_t=\operatorname{TopK}\{s_{t,e}:\operatorname{node}(e)\in\mathcal{N}_t\}$$

Experts within a node can use faster local links; crossing nodes uses the slower network. The report’s source-reported training setup places routed experts for a layer across 64 GPUs in eight nodes and limits each token to at most four nodes. V2 device-limited routing and V3 node-limited routing are therefore related locality patterns, not identical algorithms. Loop 7 derives V3’s sigmoid affinity, group scoring, selection bias, original-score gates, and no-drop controller as one contract.

0x06 DeepSeek-V3: Sigmoid Affinity, Control Bias, And Selected Gates

DeepSeek-V3 does not merely replace a softmax with a sigmoid. It changes four contracts at once: the coordinate system used for affinity, the score used to choose groups and experts, the score used to weight selected outputs, and the state machine used to control load. Compressing those four contracts into “sigmoid top-k with a bias” hides exactly where a shape-correct implementation can cease to be the released model.

The source surfaces must remain separate:

Evidence surface	What it freezes	What it does not freeze
DeepSeek-V3 report v2	sigmoid affinity, selected-affinity normalization, selection-only bias, node-limited intent, controller schedule, tiny sequence loss, no-drop system statement	public checkpoint tensor layout, reference-code branch details, production placement
671B config at 9b4e978	$d=7168$, $E=256$, $K=8$, eight logical groups, four selected groups, route scale $2.5$, FP8 model declaration	physical node ownership, trainer controller, production kernels
reference model.py at 9b4e978	exact inference score, group, mask, expert-top-k, normalization, and cast order	pretraining bias update, production dispatch, deployment balancing
production system	topology binding, redundant experts, dispatch/combine kernels, scheduling	not published by the reference forward

0x06A One Router Produces Three Non-Interchangeable Coordinates

For token state $h_t\in\mathbb{R}^{d}$ and router row $w_e\in\mathbb{R}^{d}$, first form a logit and an independent sigmoid affinity:

$$z_{t,e}=h_t^\top w_e, \qquad s_{t,e}=\sigma(z_{t,e})=\frac{1}{1+e^{-z_{t,e}}}.$$

The word independent matters. A softmax probability contains a denominator over every expert,

$$p_{t,e}=\frac{e^{z_{t,e}}}{\sum_j e^{z_{t,j}}},$$

so changing an unrelated $z_{t,j}$ changes $p_{t,e}$. By contrast,

$$\frac{\partial s_{t,e}}{\partial z_{t,j}}= \begin{cases} s_{t,e}(1-s_{t,e}), & e=j,\\ 0, & e\ne j. \end{cases}$$

The loss-free-balancing paper reports that this local score geometry makes an additive per-expert controller easier to tune than the corresponding softmax case. This is evidence for the design choice, not a theorem that sigmoid is always a better router.

The persistent FP32 correction bias $b_e$ then creates a second coordinate:

$$\tilde{s}_{t,e}=s_{t,e}+b_e.$$

s and s_tilde answer different questions:

s[t,e]       model affinity used to weight a selected expert
s_tilde[t,e] load-controlled score used to decide eligibility and selection
b[e]         persistent control state, not token state and not an output gate

Sigmoid does not make $s_t$ a probability vector. In the Loop 7 fixture, the twelve affinities for t0 sum to $5.74$. Normalization happens only after the selected expert IDs are known.

0x06B Group Eligibility Comes Before Expert Top-k

The report says that a token reaches at most $M=4$ nodes. A node score sums the highest $K_r/M=8/4=2$ expert affinities on that node. The pinned inference forward realizes the analogous logical-group calculation on the corrected selection scores. Let $G_g$ be the expert IDs in logical group $g$:

$$q_{t,g}=\sum_{r=1}^{2} \operatorname{TopValue}_r\{\tilde{s}_{t,e}:e\in G_g\}.$$

Retain four groups, mask all experts outside them, and only then choose eight experts:

$$\mathcal{G}_t=\operatorname{TopKGroups}(q_t,4),$$ $$\mathcal{E}_t=\operatorname{TopK} \{\tilde{s}_{t,e}:e\in\bigcup_{g\in\mathcal{G}_t}G_g\},8.$$

For $T$ flattened token rows, the released 671B reference executes:

hidden rows                  [T,7168]
router weight                [256,7168]
original sigmoid affinity    [T,256]
FP32 correction bias         [256]
corrected selection score    [T,256]
group view                   [T,8,32]
top-two score per group      [T,8,2]
group score                  [T,8]
selected logical groups      [T,4]
eligible expert mask         [T,256]
selected routed experts      [T,8]
selected combine weights     [T,8]

This is a logical contiguous-ID grouping contract. The config does not bind group 0 to physical node 0. The report’s physical node rule and the checkpoint’s logical group rule coincide only after an external placement manifest provides that binding; M6.6 owns it.

The report paragraph calls the inputs node affinities. The pinned reference uses corrected scores for its top-two logical-group sum. Those are compatible evidence surfaces only if named precisely; neither authorizes silently using a group maximum or a sum over all 32 experts.

0x06C A Counterexample For Group Score, Bias, And Locality

Use four groups of three experts, keep two groups, and then keep four experts. For t0, the original and corrected top-two group scores are:

Group	Original affinities	Bias	Top-two original	Top-two corrected
$G_0$	$[0.90,0.10,0.10]$	$[0,0.55,0.55]$	$1.00$	$1.55$
$G_1$	$[0.60,0.59,0.58]$	$[0,0,0]$	$1.19$	$1.19$
$G_2$	$[0.70,0.69,0.01]$	$[0,0,0]$	$1.39$	$1.39$
$G_3$	$[0.50,0.49,0.48]$	$[0,0,0]$	$0.99$	$0.99$

Without bias, groups $G_2,G_1$ survive. With the correction bias, groups $G_0,G_2$ survive and the selected expert IDs are $[0,6,7,1]$. The control state therefore changes routing, including admission of expert 1 whose model affinity is only $0.10$.

A second token proves that “some representative group score” is not enough:

Rule on corrected group	Scores for $G_0,G_1,G_2,G_3$	Selected groups
released top-two sum	$[1.51,1.87,1.83,1.79]$	$G_1,G_2$
maximum expert	$[0.95,0.94,0.92,0.90]$	$G_0,G_1$
sum all experts	$[2.07,1.88,1.84,1.80]$	$G_0,G_1$

The exact group rule excludes expert 0 even though its corrected score $0.95$ is the global maximum. Restricted routing is therefore not global top-k with a post-hoc communication annotation; it changes the candidate set.

0x06D The Output Gate Returns To Original Affinity

Selection has used $\tilde{s}$, but the released forward gathers $s$. With route scale $\rho=2.5$,

$$g_{t,e}= \rho\, \frac{s_{t,e}} {\sum_{j\in\mathcal{E}_t}s_{t,j}}, \qquad e\in\mathcal{E}_t, \qquad \rho=2.5.$$

For t0, selected original affinities are $[0.90,0.70,0.69,0.10]$ and sum to $2.39$. The gates are therefore:

Expert	Original $s$	Corrected $\tilde{s}$	Correct output gate
0	$0.90$	$0.90$	$0.941423$
6	$0.70$	$0.70$	$0.732218$
7	$0.69$	$0.69$	$0.721757$
1	$0.10$	$0.65$	$0.104603$

The gates sum to $2.5$, not $1$, because normalization is followed by route_scale. If corrected scores were wrongly reused as output weights, expert 1’s gate would become $0.552721$: the tensor shape and selected IDs would still look valid, but the model output would change substantially.

The full routed-plus-shared block is:

$$y_t=h_t+E_{\mathrm{shared}}(h_t) +\sum_{e\in\mathcal{E}_t}g_{t,e}E_e(h_t).$$

The shared expert is outside the routed top-k and outside the correction-bias controller. Conflating the shared path with a ninth selected routed expert changes both load accounting and output scale.

0x06E The Bias Is A Delayed Controller State

The auxiliary-loss-free paper defines route count $c_e$, mean count $\bar c$, load error $\epsilon_e=\bar c-c_e$, and a sign update:

$$b_e^{(r+1)}=b_e^{(r)}+\gamma_r\, \operatorname{sign}(\bar c^{(r)}-c_e^{(r)}).$$

An overloaded expert has $c_e>\bar c$, so its bias decreases. An underloaded expert’s bias increases. The update occurs after the batch. Using final counts from the current sequence to reroute earlier tokens in that sequence would violate the causal contract discussed by the loss-free paper.

def finish_training_step(route_counts, bias, gamma):
    target = mean(route_counts)
    for expert, count in enumerate(route_counts):
        error = target - count
        bias[expert] += gamma * sign(error)
    return bias  # state for later routing, not a gradient update

The two-token fixture selects route counts [1,1,0,1,1,0,2,2,0,0,0,0]; the mean is $8/12=2/3$. At $\gamma=0.001$, experts 6 and 7 decrease by $0.001$, while every zero-load expert increases by $0.001$. This is a deterministic controller transition, not a derivative of language-model loss with respect to $b$.

DeepSeek-V3 reports a schedule that is easy to lose in a one-line sketch:

0 through 14.3T consumed tokens: gamma = 0.001
final 500B tokens (to 14.8T):     gamma = 0.0

Freezing $\gamma$ does not remove the learned correction-bias state from the checkpoint. It stops the controller from changing that state.

Source: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, Figure 1. The paper has only arXiv v1. Measured expert load updates a bias used by later top-k decisions; the feedback path is control state rather than a language-model-loss gradient.

0x06F “Auxiliary-Loss-Free” Still Has A Small Sequence Loss

DeepSeek-V3 mainly uses batch-wise bias control, but its report retains a complementary loss for each sequence of $T$ tokens:

$$\mathcal{L}_{\mathrm{Bal}}= \alpha\sum_{e=1}^{N_r}f_eP_e, \qquad \alpha=10^{-4},$$ $$f_e=\frac{N_r}{K_rT}\sum_{t=1}^{T} \mathbf{1}\!\left[e\in \operatorname{TopK}(s_t,K_r)\right],$$ $$P_e=\frac{1}{T}\sum_{t=1}^{T} \frac{s_{t,e}}{\sum_{j=1}^{N_r}s_{t,j}}.$$

Notice the coordinates in the report equation: $f_e$ uses top-k of the original affinity, and $P_e$ normalizes affinity across all routed experts. This small differentiable sequence objective is not the batch-wise sign controller and does not use the checkpoint’s corrected group-selection path.

In the fixture, the unscaled $\sum_e f_eP_e$ is $1.604256$, so the scaled term is approximately $1.6043\times10^{-4}$. That numeric value only checks the equation on the toy; it is not a reported training loss.

The precise description is therefore:

main batch-level load control:
    measured route counts -> delayed selection-bias update

complementary sequence guard:
    tiny differentiable loss with alpha = 0.0001

Source: DeepSeek-V3 Technical Report, Figure 9. The domain-conditioned heatmaps are source-reported evidence that batch-wise control permits different specialization patterns; they are not a universal causal guarantee.

0x06G No Dropping Is A Reported System Outcome, Not A Router Theorem

The V3 report states that the training system drops no tokens and that its inference deployment also avoids dropping through specific balancing strategies. The score equations alone do not prove this outcome:

sigmoid + correction bias:
    influences route distribution

no token dropping:
    additionally depends on batch scale, placement, redundant experts,
    communication capacity, scheduling, and deployment controls

This is the sharp boundary from Loop 6. V2 reports a device-capacity drop path; V3 reports a controller and deployment system that avoid dropping. Importing V2’s capacity admission into V3, or claiming that every bias-balanced router is automatically dropless, would conflate two different contracts.

0x06H Paper, Checkpoint, Reference Forward, And Runtime Do Not Collapse

The released inference gate stores its correction bias in an nn.Parameter container so checkpoint state can load normally. That storage class does not prove optimizer ownership. The report and loss-free paper define a measured-load update, while the reference inference file contains no training loop.

Likewise, the reference MoE computes locally owned experts and combines partial outputs with dist.all_reduce. This is executable documentation, not evidence that production DeepSeek serving uses the same per-expert loop or collective.

paper:
    training intent, controller update, node-limited and no-drop claims

checkpoint/config:
    dimensions, logical groups, route scale, persistent bias state

reference inference code:
    one exact forward realization

production runtime:
    physical ownership, dispatch/combine, redundant experts, kernels

0x06I Exact Pinned Inference Trace

At commit 9b4e9788e4a3a731f7567338ed15d3ec549ce03b, the gate reduces to:

scores = linear(x, gate_weight)
scores = scores.sigmoid()                         # [T,256]
original_scores = scores

scores = scores + correction_bias                # FP32 selection state
grouped = scores.view(T, 8, 32)
group_scores = grouped.topk(2, dim=-1).values.sum(dim=-1)
selected_groups = group_scores.topk(4, dim=-1).indices

scores = mask_unselected_groups(scores, selected_groups)
expert_ids = scores.topk(8, dim=-1).indices

weights = original_scores.gather(1, expert_ids)
weights = weights / weights.sum(dim=-1, keepdim=True)
weights = weights * 2.5
return weights.type_as(x), expert_ids

The final cast returns weights in the hidden-state dtype after score selection and normalization. The bias itself is declared FP32. Reading dtype: fp8 in the config as “every router operation and controller state is FP8” would erase these source-visible precision boundaries.

The Loop 7 executable oracle reproduces the reduced operator, negative-tests one-field contract mutations, and pins the exact config and model digests. It does not execute 671B weights or claim production equivalence.

0x06J Runtime Telemetry For Scaled MoE

Load control must be observable beyond the training objective:

Counter	What it catches
tokens_per_expert	hot and idle experts; controller input
tokens_per_logical_group	group eligibility skew
nodes_per_token	physical topology-policy violation after binding
selection_bias_min/max	controller saturation or persistent overload
bias_delta_sign_changes	oscillatory controller behavior
sequence_balance_loss	within-sequence collapse hidden by batch averages
dropped_routes	violation of the reported V3 no-drop deployment contract
dispatch_bytes_per_rank	locality and congestion outcome
max_expert_time / mean_expert_time	straggler barrier amplification

Those counters are not interchangeable. tokens_per_expert is the controller’s measured variable; selection_bias_min/max is controller state; route churn is a discontinuous top-k response; PPL is a quality outcome; rank-level bytes and expert time are physical runtime consequences. A dashboard that compresses all of them into one “balance score” cannot distinguish a stable controller from a balanced but slow or quality-degraded system.

0x06J1 The Router Is A Delayed, Quantized Feedback System

The sign rule from 0x06E is easiest to reason about as a sampled-data feedback loop. Batch $r$ is routed using state $b^{(r)}$. Only after its assignments have been counted can the controller create $b^{(r+1)}$:

$$\begin{aligned} c_e^{(r)} &=\sum_{t\in\mathcal{B}_r} \mathbf{1}[e\in\mathcal{E}_t(b^{(r)})],\\ \bar c^{(r)}&=\frac{1}{N_r}\sum_e c_e^{(r)},\\ b_e^{(r+1)} &=b_e^{(r)}+u\, \operatorname{sign}(\bar c^{(r)}-c_e^{(r)}). \end{aligned}$$

Three facts explain why the update rate $u$ cannot be chosen from the equation alone.

1. One-batch delay. The controller acts on historical load. Increasing $u$ does not remove that delay; it increases the size of the next correction.
2. Integer measurements. Route counts move in whole assignments. For small batches, one token can be a large relative load change.
3. Discontinuous actuation. Bias changes are continuous, but top-k membership changes only when a corrected score crosses another score. A tiny update may do nothing for several batches; one larger update can flip many routes at once.

This is why “bias went down for a hot expert” is only the sign of the feedback, not a stability proof. The closed loop also depends on affinity margins, input correlation, top-k, group restriction, batch scope, and how quickly the learned router itself moves under language-model gradients.

0x06J2 Update Rate Trades Lag Against Route Oscillation

The loss-free paper tests $u\in\{10^{-4},10^{-3},10^{-2}\}$ on its 1B setup. It reports slow early convergence for $10^{-4}$, chooses $10^{-3}$, and observes late-stage bias fluctuation and worse load balance for $10^{-2}$. The three validation perplexities are close, so the figure matters: a final scalar does not expose when balance was poor or oscillatory during training.

Source-reported $u$	Validation PPL	Reported behavior	Safe conclusion
$10^{-4}$	9.51	slow early balance convergence	step size can be too small relative to score margins
$10^{-3}$	9.50	chosen configuration	suitable in this paper’s 1B experiment
$10^{-2}$	9.51	late bias fluctuation; balance deteriorates	larger correction can overshoot a discrete route boundary

Source: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, update-rate figure. Curves are MaxVio_batch averaged over 100 neighboring steps in the paper’s 1B experiment. They do not establish $10^{-3}$ as a universal rate or reproduce DeepSeek-V3’s 671B training run.

The Loop 8 executable toy isolates the threshold mechanism. Eight fixed tokens route top-1 between two experts. Their initial counts are $[6,2]$, hence

$$\operatorname{MaxVio} =\frac{\max_e c_e-\bar c}{\bar c} =\frac{6-4}{4}=0.5.$$

With lower-index tie breaking, the same sign controller produces:

u = 0       : 6:2 -> 6:2 -> 6:2          no corrective motion
u = 0.0001  : 6:2 -> 5:3 -> 4:4          first balanced at step 6
u = 0.001   : 6:2 -> 4:4 -> 4:4          first balanced at step 1
u = 0.01    : 6:2 <-> 0:8                tail route churn = 0.75

This deliberately reduced model does not reproduce the paper’s curves. It proves a narrower mechanism: the same delayed sign rule can stall below score margins or overshoot across many top-k boundaries. A production sweep therefore needs trajectories, not only final MaxVio.

0x06J3 Sign And Proportional Updates Are Different Controllers

Replacing the sign step with an error-proportional step gives

$$\Delta b_e^{\mathrm{sign}} =u\,\operatorname{sign}(\bar c-c_e), \qquad \Delta b_e^{\mathrm{prop}} =u(\bar c-c_e).$$

The sign rule gives every overloaded expert the same negative step and every underloaded expert the same positive step, regardless of violation magnitude. The proportional rule moves a severely imbalanced expert farther. It can correct large errors faster, but its effective gain also grows with count scale. Changing microbatch size, top-k, or reduction scope therefore changes the size of a proportional update unless its error is normalized.

The paper’s source-reported ablation illustrates why “lower MaxVio” and “better model” remain separate columns:

Rule	$u$	Validation PPL	MaxVio_global
sign	$10^{-3}$	9.50	0.044
proportional	$10^{-2}$	9.53	0.028
proportional	$10^{-3}$	9.51	0.036
proportional	$10^{-4}$	9.51	0.040

The proportional variants slightly improve the reported balance metric but do not improve validation PPL; the paper retains the sign rule. That result is an ablation under one model/data/training contract, not proof that sign control is always preferable.

0x06J4 Additive And Multiplicative Bias Move Different Boundaries

An additive controller selects with $s_i+b_i$. For two experts where $s_i>s_j$, expert $j$ replaces $i$ when

$$s_j+b_j>s_i+b_i \iff b_j-b_i>s_i-s_j.$$

A positive multiplicative controller selects with $s_i b_i$. Its crossing is

$$s_jb_j>s_ib_i \iff \frac{b_j}{b_i}>\frac{s_i}{s_j}, \qquad b_i,b_j>0.$$

Additive control compares an absolute affinity gap. Multiplicative control compares a ratio and must define a positive initialization, reported as one in the paper rather than zero. Two pairs can have the same absolute gap but very different ratio thresholds:

affinities 0.9 vs 0.8: additive gap 0.1; ratio threshold 1.125
affinities 0.2 vs 0.1: additive gap 0.1; ratio threshold 2.0

The paper reports similar balance but slightly worse PPL for its multiplicative variants: at $u=10^{-3}$, PPL is 9.52 and MaxVio_global is 0.036, versus 9.50 and 0.044 for additive sign control. This does not mean the additive controller dominates every parameterization; it explains why the paper chose it.

0x06J5 Sigmoid And Softmax Change Controller Geometry

Loop 7 established V3’s independent sigmoid affinities. Their local derivative is

$$\frac{\partial s_i}{\partial z_j} =\mathbf{1}[i=j]s_i(1-s_i).$$

For a softmax gate $p_i=\exp(z_i)/\sum_j\exp(z_j)$,

$$\frac{\partial p_i}{\partial z_j} =p_i(\mathbf{1}[i=j]-p_j).$$

Every softmax score depends on the common denominator, so changing one logit moves all probability coordinates. An additive post-score bias can therefore cross rankings in a score geometry whose gaps also depend on other experts. The paper’s appendix consequently uses its proportional update variant for the softmax experiment. It reports PPL/MaxVio_global of 9.604/0.937 for its loss-controlled baseline and 9.599/0.027 for loss-free control. Those numbers support the paper’s scoped comparison; they do not license silently replacing V3’s sigmoid operator with softmax.

0x06J6 Auxiliary Pressure Is A Second Control Channel

There are two distinct ways to change routing during training:

differentiable channel:
    auxiliary objective -> router-logit gradient -> future affinity geometry

external feedback channel:
    measured hard routes -> bias-state update -> future selection boundary

If both are active, they can cooperate, oppose, or mask each other. Increasing auxiliary coefficient $\alpha$ changes the router’s learned scores and task gradient. Increasing $u$ changes selection pressure without adding the same task-loss gradient. The tiny V3 sequence-wise loss adds a third, scoped guard; it is not an interchangeable substitute for the batch controller.

Source: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, auxiliary-coefficient tradeoff figure. The blue points sweep the paper’s loss coefficient; the red star is its loss-free setup. Perplexity and MaxVio_global are separate measured outcomes, not one optimization axis.

A useful one-factor study freezes model initialization, data order, affinity function, route policy, capacity/drop policy, and batch scope, then varies only one of:

Sweep	Keep fixed	Measure together
$u$	sign rule, additive bias, affinity family	time-to-balance, tail MaxVio, churn, bias span, PPL
controller rule	comparable error normalization and rate grid	update magnitude, oscillation, balance, PPL
bias mode	initialization and selection policy	crossing frequency, bias range, balance, PPL
$\alpha$	controller off, or controller state explicitly frozen	router gradients, balance, PPL
sequence guard	batch controller and routing frozen	per-sequence collapse, domain skew, PPL

Without those controls, an apparently better setting may only have seen a larger effective batch, different capacity admission, or a different physical placement.

0x06J7 Computation Batch Changes What Balance Can Be Observed

The loss-free paper defines its computation batch in expert-parallel training as

$$B_{\mathrm{comp}} =B_{\mathrm{micro}}\times D_{\mathrm{EP-data}},$$

where the second factor counts data-parallel replicas participating in one expert-parallel computation step. The controller must reduce counts over the same scope whose work will be dispatched; updating from rank-local counts while execution sees globally combined tokens is a different controller.

Even before stochastic variation, integer counts impose a granularity floor. For $n$ route assignments over $E$ experts, the best possible maximum load is $\lceil n/E\rceil$, giving

$$\operatorname{MaxVio}_{\mathrm{integer\ floor}} =\frac{\lceil n/E\rceil-n/E}{n/E}.$$

For $E=4$, five assignments have a floor of $0.6$; eight assignments can be perfectly divisible; eighteen assignments have a floor of $1/9$. Larger scope can lower this counting floor and average independent variation. It does not guarantee balance: correlated domains, repeated sequences, group eligibility, capacity, and placement can preserve or amplify skew.

Source: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts, computation-batch figure. In the paper’s 1B and 3B experiments, loss-free balance improves as the computation batch grows, while the loss-controlled baseline plateaus. The figure does not prove that expert parallelism by itself improves balance or latency.

The phrase “naturally compatible with expert parallelism” must therefore be read as a source-reported scaling observation. EP can enlarge the count scope, but it also introduces the all-to-all and straggler costs handed to M6.6.

0x06J8 Sensitivity Must Preserve Evidence Class

The evidence layers answer different questions:

Evidence	What it can establish	What it cannot establish
algebra	crossing conditions, delay, count granularity	convergence on learned nonstationary affinities
Loop 8 toy	one deterministic lag/overshoot counterexample	paper-curve or V3 training reproduction
loss-free 1B/3B ablations	source-reported PPL and balance under stated setups	671B causal attribution or universal hyperparameters
V3 report	chosen controller schedule and system outcome	private trainer details not disclosed in the report
production telemetry	realized loads, churn, drops, time, bytes	counterfactual quality without controlled experiments

The Loop 8 oracle pins the source tables, executes sign and proportional toy controllers, derives additive and multiplicative crossing thresholds, and computes integer MaxVio floors. It deliberately fails closed if a paper result is relabeled as toy output or if a teaching result is promoted to V3 evidence.

0x06J9 A Release Sweep Needs Joint Gates

A balance-controller release should record, per layer and over time:

quality:
    training loss, held-out PPL, task metrics

logical routing:
    selected counts, gate mass, MaxVio_global, MaxVio_batch,
    per-sequence skew, group eligibility, route churn

controller state:
    update rate, error rule and normalization, bias min/max/span,
    sign reversals, saturation, frozen-controller interval

execution:
    dropped routes, send/receive rows and bytes per rank,
    expert-time max/mean, collective wait, padding or redundant work

Acceptance needs bounds on all four families. Lower MaxVio with worse PPL is not a clean win; lower global skew with high per-batch or per-device skew does not remove stragglers; a stable bias trace does not prove useful routing if the router has collapsed to nearly equal affinities. Controller, model, and runtime must be reconciled without collapsing their evidence.

Physical communication and kernel realization remain owned by M6.6. Loop 9 converts the logical route relation into a typed expert-parallel transpose handoff without pretending that logical groups already name physical nodes.

0x06K DeepSeek-V3 And Llama 4 Choose Different Sparse Contracts

A29 compares DeepSeek-V3 with Llama 4. The useful comparison is not which model is “better.” It is which routing and layer-placement decisions each public source commits to.

Meta’s official Llama 4 overview reports that Llama 4 Maverick alternates dense and MoE layers. Each MoE layer contains 128 routed experts plus one shared expert, and each token uses the shared expert plus one routed expert. It reports 400B total parameters and 17B active parameters for Maverick. Those are architecture facts for that model, not a general top-1 MoE recipe.

DeepSeek-V3 reports 256 routed experts, eight selected routed experts, a shared path, three initial dense FFN layers before its MoE stack, group-limited routing, and 671B total with 37B activated parameters per token.

Choice point	DeepSeek-V3	Llama 4 Maverick	Consequence to reason about
routed bank	256 routed experts	128 routed experts	stored capacity and router width differ
routed experts per token	8	1	composition breadth versus route/dispatch fanout
shared computation	always-on shared expert path	one shared expert per MoE layer	both must include shared work in active ledgers
layer placement	first three FFNs dense, then MoE according to V3 architecture	alternating dense and MoE layers	early/general features and runtime cadence differ
public routing detail	sigmoid affinity, correction bias, group-limited selection	official overview specifies top-1 routed selection but not an equivalent V3 controller	do not invent undocumented balance semantics
modality	text-model V3 report	native multimodal early-fusion model	modality routing and data are confounders, not MoE-only variables

Top-1 routing can simplify routed composition and reduce the number of expert destinations per token. It does not make communication, load balance, or shared-expert compute disappear. Top-8 routing permits a broader mixture of fine-grained transformations, but raises route metadata and potential fanout. Alternating MoE layers reduces how often the sparse runtime is entered; using MoE after only a short dense prefix exposes more layers to routed capacity.

The comparison should stop there unless training objectives, topology, data, precision, and serving engines are controlled. Benchmark or latency claims from these systems cannot be attributed to routed-expert count alone.

0x06L Compare Contracts, Not Model Names

Switch, Mixtral, DeepSeekMoE, DeepSeek-V2, DeepSeek-V3, and Llama 4 are often placed in one table headed “MoE.” That label is too coarse to predict even the logical forward pass. A useful comparison must freeze at least seven axes:

1. score coordinate:       softmax probability, sigmoid affinity, or undisclosed
2. eligibility operator:   all experts, group-limited experts, or topology-limited experts
3. selection operator:     top-1, top-2, top-6, top-8, plus any shared path
4. combine coordinate:     full probability, selected renormalization, route scale
5. admission policy:       fixed capacity, report-only dropping, no fixed rejection, no drop
6. training controller:    auxiliary loss, device/communication losses, delayed bias
7. architecture cadence:   every block, dense prefix, alternating dense and sparse layers

Checkpoint coordinates, evidence surfaces, and runtime ownership form an eighth axis. A paper can specify a trainer that a released inference file does not contain. A checkpoint can freeze router dimensions without freezing the private training policy. A readable reference forward can prove tensor semantics without proving the production collective or kernel.

0x06L1 The Route Operator Changes Before The Expert Executes

The following table records the public or pinned contract used here. “Not disclosed” is a result, not a blank to fill from a neighboring family.

Family	Score and eligibility	Selected routed path	Combine weights
Switch	softmax over a routing group; token-choice top-1	1	selected full-softmax probability; not selected-axis renormalization
Mixtral 8x7B	FP32 softmax over all 8 experts	2	selected top-2 probabilities renormalized to one
DeepSeekMoE V1 16B	softmax over all 64 routed experts	6 + 2 shared	selected full-softmax mass, no selected renormalization, route scale 1
DeepSeek-V2	FP32 softmax; 8 logical groups, choose 3 by group maximum, then expert top-6	6 + 2 shared	original selected softmax mass times route scale 16, without selected renormalization
DeepSeek-V3	sigmoid affinity plus FP32 selection bias; 8 groups, choose 4 by sum of top-2 corrected scores, then expert top-8	8 + 1 shared	original selected sigmoid affinities renormalized, then route scale 2.5
Llama 4 Maverick	exact score and gate rule not disclosed by the pinned public source used here	1 + 1 shared	not disclosed by that source

The order of operations matters. V3 uses corrected scores to decide which experts execute, then returns to original affinities to decide how much their outputs contribute. V2 restricts the eligible domain before expert top-k. V1 does neither. These operators can return different experts and different output scales even when a tensor named topk_idx has the same rank and integer dtype.

0x06L2 Selected, Executed, And Residual Paths Are Different Ledgers

Loop 10 reuses the exact eight-token probability matrix from the M6.1 contract oracle. On that common input:

Switch teaching contract:
    selected routes = 8
    executed routes = 6
    capacity-dropped token ids = [4, 7]
    outer residual path survives for every token

Mixtral pinned-forward contract:
    selected routes = 16
    executed routes = 16
    no fixed-capacity rejection appears in the inspected forward

This is not a quality comparison. It is a mechanism counterexample. “Top-1” does not imply one executed expert for every token when capacity can reject the expert branch. “Top-2” does not imply fixed-capacity overflow when the inspected forward packs every selected route. The residual path also prevents “dropped expert route” from meaning “dropped token from the Transformer.”

0x06L3 V1, V2, And V3 Change Different Objects

The DeepSeek sequence is easiest to remember as a sequence of changed contracts, not as three increasingly large expert counts.

Transition	Representation	Locality and admission	Load control	Numerical route contract
V1	64 fine-grained routed experts, choose 6; 2 shared	global routed eligibility in the pinned forward	expert- and device-level auxiliary objectives in the paper	softmax; retain selected full probability mass; scale 1
V1 -> V2	160 routed experts, still choose 6; 2 shared	choose 3 of 8 groups; report adds device capacity 1, affinity-priority training drop, protected sequences, evaluation no-drop	expert-, device-, communication-, and sequence-scope pressure	FP32 softmax; no selected renormalization; scale 16
V2 -> V3	256 routed experts, choose 8; 1 shared	choose 4 of 8 groups by a different group statistic; report states no token dropping	delayed selection-bias controller plus a small sequence loss	sigmoid affinity; bias only for selection; original selected affinities renormalized; scale 2.5

V1 and V2 both select six routed experts, but that equality does not preserve the router. For token t2 in the Loop 6 reduced fixture:

global top-3 experts      = [0, 2, 4]
device-limited top-3      = [0, 2, 1]
selected score sum        = 1.09
teaching route scale      = 16
scaled selected gate mass = 17.44

The locality restriction changes the third expert. The scale does not silently renormalize selected mass to one. For V3 token t0, the reduced reference operator instead selects groups [0,2], then experts [0,6,7,1]; their original affinities sum to 2.39, but normalization followed by route scale makes the final combine weights sum to 2.5. These are different numerical contracts, not formatting variants of one top-k equation.

0x06L4 Gate Mass Has No Universal Invariant

A common implementation mistake asserts that every selected gate row must sum to one. The compared families disprove that as a universal rule:

Switch: selected top-1 weight is its full-softmax probability
Mixtral: selected top-2 weights renormalize to 1
V1 reduced fixture: selected sums are [0.70, 0.75]
V2 reduced fixture: 1.09 * 16 = 17.44
V3 reduced fixture: selected normalization * 2.5 = 2.5
Llama 4 Maverick: exact public gate rule remains undisclosed here

The valid assertion belongs to a pinned model contract. It cannot be inferred from k, from the number of experts, or from a neighboring model family.

0x06L5 Equal Unit Counts Still Do Not Mean Equal FLOPs

For a teaching batch of $T=8$ tokens at one MoE layer, define only a semantic unit ledger:

$$R_{units}=T k_r, \qquad S_{units}=T k_s, \qquad U_{units}=R_{units}+S_{units}.$$

Here $k_r$ is the number of selected routed expert units and $k_s$ is the number of always-on shared units. These are invocation counts before physical placement, not FLOPs:

Family	Routed rows	Shared-unit evaluations	Total expert-unit evaluations
Switch	8 selected; 6 execute in the capacity fixture	0	8 selected, 6 executed
Mixtral	16	0	16
DeepSeekMoE V1	48	16	64
DeepSeek-V2	48	16	64
DeepSeek-V3	64	8	72
Llama 4 Maverick	8	8	16

Two equalities are traps. Mixtral and Llama 4 both total 16 teaching units, but Mixtral has 16 routed rows while Llama has 8 routed plus 8 always-on shared evaluations. V1 and V2 both total 64, but V2 adds group restriction, route scaling, report-level device admission, and communication-aware training pressure. Expert width, FFN form, layer cadence, dense blocks, modalities, padding, and kernels still differ. Therefore $U_{units}$ cannot be promoted to FLOPs, memory traffic, latency, or quality.

0x06L6 Published Parameter Splits Are Not An Equal-Budget Experiment

The source-reported active-to-total parameter fractions are approximately:

Mixtral 8x7B:       13 / 47   = 0.2766
DeepSeekMoE 16B:   2.8 / 16.4 = 0.1707
DeepSeek-V2:        21 / 236  = 0.0890
DeepSeek-V3:        37 / 671  = 0.0551
Llama 4 Maverick:   17 / 400  = 0.0425

Those ratios answer a model-specific storage/activation question. They do not hold expert width, dense layers, attention, vocabulary, modality encoders, precision, training tokens, or runtime constant. Lower fraction does not prove lower per-token FLOPs, lower communication, or better deployment efficiency.

0x06L7 Evidence Surfaces Must Stay Separable

Evidence surface	Can establish	Cannot establish by itself
paper/report	claimed algorithm, training objective, source-reported system result	exact released checkpoint fields or private runtime
checkpoint config	persistent dimensions and declared route options	learned behavior or complete trainer policy
readable reference forward	one executable selection/combine realization	production placement, collective, fusion, or latency
local teaching oracle	exact consequences for a pinned tiny fixture	paper reproduction or model benchmark
production trace	realized route, byte, time, and failure measurements	counterfactual quality or undocumented training causality

Llama 4 makes the unknown rule especially important. Its pinned official model card supports the 128-routed-plus-one-shared architecture, top-1 routed path, alternating dense/MoE cadence, 400B/17B parameter statement, and native multimodal scope used here. It does not expose a DeepSeek-V3-equivalent score, bias, capacity, or training-controller contract. Importing those fields would manufacture evidence.

0x06L8 Executable Contract Oracle

The checked-in Loop 10 oracle validates the six family records, derives the eight-token unit ledgers, replays the cross-family counterexamples, and rejects one-field mutations. Its decision order is:

def compare_sparse_family(contract, tokens=8):
    require_explicit_evidence_surface(contract)
    preserve_not_disclosed_fields(contract)

    routed_rows = tokens * contract.active_routed_experts
    shared_evals = tokens * contract.always_on_shared_units

    return {
        "route_operator": (
            contract.score_family,
            contract.eligibility,
            contract.selection,
            contract.selected_weight_policy,
        ),
        "admission_and_control": (
            contract.capacity_and_drop,
            contract.training_control,
        ),
        "architecture": contract.layer_cadence,
        "semantic_unit_ledger": (routed_rows, shared_evals),
        "runtime": "deferred to M6.6",
    }

The oracle fails if an unknown is imputed, if source-reported parameters are presented as an equal-budget benchmark, if a logical group is relabeled as a physical device or node without a placement manifest, or if a reference forward is promoted to a production runtime. It is a contract oracle, not a leaderboard.

0x07 Parameter Examples Are Source-Reported

Use DeepSeek and Mixtral numbers as examples, not universal constants.

System	Source-reported point	Safe use
Mixtral	8 experts per layer, 2 selected per token, total and active parameters differ	modern decoder-only MoE example
DeepSeekMoE	fine-grained experts plus shared experts	architecture mechanism
DeepSeek-V2	236B total, 21B activated per token	total-vs-active parameter example
DeepSeek-V3	671B total, 37B activated per token, auxiliary-loss-free balancing and MTP objective	scaled DeepSeek example

Do not write:

DeepSeek numbers prove any MoE deployment will be cheaper.

Write:

DeepSeek reports this total/activated split under its architecture, data, training, and system choices.

0x08 Failure Lab

Failure	Why it happens	Symptom
expert collapse	router repeatedly favors a small subset	overloaded experts and undertrained experts
balanced experts but imbalanced devices	many hot experts live on one rank	step time waits on one device
all-to-all ignored	expert placement treated like local indexing	latency jumps at scale
shared experts counted as free	always-on path omitted from active compute	active FLOP estimate too low
fine-grained experts over-fan out	token selects many remote experts	communication dominates
auxiliary-loss-free overclaimed	”no aux loss” read as “no control and no small sequence loss”	incorrect V3 training description
V2 device dropping implemented per expert	capacity scope changed	different routes are lost despite matching tensor shapes
V3 bias included in combine weights	load-control state leaks into expert quality weights	numerically wrong MoE output
V2 device-limited and V3 node-limited routing conflated	topology unit lost	wrong communication fanout
sigmoid affinities treated as already normalized	selected-axis normalization omitted	mixture scale varies with selected affinity mass
parameter counts misread	total and activated parameters mixed	serving memory and FLOP ledgers disagree
route-frequency conventions mixed	a statistic summing to $k$ is inserted into a formula expecting sum one	balance-loss magnitude changes with top-k
hard occupancy used without a proxy	discrete top-k count is treated as differentiable	router receives no useful balancing gradient
every selected gate normalized by habit	checkpoint-specific norm_topk_prob=false and route scaling are ignored	numerically different V1/V2 output
V3 group score implemented as a generic sum	released bias-enabled top-two group rule is lost	different eligible experts and fanout
paper, checkpoint, and serving code conflated	evidence from different lifecycle stages is merged	imaginary implementation matches no released system
Llama 4 comparison reduced to expert count	layer cadence, shared path, modality, and routing contract omitted	unsupported quality or latency conclusion

0x08A A Failure Record Needs Four Coordinates

A symptom is not yet an actionable failure contract. Each release failure needs:

invariant:
    what semantic or numerical relation must remain true?

detector:
    which tensor assertion, counter, trace, or source check can observe it?

consequence:
    wrong output, wrong gradient, wasted work, skew, or unsupported claim?

recovery owner:
    which article or runtime layer is allowed to repair it?

This prevents a common ownership error: fixing a physical imbalance by changing the router after M6.2 has frozen selected experts, or fixing a route-semantic bug by tuning a collective in M6.6. A faster implementation of the wrong route relation is still wrong.

0x08B Same Shape Does Not Mean Same Semantics

The most dangerous failures in this article often preserve tensor shape. A top-k result can still contain the wrong experts; a gate matrix can still have the right dimensions but the wrong normalization; an all-to-all can still return the expected number of rows but attach them to the wrong origin token.

Contract failure	Invariant and detector	Consequence	Recovery owner
selected-axis normalization copied into V1/V2	compare selected gate sums and route scale against pinned checkpoint policy	numerically different expert mixture	M6.2 router contract
V3 control bias reused as combine weight	trace original_affinity, corrected_score, and combine_gate as separate tensors	load controller changes expert quality weights	M6.2 V3 operator
V3 group score reduced to group maximum or sum-all	replay the Loop 7 counterexample and compare eligible groups/experts	different expert set with shape-correct top-k	M6.2 V3 eligibility
V2 logical group treated as a physical device	compare logical group IDs with an explicit placement-version map	claimed fanout bound may not exist physically	M6.6 placement, without changing M6.2 route semantics
teaching destination-row capacity counted as expert routes	record both destination-token rows and expert-route rows before admission	a different set of routes is dropped	M6.2 admission contract
protected/evaluation rows enter training drop path	assert sequence policy and lifecycle mode before admission	protected or evaluation tokens lose routed work	M6.2 training/evaluation policy
shared expert omitted from active work	reconcile routed rows and always-on shared evaluations	active compute and residency are understated	M6.2 architecture ledger
expert-major return uses token row without origin partition/slot	assert route identity and inverse-permutation bijection	outputs from different partitions or selected slots collide	M6.2 logical transpose
route metadata counted as free	maintain separate activation, output, and metadata byte ledgers	communication model understates payload and storage	M6.2 logical ledger; M6.6 realizes bytes
logical byte count promoted to latency	require placement, topology, collective, kernel, and measurement provenance	unsupported performance prediction	M6.6 runtime measurement

Two minimal anti-patterns show why ordinary shape assertions are insufficient:

# Wrong but shape-correct: V3 selection state leaks into output weighting.
selected = topk(original_affinity + control_bias, k)
gate = normalize((original_affinity + control_bias)[selected])

# Required separation.
selected = topk(original_affinity + control_bias, k)
gate = normalize(original_affinity[selected]) * route_scale

# Wrong but count-preserving: local token row is not globally unique.
scatter[origin_token_row] += gate * expert_output

# Required identity survives dispatch and return.
identity = (execution_scope, origin_partition, origin_token_row, selected_slot)
scatter[identity] += gate * expert_output

0x08C Model, Controller, And Evidence Failures Need Different Alarms

Failure class	Required detector	Release response
expert collapse or cold experts	selected/executed count, gate mass, entropy, route churn, gradient exposure by expert	adjust the owned training objective or controller and retrain; do not hide it with placement alone
balanced experts but hot devices/nodes	project accepted routes through a versioned placement; inspect max/mean receive rows and bytes	change placement or replication in M6.6 if semantic routes remain fixed
controller lag or oscillation	time series of bias span, sign reversals, update intervals, MaxVio, and quality metrics	tune controller schedule under joint quality/balance gates
source-reported no-drop violated	dropped_routes != 0 under the claimed V3 deployment contract	block release or relabel the runtime contract
paper/checkpoint/reference collapse	provenance check requires each claim to name its evidence surface and immutable revision	reject the claim; do not patch missing evidence with inference
unknown Llama 4 field imputed	disclosure manifest compares public fields with the contract table	restore not disclosed; do not borrow DeepSeek policy
parameter ratio used as efficiency ranking	release table lacks equal expert width, cadence, precision, workload, and runtime controls	relabel as source-reported architecture example
modality difference attributed to MoE alone	comparison contains text-only and native multimodal systems without controlled data/model study	remove causal quality claim

Balance alarms must be joint. A lower MaxVio can coincide with worse language modeling loss. A flat global histogram can conceal per-sequence or per-device skew. A stable controller can coexist with nearly equal affinities and useless specialization. Release gates therefore need quality, logical routing, controller state, and physical execution observations together.

0x08D Complete Source, Link, Visual, And Negative-Test Surface

The Loop 11 publication-surface oracle closes the static evidence inventory:

source manifest:
    36 unique source records
    17 local evidence records
    11 external primary sources
    8 official implementation/project sources

article links:
    19 external URLs, all represented by the source manifest
    7 normalized local post routes, all resolving to source Markdown

article visuals:
    38 unique image routes after this loop
    every route resolves to a repository asset and provenance record
    SVGs require role=img, title, description, and viewBox
    responsive pictures require both desktop and mobile contracts

negative evidence:
    9 invalid-fixture families
    169 one-field mutation cases across Loops 2-10

Static closure does not prove browser legibility or live external availability. Loop 12 owns normalized desktop/mobile, light/dark rendering, MathJax, overflow, alt-text, and final link acceptance. Loop 11 proves that every current route has an asset and provenance record, every local post target exists, and every published external URL is covered by a declared source.

0x09 What M6.2 Hands Forward

M6.2 completes the routing-algorithm and load-control side of distributed MoE. M6.6 continues from its logical placement and communication ledgers into the executable runtime:

M6.1:
    local routed capacity

M6.2:
    scaled routed capacity, load, placement, communication

M6.6:
    logical-to-physical expert maps, variable-size dispatch,
    local expert kernels, topology, replication, and overlap

M6.3 changes topic. It is not about routing tokens through many base experts. It is about adapting a frozen model with small trainable parameter deltas. Readers focused on the MoE execution path can continue directly to M6.6.

0x0A Publish Gate

Check	Status
Logical routes, physical placement, capacity, and collectives are established before DeepSeek mechanisms	pass
The complete A29 difficulty inventory appears before the systems formalism	pass
The specialization-team analogy is included with explicit limits	pass
Router positive feedback, fragmented expert batches, and uneven training signal are connected	pass
Expert, device, and communication imbalance are separated	pass
Hard route counts and differentiable router proxies are derived separately	pass
Shazeer, GShard, Switch, DeepSeek auxiliary-loss evolution is problem-driven	pass
Route-frequency normalization conventions are explicit	pass
All-to-all is included as a first-class ledger	pass
Fine-grained experts and shared experts are defined	pass
V1 knowledge mixing and redundancy motivate the architecture before its formulas	pass
Fine-grained total/active parameter preservation is derived	pass
The released DeepSeekMoE 16B checkpoint has a tensor and code trace	pass
Device-limited routing is explained as a communication constraint	pass
Auxiliary-loss-free balancing is not overclaimed	pass
Device-limited routing pseudocode is included	pass
Report device locality and checkpoint logical-group locality remain distinct	pass
Destination-token rows and expert-route rows are counted as different capacity units	pass
The V2 report’s unspecified capacity unit and rounding are not invented	pass
Protected training sequences and evaluation no-drop behavior are separated	pass
A dropped routed destination preserves shared and residual paths	pass
Bias control is separated from final expert weighting	pass
DeepSeek-V2’s three balance objectives and token-dropping policy are separated	pass
DeepSeek-V3 sigmoid affinity and selected-axis gate normalization are explicit	pass
Complementary sequence-wise auxiliary loss qualifies “auxiliary-loss-free”	pass
V2 device-limited and V3 node-limited routing are distinguished	pass
No-drop behavior is scoped to the reported V3 system rather than generalized	pass
Forward and backward expert-parallel ownership paths are explicit	pass
Runtime counters distinguish expert, device, and communication imbalance	pass
DeepSeek parameter numbers are marked source-reported	pass
V2 report, checkpoint model, and training-system evidence are not conflated	pass
V3 paper control state and reference inference code are separated	pass
Llama 4 comparison uses canonical Meta architecture facts and avoids prestige claims	pass
Switch, Mixtral, V1, V2, V3, and Llama 4 are compared by score, eligibility, selection, combine, admission, controller, and cadence contracts	pass
Common-input and equal-unit counterexamples prevent top-k or unit counts from standing in for the full router	pass
Publicly undisclosed Llama 4 gate, capacity, and controller fields remain unknown	pass
Expert-unit evaluations and source-reported parameter fractions are not promoted to FLOPs, latency, or equal-budget results	pass
Paper, checkpoint, reference, teaching-oracle, and production-runtime evidence remain separate	pass
Cross-loop failures name an invariant, detector, consequence, and bounded recovery owner	pass
Same-shape route, gate, capacity, and transpose failures have executable detectors	pass
All 38 image routes resolve to assets and static provenance records with nonempty alternatives	pass
All normalized local article routes resolve and all 19 external URLs are covered by the source manifest	pass
Static publication-surface closure is not promoted to browser or live-link acceptance	pass
M5 serving placement and M6.3 PEFT are deferred	pass

0xFF Reference

• Transformer Series A21 - MoE
• Transformer Series A29 - DeepSeek MoE
• M6.1 - Routed Capacity
• M6.6 - MoE Runtime Systems
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
• DeepSeekMoE 16B Base checkpoint configuration at revision 7c0fdaa
• DeepSeekMoE 16B Base modeling implementation
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• DeepSeek-V2 checkpoint configuration at revision 4461458
• DeepSeek-V2 checkpoint modeling implementation at revision 4461458
• Scaling Vision with Sparse Mixture of Experts
• DeepSeek-V3 Technical Report
• DeepSeek-V3 reference configuration at commit 9b4e978
• DeepSeek-V3 reference inference model at commit 9b4e978
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
• Adaptive Mixtures of Local Experts
• GShard
• DeepSpeed-MoE
• Switch Transformers
• Mixtral of Experts
• Meta: The Llama 4 herd
• Official Llama 4 model card at commit 0e0b8c5