M6.6 - MoE Runtime Systems: Placement, Dispatch, Kernels, and Topology

0x00 Overview: A Route Is Not Yet An Execution Plan

M6.1 followed a token through a router, capacity check, expert buffer, and weighted combine. M6.2 moved the same computation across expert-parallel ranks and introduced all-to-all traffic, device-limited routing, shared experts, and load balancing. Those mechanisms define the mathematical layer, but a production runtime must answer a more physical question:

How does a logical top-k route become executable distributed work under device placement, variable token counts, kernel-shape, precision, topology, and latency constraints?

The distinction matters because a router returns logical expert identifiers. It does not inherently name a GPU, a process, a physical replica, a memory address, a communication offset, or a grouped-GEMM segment. Those objects belong to the runtime.

An MoE forward pass therefore contains two related plans:

semantic plan:
    token -> logical expert -> gate weight -> weighted expert output

physical plan:
    token row
        -> physical expert replica
        -> destination rank
        -> send offset
        -> local expert segment
        -> kernel row
        -> return offset
        -> origin token and route slot

The semantic plan determines model behavior. The physical plan determines whether the behavior can be executed correctly and efficiently on the current cluster.

0x00A Five Runtime Eras, Not One Kernel

The runtime did not appear after the MoE algorithm was finished. Each stage of MoE scaling exposed a different physical problem, and later systems changed the execution contract rather than merely making one old kernel faster.

Era	Newly visible constraint	Runtime consequence
sparse conditional computation	one example activates only a few remotely stored experts	route rows to expert owners, then return them without changing semantic identity
expert-parallel Transformers	every data-parallel rank contributes rows to partitioned experts	encode, all-to-all dispatch, local expert compute, all-to-all return, decode
static-capacity systems	collectives and dense kernels prefer bounded regular buffers	capacity slots, padding, overflow policy, and fixed split layouts enter the schedule
adaptive and dropless systems	expert row counts are data-dependent and often highly uneven	fast permutation, variable splits, grouped GEMM, or block-sparse execution replace one padded dense batch
low-latency serving	decode exposes tiny segments, repeated synchronization, changing request sets, and graph constraints	phase-specific dispatch, persistent buffers, masked execution, topology-aware placement, replicas, and explicit completion handles

The first sparsely gated systems already mixed data and model parallelism: ordinary layers were replicated over input batches while experts were shared across devices. Combining rows from several input shards enlarged each expert’s effective batch, but it also made permutation and cross-device movement part of the layer definition. GShard and Switch made that pattern routine at Transformer scale; DeepSpeed-MoE and Tutel then treated communication topology, parallel composition, and adaptive execution as system design variables rather than hidden framework behavior.

The A21 figure below preserves that expert-parallel baseline. Attention and gating remain local to each worker, routed rows cross an all-to-all boundary to partitioned experts, and expert outputs cross a second boundary before the next replicated block. It is a lineage diagram, not a claim that every modern runtime uses the same collective or buffer layout.

Source-local visual: A21 - MoE, Figure 28. The source article uses it to explain expert parallelism. M6.6 retains only the physical ownership lesson; M6.1 owns route semantics and M6.2 owns logical balance and locality constraints.

The next pressure was irregularity. A route tensor is token-major, but an expert kernel wants expert-major contiguous rows. FastMoE made the scatter/compute/gather transformation explicit and implemented the memory movement as a dedicated operator. Tutel generalized the systems problem toward adaptive parallelism and pipelining. MegaBlocks changed the local compute representation again: instead of dropping rows or padding every expert to a common capacity, it maps uneven expert segments into block-sparse operations.

Source-local visual: A21 - MoE, Figure 42, which attributes the diagram to FastMoE. It proves the permutation problem’s historical visibility, not the exact metadata layout or CUDA implementation reconstructed later in this article.

Modern serving adds another discontinuity. DeepSeek-V3 reports restricting communication fanout and overlapping communication with computation. DeepEP exposes paired dispatch/combine communication objects; EPLB separates logical experts from physical replicas; Megatron Core carries several dispatcher families; and SGLang chooses communication and expert-runner backends according to phase and deployment constraints. These are related systems, but they are not interchangeable versions of one algorithm.

The result is a family of execution contracts. A correct article must first say which contract it is describing.

0x00A1 What The History Does And Does Not Prove

The historical sequence supports three conclusions:

1. permutation and reverse permutation are semantic necessities once experts are physically partitioned;
2. regular capacity-padded execution, variable-size grouped execution, and block-sparse dropless execution solve different shape contracts;
3. prefill throughput and decode latency can legitimately select different communication and compute plans.

It does not prove that a paper benchmark transfers to another model, topology, batch distribution, or library version. It also does not prove that a released reference implementation is the private training or production runtime described by a paper.

0x00B Why Dynamic Routing Fights Static Kernels

Dense Transformer layers expose regular matrices. If a dense FFN receives $T$ rows, both projections see a known $[T,d]$ input. An MoE layer begins with the same hidden-state tensor but partitions its routed rows according to data-dependent expert choices.

For top-$k$ routing:

input rows:              T
routed rows:             R = T * k
rows assigned to expert e: n_e, data-dependent
sum_e n_e:               R

The total routed-row count is known once $T$ and $k$ are known. The vector $(n_0,\ldots,n_{E-1})$ is not. Yet high-throughput kernels prefer:

contiguous operands
known alignment
few launches
stable workspaces
predictable synchronization
reusable graph shapes

The runtime is largely the machinery that reconciles variable expert loads with those kernel requirements.

0x00C Prefill And Decode Are Different Operating Points

The same MoE layer can see thousands of prompt rows during prefill and only one row per active request during decode. A throughput path can amortize metadata and communication over large contiguous segments. A latency path may instead prioritize fixed buffers, graph capture, fewer synchronization points, and masked execution.

This article will not use “the MoE kernel” as if one path were optimal for both regimes.

0x00D Evidence Ladder And Runtime Pins

No one source proves the complete path from a router decision to production latency. M6.6 keeps five evidence levels separate:

Evidence level	What it can establish	What it cannot establish alone
source-local A21/A29	teaching lineage, terminology, original figure context, and links worth following	correctness of every translated explanation or current runtime behavior
primary paper/report	stated algorithm, experimental setup, and source-reported result	unreleased implementation details or performance on another system
released project documentation	supported public concepts and configuration at that documented version	exact behavior after later commits or undocumented production policy
immutable source artifact	one concrete interface, tensor path, and implementation state at one commit	universal API semantics or reproduced end-to-end performance
local oracle and measurement	behavior of the declared fixture or benchmark environment	paper reproduction or production generality unless the protocol matches

The runtime artifacts used by the later implementation trace are frozen at Loop 1 rather than resolved from floating branches:

Runtime surface	Immutable revision	Loop-owned use
DeepEP V2 ElasticBuffer README and legacy wrapper	01dc3aaac82068020353dce2c302e38153c0bfaa	paired dispatch/combine handle, phase-specific operation, event ownership
EPLB solver	d52c72d5b2f2fb4c41afbf8eb21366820239913d	logical-to-physical replication and placement maps
Megatron Core dispatchers	0896be0f5735e910a906dc4a8b5ed0363e4a7be6	all-to-all, flex, and graph-compatible dispatcher lifecycle
SGLang expert-parallel stack	e76d0acdc923d992bbda20d4b2bc51db9ac314a7	serving-stage dispatch/runner/combine integration
MegaBlocks DMoE implementation	952db33d6eac334d22c61e47a0d5d41446298784	block-sparse dropless local execution
Tutel fast dispatcher	ac0e51b943bb7084d4a6a7243e44a4afe04b9911	encode/decode permutation and adaptive runtime lineage

The commit hash is part of every implementation claim. Later source-to-instruction work may compare newer revisions, but it must add a new artifact record rather than silently moving this baseline.

0x00E Ownership And Boundaries

This article owns:

logical-to-physical expert placement
expert replicas and placement versions
dispatch metadata and variable-size collectives
local expert execution layouts
prefill and decode runtime modes
topology-aware dispatch
expert-parallel load balancing
communication-compute overlap
payload precision and alignment
runtime observability, evaluation, and failure recovery

It defers:

router learning and capacity semantics         -> M6.1
DeepSeek routing and balancing algorithms      -> M6.2
general serving/KV lifecycle                   -> M5
quantization method selection and release      -> M8

The same boundary as a contract table is:

Relation	Owner	Exact contract
consumes	M6.1	accepted route rows, route-slot identity, logical expert identity, gate weights, admission state, and inverse-combine identity
consumes	M6.2	route-exact expert-major segmentation, logical locality records, replay state, and a producer contract that explicitly excludes physical ranks and kernels
consumes	M4	byte, traffic, launch, occupancy, and measured-kernel accounting conventions
consumes	M5	request phase, batch composition, graph-capture, stream, and serving-lifecycle constraints
owns	M6.6	logical-to-physical maps, replicas, ranks, variable exchanges, expert-runner layouts, completion handles, placement generations, and runtime evidence
hands forward	M8	payload/weight representation requirements, alignment, scale ownership, kernel eligibility, and release compatibility
defers	M6.1/M6.2	route selection, route admission, gate mathematics, balance objectives, and training control

The firewall is operational: M6.6 may choose a replica for an already-selected logical expert, but it may not replace that logical expert, renormalize its gate, reinterpret a dropped route, or infer semantic admission from a physical buffer limit.

0x01 Vocabulary And Runtime Invariants

Let:

Symbol	Meaning
$T_l$	input token rows initially owned by one rank
$T$	token rows across the expert-parallel group
$d$	hidden width of each routed row
$E$	number of logical routed experts
$k$	selected experts per token
$R=T k$	routed rows before any overflow or duplicate suppression
$P$	number of expert-parallel ranks
$Q_v$	number of physical expert slots in placement version $v$, including replicas
$\phi_v(q)$	logical expert stored in physical slot $q$ under placement version $v$
$\rho_v(e)$	set of physical slots that replicate logical expert $e$ under version $v$
$n_e$	routed rows assigned to logical expert $e$
$n_q$	routed rows assigned to physical slot $q$ after replica selection
$b_a$	bytes per dispatched activation element
$b_o$	bytes per returned output element

A correct runtime must preserve these invariants.

0x01A0 Six Identity Namespaces Before Any Collective

An integer such as 3 is not a sufficient expert identifier. It can mean logical expert 3, local slot 3, rank 3, replica 3, or the fourth element of a process-group array. A runtime manifest has to type those namespaces before it can derive counts or offsets.

Namespace	Stable within	Can change without changing model semantics?	Example in the executable fixture
logical expert ID	model architecture and checkpoint	no; replacing it changes the selected function	E0 through E3
physical slot ID	one placement generation	yes, if it remains a compatible replica	q0 through q5
replica ID	one logical expert and checkpoint generation	yes; another replica may execute the same logical route	e0-r0, e0-r1
destination rank	one process-group epoch	yes; placement may move or replicate an expert	rank 0 through rank 3
process-group ID	one communicator membership epoch	only through an explicit runtime transition	ep-group-0
placement/checkpoint version	dispatch-handle lifetime	no; dispatch and combine must agree	placement-v17, checkpoint-0042

The semantic identity of accepted route row $r$ is:

$$I_r^{\mathrm{semantic}} = (s,r,p,t,j,e,g)$$

where $s$ is the execution scope, $r$ the route ID, $p$ the origin partition, $t$ the origin token row, $j$ the selected route slot, $e$ the logical expert, and $g$ the combine gate. M6.6 receives this tuple; it does not recompute it.

The runtime appends a physical tuple:

$$I_r^{\mathrm{physical}} = (v,q,a,u,G,D,c)$$

where $v$ is the placement version, $q$ the physical slot, $a$ the replica ID, $u$ the destination rank, $G$ the process group, $D$ the topology domain, and $c$ the expert checkpoint version. Keeping the tuples separate is what permits replica selection without semantic rerouting.

The Loop 2 fixture freezes all 17 fields in the M6.2 handoff with a canonical SHA-256 digest before adding any physical field. A mutation to logical_expert_id or combine_gate therefore fails before dispatch planning begins.

0x01A Semantic Invariant

For token $t$, if the router selects logical expert set $\mathcal{E}_t$, the output remains:

$$y_t=\sum_{e\in\mathcal{E}_t}g_{t,e}E_e(h_t)$$

Physical replication must not silently change $E_e$. Every usable replica of logical expert $e$ must represent the same model state for that execution version.

0x01B Conservation Invariant

Without overflow, dropping, or route deduplication:

$$\sum_{e=0}^{E-1}n_e = \sum_{q=0}^{Q-1}n_q = T k$$

Every accepted routed row must be dispatched once, executed once, returned once, and combined once. A mismatch usually indicates a lost route, duplicate route, stale placement map, or incorrect collective count.

0x01C Identity Invariant

A routed row is not identified only by its token index. It needs at least:

origin expert-parallel rank
origin token row
route slot in [0, k)
logical expert id
physical expert slot
destination rank
gate weight
placement version
send and return positions

Two routes from the same token are distinct rows even when they carry identical hidden values. They may use different experts and different gate weights.

0x01D State-Version Invariant

The following objects must agree:

router output logical ids
logical-to-physical map
replica weights
dispatch counts and offsets
combine handle
expert checkpoint/version

Changing expert placement between dispatch and combine without a versioned transition can return a numerically valid tensor from the wrong expert.

0x01D1 Compatibility Is A Predicate, Not A Lookup

For accepted route $r$ targeting logical expert $e_r$, physical slot $q$ is legal only when every coordinate agrees:

$$\operatorname{compatible}(r,q;v) \iff \phi_v(q)=e_r \land c_v(q)=c_v(e_r) \land u_v(q)\in G_v \land D_v(q)=D(u_v(q))$$

The four clauses reject four different failures:

1. the slot stores another logical expert;
2. a replica has stale or mixed checkpoint state;
3. the destination rank is outside the pinned expert-parallel communicator;
4. the topology label does not own that rank.

Replica selection must then satisfy a cardinality contract:

$$\forall r\in\mathcal{R}_{\mathrm{accepted}}, \qquad \sum_{q\in\rho_v(e_r)} \mathbf{1}[A_v(r)=q] =1$$

Zero assignments lose a route. Two assignments execute it twice. Assigning one row to a slot outside $\rho_v(e_r)$ conserves the row count but changes the model function. Count conservation alone is therefore necessary and insufficient.

0x01D2 One Immutable Execution Scope

The runtime snapshot used by one layer execution binds:

semantic contract version
execution scope id
placement version
expert checkpoint version
process-group id and membership epoch
dispatch handle id
route count and inverse identity

The snapshot is acquired before physical assignment. Dispatch records copy its version coordinates, and combine consumes the same handle. A new placement may become active concurrently, but it cannot service an old handle.

def acquire_runtime_scope(logical_routes, placement_registry):
    semantic_digest = sha256(canonical(logical_routes))
    placement = placement_registry.acquire_active_generation()

    handle = DispatchHandle(
        execution_scope_id=logical_routes.execution_scope_id,
        semantic_digest=semantic_digest,
        placement_version=placement.version,
        checkpoint_version=placement.checkpoint_version,
        process_group_id=placement.process_group_id,
        generation_reference_acquired=True,
    )
    return placement, handle

Acquisition does not imply that placement is immutable forever. It means that the selected generation cannot be retired until every handle that references it has combined and released.

0x02 Logical Experts, Physical Slots, And Replicas

0x02A The One-Expert-One-Rank Baseline

Suppose $E=P=4$ and each rank owns one expert:

logical expert 0 -> rank 0, slot 0
logical expert 1 -> rank 1, slot 1
logical expert 2 -> rank 2, slot 2
logical expert 3 -> rank 3, slot 3

The map is easy, but a hot expert becomes a hot rank. If $n_0=50\%$ of routed rows while the other experts split the rest, rank 0 determines the layer barrier even when the cluster has unused compute elsewhere.

0x02B Replication Changes Placement, Not The Router Vocabulary

Replicate logical expert 0:

physical slot q0 on rank 0 -> logical expert 0
physical slot q1 on rank 1 -> logical expert 0
physical slot q2 on rank 1 -> logical expert 1
physical slot q3 on rank 2 -> logical expert 2
physical slot q4 on rank 3 -> logical expert 3

The router still emits expert id 0. A replica selector chooses either $q_0$ or $q_1$ for that route. This second decision can account for locality, predicted load, queue depth, or a deterministic balancing rule.

If logical expert $e$ has $m_e=|\rho(e)|$ replicas and the runtime balances perfectly, its ideal per-replica load is:

$$\bar{n}_{e,\text{replica}}=\frac{n_e}{m_e}$$

The actual load may differ because routes are indivisible, topology constraints restrict candidates, and replica selection itself can be stale.

0x02B1 A Six-Slot Manifest With Two Replicated Experts

The executable Loop 2 fixture uses four logical experts, four ranks, two nodes, and six physical slots:

Slot	Logical expert	Replica	Rank	Domain	Checkpoint
q0	0	e0-r0	0	node-0	checkpoint-0042
q1	0	e0-r1	1	node-0	checkpoint-0042
q2	1	e1-r0	1	node-0	checkpoint-0042
q3	1	e1-r1	2	node-1	checkpoint-0042
q4	2	e2-r0	2	node-1	checkpoint-0042
q5	3	e3-r0	3	node-1	checkpoint-0042

The semantic expert counts are [4, 2, 1, 1]. Replica selection maps them to physical slot counts {q0: 2, q1: 2, q2: 1, q3: 1, q4: 1, q5: 1} and rank counts {0: 2, 1: 3, 2: 2, 3: 1}. These ledgers answer different questions:

logical expert counts:
    did execution preserve the selected model functions?

physical slot counts:
    how much expert-runner work did each replica receive?

destination-rank counts:
    how much row work and communication pressure reached each rank?

Balancing the last two ledgers is legal. Altering the first is not.

0x02B2 The Versioned Placement Manifest

The manifest stores physical slots under a placement generation rather than as one timeless table:

{
  "placement_version": "placement-v17",
  "generation": 17,
  "state": "DRAINING",
  "process_group_id": "ep-group-0",
  "checkpoint_version": "checkpoint-0042",
  "in_flight_refcount": 1,
  "slots": [
    {
      "physical_slot_id": "q1",
      "logical_expert_id": 0,
      "replica_id": "e0-r1",
      "destination_rank": 1,
      "topology_domain_id": "node-0",
      "parameter_digest": "expert-0-state-v42"
    }
  ]
}

physical_slot_id is unique only inside the placement version. The complete key is (placement_version, physical_slot_id). Likewise, a replica name is not proof of compatible weights; all replicas of one logical expert must agree on checkpoint version and parameter-state digest for that generation.

0x02B3 Dispatch/Combine Generation Fence

The fixture models a safe placement transition:

1. placement-v17 is active
2. dispatch-h17 acquires one v17 generation reference
3. placement-v18 becomes active for new dispatches
4. v17 enters DRAINING
5. retirement of v17 is denied: refcount = 1
6. combine uses dispatch-h17 and v17 metadata
7. the handle releases its generation reference
8. retirement of v17 is now allowed

The state machine is deliberately stricter than “the new map exists.” ACTIVE means eligible for new dispatches. DRAINING means ineligible for new dispatches but still valid for pinned handles. RETIRED means no future dispatch or combine may resolve through that generation.

def combine_and_release(local_outputs, handle, registry):
    placement = registry.require_live(handle.placement_version)
    assert placement.checkpoint_version == handle.checkpoint_version
    assert placement.process_group_id == handle.process_group_id

    output = inverse_dispatch_and_combine(local_outputs, handle)
    registry.release_generation(handle.placement_version, handle.handle_id)
    return output

If combine silently resolves placement_registry.active() after step 3, it can use v18 offsets or replica owners for v17-dispatched rows. Shapes may still match; identity does not.

0x02C Resident Parameter Cost

If one expert contains $P_e$ parameters stored at $b_w$ bytes per parameter, its resident weight memory is:

$$M_e=P_e b_w$$

Physical expert residency is:

$$M_{\text{physical experts}} = \sum_{q=0}^{Q-1} P_{\phi(q)}b_w$$

For equal-sized experts and $R_{\text{extra}}=Q-E$ additional replicas:

$$M_{\text{physical experts}} =(E+R_{\text{extra}})P_{\text{expert}}b_w$$

Replication buys compute and placement flexibility by spending parameter memory. It does not reduce the model’s logical parameter count, and it does not make the expert output an ensemble average. One physical replica is selected for each logical route.

0x02D Rank Load

Let $\mathcal{Q}_p$ be the physical slots on rank $p$. A row-count approximation to rank work is:

$$L_p^{\text{rows}}=\sum_{q\in\mathcal{Q}_p}n_q$$

A better estimate includes expert widths, kernel efficiency, dtype, and communication:

$$L_p^{\text{time}} \approx \sum_{q\in\mathcal{Q}_p}C_q(n_q) +C_{\text{dispatch},p} +C_{\text{combine},p}$$

Equal token counts do not guarantee equal time. An expert segment below a kernel-efficiency threshold can cost more per row than a larger segment, and inter-node routes can cost more than local routes.

0x03 From Router Output To Dispatch Layout

The dispatch pipeline transforms token-major routing outputs into destination-major expert segments.

0x03A Router-Side Tensors

For one origin rank:

hidden_states:      [T_l, d]
topk_ids:           [T_l, k]
topk_weights:       [T_l, k]
placement_version: scalar or plan handle

Flattening route slots creates $R_l=T_l k$ route records:

route_hidden:       [R_l, d]       # gathered or logically indexed from hidden_states
origin_token:       [R_l]
route_slot:         [R_l]
logical_expert:     [R_l]
physical_slot:      [R_l]
destination_rank:   [R_l]
gate_weight:        [R_l]

Materializing route_hidden duplicates each token row $k$ times. Some implementations delay or fuse that gather, but the logical routed-row count remains $R_l$.

0x03A1 A Columnar Eight-Route Record

The executable trace does not invoke placement.select_replica again. Loop 2 already froze one compatible physical assignment for every accepted semantic route under placement-v17. Loop 3 joins that assignment to four hidden rows of width three:

hidden[p0,t0] = [1, 10, 100]    hidden[p0,t1] = [2, 20, 200]
hidden[p1,t0] = [3, 30, 300]    hidden[p1,t1] = [4, 40, 400]

The resulting relation is columnar even though the table prints it by row:

Route	Source rank	Origin	Route slot	Logical expert	Gate	Physical slot	Destination rank
r00	0	p0.t0	0	0	0.65	q0	0
r01	0	p0.t0	1	1	0.35	q2	1
r02	0	p0.t1	0	0	0.70	q1	1
r03	0	p0.t1	1	2	0.30	q4	2
r04	1	p1.t0	0	3	0.58	q5	3
r05	1	p1.t0	1	0	0.42	q0	0
r06	1	p1.t1	0	1	0.55	q3	2
r07	1	p1.t1	1	0	0.45	q1	1

Ranks 2 and 3 own no input partitions in this fixture. They nevertheless remain members of the four-rank process group. Removing those zero-count source axes would change every split-vector coordinate and can make peers disagree about collective shape.

Before counting, every row is checked against the immutable slot manifest:

slot = placement.slots[route.physical_slot_id]
assert slot.logical_expert_id == route.logical_expert_id
assert slot.replica_id == route.replica_id
assert slot.destination_rank == route.destination_rank
assert route.source_rank in process_group.member_ranks
assert route.destination_rank in process_group.member_ranks

This is stronger than checking that there are still eight rows. A route sent once to the wrong expert conserves counts and still changes the function.

0x03B Count, Prefix Sum, Permute

For source rank $p$ and destination rank $u$, define one count-matrix entry:

$$C_{p,u} = \sum_{j=0}^{R-1} \mathbf{1}[s_j=p]\mathbf{1}[d_j=u]$$

The fixture produces:

             destination rank u
source p       0   1   2   3
       0       1   2   1   0
       1       1   1   1   1
       2       0   0   0   0
       3       0   0   0   0

Every row sum is the source’s routed-row count. Every column sum is the destination’s receive count. Both total eight:

$$\sum_{u=0}^{P-1}C_{p,u}=R_p, \qquad \sum_{p=0}^{P-1}\sum_{u=0}^{P-1}C_{p,u}=R=8$$

The send offset for destination axis $u$ is the exclusive prefix sum:

$$O^{\mathrm{send}}_{p,u} = \sum_{v=0}^{u-1}C_{p,v}, \qquad O^{\mathrm{send}}_{p,P}=R_p$$

Thus rank 0 obtains [0,1,3,4,4] and rank 1 obtains [0,1,2,3,4]; zero-source ranks obtain [0,0,0,0,0]. The final sentinel offset is part of the contract because segment $u$ occupies:

$$\left[O^{\mathrm{send}}_{p,u},O^{\mathrm{send}}_{p,u+1}\right)$$

Rows are stably sorted by (destination_rank, physical_slot_id, semantic_route_position). The last key makes ties deterministic and retains a path back to semantic route identity:

rank 0 send order: r00 | r02 r01 | r03 | empty
rank 1 send order: r05 | r07     | r06 | r04

def pack_source(routes, hidden, group_ranks):
    counts = zeros([len(group_ranks)], dtype=int64)
    for route in routes:
        counts[group_axis(route.destination_rank)] += 1

    offsets = exclusive_prefix_sum(counts)  # length P + 1
    order = stable_argsort(
        routes,
        key=(destination_rank, physical_slot_id, semantic_route_position),
    )
    send_rows = gather(hidden, routes[order].origin_identity)
    send_meta = gather(route_columns, order)

    assert offsets[-1] == len(routes)
    return send_rows, send_meta, counts, offsets, order

This is mechanism pseudocode. A real implementation avoids Python objects and uses device-side histograms, prefix sums, fused permutation kernels, and compact metadata arrays.

0x03C Variable-Size Exchange

The distributed transpose is easier to audit when token rows stay visibly color-coded by origin and destination:

Source-local visual: A21 - MoE, citing Tutel. Stage 1 is the all-to-all dispatch from origin-GPU token order into expert-owner order; stage 2 returns expert outputs to the origin ranks for combine. The gate may be replicated, but expert FFNs are partitioned. The figure shows a regular dense layout; production exchanges can be variable-sized and topology-aware.

After each source rank computes destination counts, the expert-parallel group exchanges counts and then payloads. Conceptually:

origin rank p:
    send_counts[p, :]
    send_buffer grouped by destination

destination rank r:
    receive_counts[:, r]
    receive_buffer grouped by origin and physical expert

An all-to-all-v operation is a natural representation because each source-destination pair can carry a different row count. Implementations may realize this through native variable-size collectives, padded all-to-all, all-gather plus local filtering, or a specialized dispatch library.

0x03C1 Split Vectors Are A Matrix Transpose

The count matrix is not only a traffic summary. It is the shape agreement between senders and receivers. Destination rank $u$ receives source-indexed split vector:

$$S^{\mathrm{recv}}_{u,p}=C_{p,u}$$

For the exact fixture:

rank 0 receives [1,1,0,0] -> [r00,r05]
rank 1 receives [2,1,0,0] -> [r02,r01,r07]
rank 2 receives [1,1,0,0] -> [r03,r06]
rank 3 receives [0,1,0,0] -> [r04]

The payload arrives source-major because each sender contributed one contiguous destination slice. An implementation that assumes this is already expert-major can feed the right number of rows to the wrong expert.

The count-control alternatives carry different contracts:

Design	Shape agreement	What stays regular	What may be wasted or delayed
native all-to-all-v	exchange or derive every peer split	useful payload rows	count/control path and dynamic offsets
uniform padded all-to-all	fixed peer capacity	allocation, collective shape, graph capture	padding to the hottest source-destination pair
all-gather and filter	common gathered shape	ownership and collective API	rows received by ranks that do not execute them
fused runtime transport	implementation-owned handle and buffers	pack/transport/decode pipeline	portability and version-specific assumptions

For this fixture, all-to-all-v carries eight useful row slots. Uniformly padding all $P^2=16$ source-destination cells to the maximum split of two allocates:

$$R_{\mathrm{padded}}=P^2\max_{p,u}C_{p,u}=4^2\cdot2=32$$

so 24 of 32 row slots are padding. This is a row-slot comparison, not yet a byte or latency claim; Loop 4 owns payload dtype, metadata, liveness, events, and measured transfer cost.

0x03C2 Receive-Side Expert Repack

The destination performs a second stable permutation, now by (physical_slot_id, source_rank, source_send_position). Let $n_{u,q}$ be the rows for physical slot $q$ resident on destination $u$. Its local expert offsets are:

$$O^{\mathrm{expert}}_{u,q} = \sum_{a<q}n_{u,a}$$

where $a<q$ means the manifest’s explicit local slot order, not lexicographic coincidence. The fixture yields:

rank 0: q0 [r00,r05]                   counts [2],   offsets [0,2]
rank 1: q1 [r02,r07], q2 [r01]         counts [2,1], offsets [0,2,3]
rank 2: q3 [r06],     q4 [r03]         counts [1,1], offsets [0,1,2]
rank 3: q5 [r04]                       counts [1],   offsets [0,1]

Three coordinates must survive for each route: its position in the source send buffer, the destination receive buffer, and the local expert buffer. None can be reconstructed safely from token ID alone.

for route in semantic_route_order:
    metadata[route.id] = {
        "source_send_position": inverse_send[route.id],
        "destination_receive_position": inverse_exchange[route.id],
        "local_expert_position": inverse_expert_repack[route.id],
        "origin_partition": route.origin_partition,
        "origin_token": route.origin_token,
        "route_slot": route.route_slot,
        "gate": route.combine_gate,
    }

0x03C3 Round-Trip Identity And Weighted Combine

After expert execution, inverse local repack and inverse exchange restore the eight route rows to semantic order:

[r00,r01,r02,r03,r04,r05,r06,r07]

If $\pi_s$ is source packing, $\pi_x$ the variable exchange, and $\pi_e$ the receive-side expert repack, the return path must implement the inverse composition:

$$\Pi^{-1} = \pi_s^{-1}\circ\pi_x^{-1}\circ\pi_e^{-1}, \qquad \Pi^{-1}(\Pi(r))=r$$

The combine is then a weighted scatter-add over the original semantic key, not over physical slot or destination rank:

$$y_{p,t} = \sum_{r:\,(p_r,t_r)=(p,t)}g_r\,z_r$$

The oracle uses scalar expert transforms [1,2,3,4] only to make wrong-expert execution observable. Packed execution and direct semantic execution both produce:

p0.t0 -> [1.35, 13.5, 135]
p0.t1 -> [3.20, 32.0, 320]
p1.t0 -> [8.22, 82.2, 822]
p1.t1 -> [6.20, 62.0, 620]

Changing only a count, offset, stable order, split vector, local slot offset, inverse route, or combined value causes the checked fixture to fail. The oracle executes no collective and no expert GPU kernel; it proves tensor identity and algebra for this declared layout.

0x03D Activation Byte Ledger

Loop 3 counted rows and positions. Bytes appear only after Loop 4 pins a representation. The baseline uses BF16 activation and expert-output elements, INT64 count/offset coordinates, INT32 transport metadata, and FP32 combine gates retained in the origin handle.

Five ledgers must remain distinct:

Ledger	Question answered	Evidence required
row slots	how many useful or padded tensor rows exist?	route and layout manifest
logical bytes	what payload follows from declared shapes and dtypes?	representation manifest
live allocation bytes	which declared buffers coexist?	half-open lifetime schedule and alias rules
measured fabric/HBM bytes	what did hardware actually transfer?	profiler counters under a pinned runtime/topology
elapsed time	how long was the dependency path?	synchronized benchmark protocol and clocks

The first three are executable in the local oracle. The last two are deliberately unclaimed.

0x03D1 Staging Payload Is Not Remote Network Payload

Let $R$ be all useful route rows in a collective staging layout, $R_{remote}$ the subset whose destination rank differs from its source, and $b_a,b_o$ the dispatch and combine element widths. Then:

$$B_{\mathrm{stage,dispatch}}=Rdb_a, \qquad B_{\mathrm{stage,combine}}=Rdb_o$$

but the representation-level remote payload is:

$$B_{\mathrm{remote,payload}} = R_{remote}d(b_a+b_o)$$

For the exact fixture, $R=8$, $R_{remote}=6$, $d=3$, and $b_a=b_o=2$ bytes:

collective staging payload:
    dispatch = 8 * 3 * 2 = 48 B
    combine  = 8 * 3 * 2 = 48 B

remote row payload:
    dispatch = 6 * 3 * 2 = 36 B
    combine  = 6 * 3 * 2 = 36 B

The two local route rows remain in the packed staging layout but do not traverse a network link under the declared local-bypass policy. Calling 48 bytes “network traffic” would therefore overcount even before protocol effects.

0x03D2 Transport Metadata, Handle State, And Control Tables

Payload rows do not contain enough identity to return themselves. The baseline sends two INT32 fields per remote dispatch row, one INT32 handle index per remote combine row, and retains eight INT32-sized semantic/physical fields per useful route in the origin handle:

$$\begin{aligned} B_{\mathrm{dispatch\ meta}}&=R_{remote}\cdot2\cdot4=48\ \mathrm{B},\\ B_{\mathrm{combine\ meta}}&=R_{remote}\cdot1\cdot4=24\ \mathrm{B},\\ B_{\mathrm{route\ handle}}&=R\cdot8\cdot4=256\ \mathrm{B}. \end{aligned}$$

This is one selectable schema, not a universal ABI. A runtime can pack fields, derive them from offsets, transmit gates, or keep richer metadata. It must still preserve route identity and declare where each field lives.

The four-rank count and split tables use INT64 coordinates:

$$\begin{aligned} B_C&=P^2b_c=4^2\cdot8=128\ \mathrm{B},\\ B_{O^{send}}&=P(P+1)b_c=4\cdot5\cdot8=160\ \mathrm{B},\\ B_{S^{recv}}&=P^2b_c=128\ \mathrm{B}. \end{aligned}$$

The local expert-offset arrays contain ten INT64 entries across the four destinations, adding 80 bytes. Summing table storage is not a network claim: these tensors can be produced locally, exchanged, derived, cached, or embedded in a specialized handle depending on the runtime.

Under this declared schema, remote row payload plus remote transport metadata is:

$$B_{\mathrm{remote,logical}} =36+36+48+24 =144\ \mathrm{B}$$

It excludes collective headers, scale tensors, alignment, retransmission, link-layer transactions, topology duplication, and physical HBM reads/writes. The name therefore remains logical remote bytes excluding protocol.

0x03D3 Half-Open Buffer Lifetimes And Peak

The fixture allocates fourteen named buffers. A buffer is live on half-open interval $[a_i,z_i)$, so the declared live payload at epoch $\tau$ is:

$$M_{live}(\tau) = \sum_i B_i\mathbf{1}[a_i\le\tau<z_i], \qquad M_{peak}=\max_\tau M_{live}(\tau)$$

Buffer or allocation	Bytes	Live interval	Last completion required
input hidden	24	[0,9)	handle released
immutable route handle	256	[0,9)	handle released
count matrix	128	[1,7)	combine complete
send offsets	160	[1,7)	combine complete
receive splits	128	[1,7)	combine complete
packed send payload	48	[1,3)	dispatch complete
send metadata	64	[1,3)	dispatch complete
source-major receive	48	[2,4)	expert input ready
local expert offsets	80	[3,5)	expert complete
local expert input	48	[3,5)	expert complete
expert output / return payload	48	[4,7)	combine complete
return metadata	32	[4,7)	combine complete
returned source-order rows	48	[6,8)	output ready
output hidden	24	[7,10)	downstream consumer

Adding every allocation gives 1,136 bytes, but the peak is 904 bytes at epoch 4. The sum and peak differ because lifetimes differ. The expert_output_return_payload is one declared alias: expert output is reused as the combine send payload without a second overlapping allocation. Two independently live buffers assigned that alias group are rejected.

0x03D4 Submission, Completion, And Stream Ownership

The correctness DAG is:

route_ready [route stream]
    -> dispatch_packed [pack stream]
    -> dispatch_complete [communication stream]
    -> expert_input_ready [expert stream]
    -> expert_complete [expert stream]
    -> combine_complete [communication stream]
    -> output_ready [output stream]
    -> handle_released [output stream]

An API returning from dispatch(...) can mean “submitted,” “payload available,” or “operation complete.” The handle must say which. Reusing send_payload immediately after submission is safe only if the transport has copied or finished reading it; the abstract baseline waits for dispatch_complete.

dispatch_ticket = dispatch_stream.submit(send_payload, send_metadata, splits)
dispatch_complete = dispatch_ticket.completion_event

expert_stream.wait(dispatch_complete)
expert_input = repack(receive_source_major, route_handle)
expert_output = run_local_experts(expert_input)
expert_complete = expert_stream.record_event()

combine_stream.wait(expert_complete)
combine_ticket = combine_stream.submit(expert_output, return_metadata)
combine_complete = combine_ticket.completion_event

output_stream.wait(combine_complete)
output = inverse_permute_and_scatter(returned_rows, route_handle)
output_ready = output_stream.record_event()

route_handle.release_after(output_ready)

This is a dependency contract, not a claim that a runtime uses these stream names or serializes all experts. Loop 10 owns legal communication-compute overlap and contention. Loop 7 will map the abstract tickets to pinned public interfaces.

0x03D5 Safe Release Is A Last-Reader Property

For buffer $U$, let $\mathcal{R}(U)$ be operations that may still read it. Safe reuse requires:

$$z_U \ge \max_{o\in\mathcal{R}(U)}\tau_{complete}(o)$$

That rule explains several otherwise same-shape failures:

Premature action	Missing fence	Consequence
overwrite packed send rows after host submission	dispatch completion	in-flight transport reads new rows
release source-major receive before repack completes	expert-input-ready event	expert rows become nondeterministic
reuse expert output while combine is in flight	combine completion	returned values are corrupted
retire placement generation after dispatch but before scatter	output-ready/handle release	inverse metadata or replica state disappears

The checked oracle rejects an early send release, early handle release, missing dependency, cyclic event graph, epoch-before-dependency, and overlapping live alias.

0x03D6 Regular Padding Changes Allocation Before It Changes Time

The Loop 3 count matrix has maximum peer split two. Uniformly padding all 16 source-destination cells therefore reserves 32 row slots. At BF16 and width three:

per direction: 32 * 3 * 2 = 192 B
round trip:    384 B

useful staging per direction: 48 B
useful staging round trip:     96 B

That proves a fourfold row-payload reservation for this fixture. It does not prove a fourfold slowdown: regular buffers can reduce control cost, enable graph capture, or select a faster transport. Conversely, all-to-all-v can send fewer useful bytes and still lose to launch, synchronization, or skew. Loop 6 owns the prefill/decode and graph-capture operating-point decision.

For a larger shape with $T=4096$, $k=2$, $d=4096$, BF16 rows, and remote fraction $f_{remote}=0.75$:

all useful routed rows, one direction:
    T * k * d * 2 = 64 MiB of staging payload

dispatch plus combine staging payload:
    128 MiB

remote row payload only:
    0.75 * 128 MiB = 96 MiB

Metadata, control tensors, protocol overhead, quantization scales, padding, and physical transactions remain separate. The ledger describes representation-level bytes, not elapsed time.

0x03E A Two-Rank Trace

Suppose each rank starts with two tokens, $k=2$, and two logical experts:

rank 0 owns physical expert 0
rank 1 owns physical expert 1

rank 0 routes:
    token 0 -> expert 0, expert 1
    token 1 -> expert 1, expert 0

rank 1 routes:
    token 0 -> expert 1, expert 0
    token 1 -> expert 1, expert 0

Each rank has four routed rows. Rank 0 sends two rows to rank 1 and keeps two locally. Rank 1 sends two rows to rank 0 and keeps two locally. After expert execution, each returned row must recover:

origin rank
origin token row
route slot
gate weight

Combining only by token id would collide rank 0 token 0 with rank 1 token 0. Combining only by (origin_rank, token_id) would still lose the two route slots. The full route identity is required until reduction.

0x04 Local Expert Execution: Padding, Grouped GEMM, Or Blocks

After dispatch, a destination rank owns variable row counts per physical expert slot. A local runner does not receive a router probability matrix. It receives the already selected, already placed, already permuted segments from Loop 3:

received_rows:      [R_recv, d]
expert_offsets:     [Q_local + 1]
rows for slot q:    received_rows[offsets[q]:offsets[q+1]]
route handle:       inverse coordinates + combine gates + generation

The runtime can encode those segments in several ways without changing which expert function each useful row must evaluate. This section therefore compares representations of one fixed route tensor, not different routing policies. Capacity selection and token dropping remain M6.1/M6.2 decisions.

0x04A The Frozen Six-Segment Input

Loop 3 fixes the local physical-slot order and route membership:

Destination rank	Physical slots	Useful rows in slot order
0	q0	q0=[r00,r05]
1	q1,q2	q1=[r02,r07], q2=[r01]
2	q3,q4	q3=[r06], q4=[r03]
3	q5	q5=[r04]

In global physical-slot order, the useful row counts and exclusive offsets are

$$\mathbf n=[2,2,1,1,1,1], \qquad \mathbf o=[0,2,4,5,6,7,8].$$

Those eight rows are the semantic work. A representation may add zero-valued fringe rows, descriptor state, or sparse topology metadata, but it may neither discard a useful route nor return a padding row to combine.

Each production expert commonly contains a gated FFN such as SwiGLU:

$$G_q=X_qW_{g,q}^{\top}, \qquad U_q=X_qW_{u,q}^{\top},$$ $$E_q(X_q)=\left(\operatorname{SiLU}(G_q)\odot U_q\right)W_{2,q}^{\top}, \qquad X_q\in\mathbb{R}^{n_q\times d}.$$

Under the M4 convention that one multiply-add is two FLOPs, the three matrix multiplications contribute

$$F_{\mathrm{SwiGLU,MM}}(n_q) =2n_qdd_{ff}+2n_qdd_{ff}+2n_qd_{ff}d =6n_qdd_{ff}.$$

The SiLU and Hadamard product are non-matmul work and must be accounted for separately if a kernel-level estimate needs them. The executable fixture uses the much smaller operator $E_e(x)=s_ex$ with $s=[1,2,3,4]$. That substitution makes representation parity exactly inspectable; it does not claim to run a SwiGLU GPU kernel. The SwiGLU shapes and FLOPs are a separate analytical ledger.

0x04B Capacity-Padded Dense Execution

The fixed-capacity construction first reserves $C$ rows for every physical slot, then may round that capacity to a kernel row alignment $a$:

$$C_{\mathrm{exec}} =a\left\lceil\frac{C}{a}\right\rceil, \qquad R_{\mathrm{padded}}=Q_{\mathrm{physical}}C_{\mathrm{exec}}.$$

For the frozen fixture, let $C=3$, $a=2$, and $Q_{\mathrm{physical}}=6$. Capacity alone reserves $6\times3=18$ row slots. Kernel alignment raises the stride to $C_{\mathrm{exec}}=4$, so the runner actually presents 24 rows:

useful rows                   8
capacity row slots           18   (10 capacity-unused)
aligned executed row slots   24   (6 more alignment-only)
total non-useful rows        16
row utilization              8 / 24 = 33.3%
input scalar values          24 * d     = 72
SwiGLU intermediate values   24 * d_ff  = 96

For $d=3$ and $d_{ff}=4$, the analytical matmul ledger is

$$F_{\mathrm{padded}}=6(24)(3)(4)=1{,}728\ \text{FLOPs},$$

while only $6(8)(3)(4)=576$ FLOPs correspond to useful rows.

The padding mask belongs to the runner contract, not to the router:

for physical_slot, useful_segment in segments:
    dense = zeros([aligned_capacity, d], dtype=activation_dtype)
    dense[:len(useful_segment)] = useful_segment

    dense_out = expert[physical_slot](dense)

    # Only useful rows retain route identities. Padding never reaches combine.
    useful_out = dense_out[:len(useful_segment)]
    return_rows(useful_out, route_metadata[physical_slot])

Regular shapes can simplify a batched kernel, static allocation, and graph capture. The row ledger alone does not decide latency: a more regular kernel may outperform an irregular representation despite doing more arithmetic. That is a measurement question for Loop 11.

0x04C Variable Grouped GEMM

A grouped interface keeps the exact row dimension $m_q=n_q$ for every active physical slot while sharing one launch or scheduler:

problem q, gate/up:   [n_q, d]    @ [d_ff, d]^T
problem q, down:      [n_q, d_ff] @ [d, d_ff]^T
problem list m:       [2, 2, 1, 1, 1, 1]
segment offsets:      [0, 2, 4, 5, 6, 7, 8]

The grouped runner executes eight row slots and has row utilization one for this fixture. It still needs a descriptor or equivalent launch state for each active problem. The teaching ABI uses eight INT64 fields per problem:

m, n, k,
input_offset, gate_weight_offset, up_weight_offset,
down_weight_offset, output_offset

Six active problems therefore use $6\times8\times8=384$ descriptor bytes in the fixture. This is a selectable teaching schema, not a universal grouped GEMM ABI. Real libraries may use pointer arrays, device-side problem shapes, group tables, persistent schedulers, or additional workspace.

problems = []
for slot, begin, end in zip(slot_ids, offsets[:-1], offsets[1:]):
    if begin == end:
        continue
    problems.append(
        ExpertProblem(
            m=end - begin,
            n=d_ff,
            k=d,
            x=received_rows[begin:end],
            weights=weights[slot],
            output=expert_outputs[begin:end],
        )
    )

grouped_swiglu(problems, workspace)

The analytical matmul work is exactly the useful-row ledger, $F_{\mathrm{grouped}}=576$ FLOPs for the toy dimensions. That does not mean the runtime cost is zero beyond those FLOPs. Descriptor construction, very small segment efficiency, load scheduling, workspace, and tail dimensions can dominate.

0x04D Block-Sparse Dropless Execution

MegaBlocks starts from a useful equivalence. Fixed-capacity batched expert GEMMs can be written as one block-diagonal multiplication. Uneven expert row counts then become diagonal blocks with unequal heights. Splitting those variable regions into smaller fixed blocks makes the operation representable by block-sparse matrix multiplication.

Source: MegaBlocks, Figure 3. Panel A is equal-size batched expert computation, panel B its equal-size block-diagonal form, and panel C the variable-row block-sparse construction. It is not a generic layout used by every grouped-GEMM or inference runtime.

For a token-row block size $B_r$, each expert is rounded independently:

$$\widetilde n_q =B_r\left\lceil\frac{n_q}{B_r}\right\rceil, \qquad R_{\mathrm{block}}=\sum_q\widetilde n_q.$$

The teaching fixture uses $B_r=2$, giving

$$\widetilde{\mathbf n}=[2,2,2,2,2,2], \qquad \widetilde{\mathbf o}=[0,2,4,6,8,10,12].$$

It executes 12 row slots: eight useful rows plus four block-fringe rows, for $66.7\%$ row utilization. The analytical matmul ledger is $6(12)(3)(4)=864$ FLOPs.

The output-column teaching block is $B_c=2$ and $d_{ff}=4$, so each expert row block has two nonzero column blocks. Six row blocks produce 12 nonzero $2\times2$ matrix blocks. A minimal fixture topology stores:

row_offsets:        7 INT32 values
column_indices:    12 INT32 values
row_indices:       12 INT32 values
transpose_indices: 12 INT32 values
total:             43 INT32 values = 172 bytes

Those 172 bytes do not include gather indices, padded bins, allocator headers, or kernel workspace. They are a schema-specific topology ledger, not a claim about total MegaBlocks metadata.

counts = histogram(physical_slot_ids)
padded_counts = round_up_each(counts, block_rows)
padded_bins = inclusive_prefix(padded_counts)

x_blocked = padded_gather(received_rows, route_order, padded_bins)
topology = make_block_diagonal_topology(padded_bins, d_ff, block_shape)

# A gated MLP alternates sparse-output and sparse-input products.
hidden = sdd(x_blocked, W_gate_and_up, topology)
blocked_out = dsd(activation(hidden), W_down, topology)

useful_out = padded_scatter_useful_rows_only(blocked_out, route_order)

The pinned MegaBlocks implementation at revision 952db33d6eac334d22c61e47a0d5d41446298784 uses a block size of 128, rounds each expert’s token count before padded_gather, constructs dynamic topology, and supports a grouped path as a distinct implementation choice. The oracle’s two-row block exists only so every index fits in a visible example. It must not be presented as MegaBlocks’ production block size or as a performance reproduction.

0x04E Capacity, Alignment, And Block Padding Are Different

The three sources of extra rows answer different questions:

Extra-row contract	Question answered	Frozen-fixture result
capacity padding	How many rows may one expert admit under the model/runtime policy?	18 slots before alignment
alignment padding	What row multiple does this dense kernel path require?	24 executed slots
block-fringe padding	How many rows are needed to complete each sparse block?	12 executed slots

Token dropping is a fourth choice and is not padding. Dropping would reduce the set of eight useful routes and therefore change the semantic contract. None of the three exact runner alternatives in this loop drops a route.

0x04F One Function, Three Physical Representations

The CPU oracle executes the teaching expert $E_e(x)=s_ex$ through all three layouts. For example:

q0 -> expert 0: r00=[1,10,100], r05=[3,30,300]
q2 -> expert 1: r01=[2,20,200]
q4 -> expert 2: r03=[6,60,600]
q5 -> expert 3: r04=[12,120,1200]

After inverse routing and gate-weighted combine, every runner recovers the Loop 3 outputs:

p0.t0 = [1.35, 13.5, 135]
p0.t1 = [3.2, 32, 320]
p1.t0 = [8.22, 82.2, 822]
p1.t1 = [6.2, 62, 620]

The representations therefore have equal semantics in this declared fixture, but different work and state:

Runner	Executed rows	Row utilization	Input values	Intermediate values	Analytical SwiGLU matmul FLOPs	Additional state shown
capacity-padded dense	24	33.3%	72	96	1,728	valid-row mask
variable grouped GEMM	8	100%	24	32	576	384-byte teaching descriptor table
teaching block sparse	12	66.7%	36	48	864	172-byte minimal topology

This table is an accounting result, not a winner declaration. It omits physical kernel throughput, launch cost, occupancy, cache behavior, topology-build cost, and hardware-specific alignment. Those measurements may reverse the ordering.

0x04G Runner Failure Lab

Same-shape mistake	Violated contract	Consequence
capacity chosen below max(n_q)	all useful rows execute	overflow is silently converted into a runner drop
padded row retains a route id	padding has no semantic identity	zero/fringe output is combined into a real token
grouped descriptor points to the wrong expert weight offset	physical slot owns its parameter state	numerically valid output from the wrong expert
grouped offsets omit a zero-row expert without preserving slot identity	descriptor index and physical slot are conflated	later descriptors bind to shifted weights
block bins use raw counts while gather uses rounded counts	topology and payload share offsets	one expert reads another expert’s fringe rows
block topology is reused after route counts change	topology is generation- and shape-dependent	sparse products read stale coordinates
MegaBlocks’ 128-row source pin is replaced by the toy block size	evidence scopes are conflated	teaching arithmetic is misreported as implementation fact
fewer analytical row FLOPs are called lower latency	logical work is confused with measurement	unsupported performance claim

0x05 Prefill And Decode Runtime Modes

The words prefill and decode do not select a kernel by themselves. They change the number and provenance of current token rows, the frequency with which the path repeats, the useful expert-segment sizes, and which state can remain at stable addresses. The semantic route still obeys

$$R=T k,$$

where $T$ is the number of current input rows in this MoE invocation, not the total number of tokens represented by the KV cache. During one-token decode, $T$ is normally the active sequence count. During prefill, $T$ is the number of uncached prompt rows admitted to that invocation.

0x05A One Phase Manifest, Four Workloads

Loop 6 keeps the Loop 5 dimensions $d=3$, $d_{ff}=4$, $k=2$, six physical slots, and the same placement-v17 generation. It changes only current-row shape and phase policy:

Profile	Rank-local current tokens	Physical-slot route counts	Useful routes
prefill	[32,24,0,0]	[28,22,18,16,14,14]	112
decode	[3,2,0,0]	[2,2,2,2,1,1]	10
mixed outer batch	[8,7,0,0]	[7,6,5,4,4,4]	30
oversize decode	[5,4,2,0]	[5,4,4,3,3,3]	22

Every row satisfies $\sum_q n_q=k\sum_u T_u$. The profiles declare no latency. They count logical host submissions and control items only to expose how fixed orchestration becomes more prominent as route rows shrink.

0x05B Prefill: Variable Shapes And Amortized Planning

The prefill fixture has 56 current input rows and 112 routed rows. Its selected path keeps variable dispatch counts and invokes grouped expert problems with the exact segment sizes:

dispatch useful / executed token slots   56 / 56
runner useful / executed route rows      112 / 112
logical host submissions                 7
declared control items                    34
control items per useful route            34/112 = 0.304

The 34 items are a teaching control ledger: count cells, slot coordinates, and prefix/dispatch decisions under one declared schema. They are not FLOPs, bytes, or microseconds. The ratio

$$I_{\mathrm{control}}=\frac{C_{\mathrm{control}}}{R}$$

shows only that a fixed amount of planning is amortized over more useful rows. It does not show how much time each control item costs.

Large prefill segments often make contiguous permutation and grouped or block-sparse expert execution attractive. They do not guarantee balance: one long prompt or one hot expert can still dominate a rank. Loop 8 owns the topology consequences, and Loop 11 owns measurements.

0x05C Decode: A Fixed Bucket With Mutable Valid State

The decode fixture has five active requests: ranks contribute [3,2,0,0] current rows. Its graph manifest reserves four token slots per rank and two route rows per physical slot:

graph id                         decode.bucket4.v17
rank token capacities            [4,4,4,4]
physical-slot route capacities   [2,2,2,2,2,2]
valid input rows                 5
reserved input slots             16
valid route rows                 10
executed masked runner rows      12

The graph padding is physical state, not semantic input. Device-side real-count and valid-route tensors must mask all 11 unused input slots and both unused runner rows before they can acquire expert identities.

The teaching graph reserves five stable buffers:

graph.hidden
graph.route_ids
graph.gates
graph.expert_output
graph.step_metadata

For BF16 hidden/output, INT32 route ids, FP32 gates, $P=4$, $k=2$, $d=3$, 16 token slots, and three INT32 metadata values per rank, the reserved payload ledger is

$$M_{\mathrm{graph}} =16d(2)+16k(4)+16k(4)+16d(2)+P(3)(4) =496\ \text{bytes}.$$

This is not total graph memory. It excludes allocator overhead, expert weights, collective workspace, kernel workspace, graph runtime objects, and every buffer not named by the fixture.

The captured addresses stay fixed, while these values change on device each step:

real_tokens_by_rank
valid_tokens
rank_token_offsets
valid_routes_by_slot

The fixture therefore executes more physical slots than it uses:

dispatch useful / executed token slots   5 / 16
runner useful / executed route rows      10 / 12
logical host submissions                 1
declared control items                    12
control items per useful route            12/10 = 1.2

One logical graph submission does not prove one GPU launch, nor does it prove a latency value. It records the host orchestration boundary only.

0x05D Graph Admission Is An Exact Predicate

The route counts may vary inside captured capacities, but the graph’s immutable key may not drift:

$$K_{\mathrm{graph}} =(G,v,c,d,k,\tau_x,\tau_g,D_{\mathrm{dispatch}},D_{\mathrm{runner}},\mathbf C_T,\mathbf C_q),$$

where $G$ is process-group identity, $v$ placement generation, $c$ checkpoint, $\tau_x/\tau_g$ dtypes, and $\mathbf C_T/\mathbf C_q$ rank-token and physical-slot capacities.

def admit_decode_graph(step, graph):
    if step.phase != "decode":
        return Fallback("phase_not_decode")
    if step.immutable_key != graph.immutable_key:
        return Fallback("graph_key_mismatch")
    if any(t > cap for t, cap in zip(step.rank_tokens, graph.rank_caps)):
        return Fallback("rank_token_capacity_exceeded")
    if any(n > cap for n, cap in zip(step.slot_routes, graph.slot_caps)):
        return Fallback("expert_route_capacity_exceeded")
    if not graph.stable_buffers_ready:
        return Fallback("stable_buffer_unavailable")

    update_device_validity_state(step)
    mask_graph_padding_routes()
    return Replay(graph.graph_id)

Placement is part of the key because captured expert pointers and physical-slot coordinates belong to one generation. Replaying a placement-v17 graph against placement-v18 is not dynamic load balancing; it is stale state. The valid fallback is an uncaptured path with the current immutable plan, or a graph built for the new generation.

0x05E Mixed Batches Preserve Phase Provenance

The outer mixed fixture contains 12 prefill rows and three decode rows. Merging all 15 rows into one homogeneous execution would erase whether a row is latency sensitive and whether it is eligible for the captured decode bucket. The declared policy partitions the batch before dispatch:

prefill unit:
    rank tokens [7,5,0,0]
    24 routes
    variable dispatch + grouped runner

decode unit:
    rank tokens [1,2,0,0]
    6 routes
    decode graph admitted
    16 dispatch slots + 12 runner rows

The aggregate accounting is then

useful input rows                 15
executed dispatch token slots     12 + 16 = 28
dispatch padding slots            13
useful route rows                 30
executed runner rows              24 + 12 = 36
runner padding rows               6
phase execution units             2
graph execution units             1
logical host submissions          7 + 1 = 8

Splitting is not universally optimal. It can reduce segment sizes and introduce another scheduling boundary. It is the declared fixture policy because it keeps phase objectives and graph eligibility explicit. M5 owns the ordinary scheduler decision that forms these units.

0x05F Oversize Decode Falls Back Without Semantic Drift

The oversize decode contributes [5,4,2,0] current rows. Rank 0 exceeds its captured capacity of four, and several expert segments exceed the graph’s two-route slot capacities. Admission fails first at rank_token_capacity_exceeded.

The fallback executes 11 variable dispatch rows and 22 grouped expert rows. It does not truncate the fifth rank-0 token, drop routes to fit the graph, or reuse stale graph metadata:

graph replay:                 rejected
semantic input rows:         11 unchanged
semantic route rows:         22 unchanged
fallback dispatch slots:     11
fallback runner rows:        22

A fallback is correct only if it preserves the route and generation contracts. Graph admission is a performance-path decision, never permission to alter model semantics.

0x05G Source Contracts Behind The Phase Split

The phase distinction is visible in several versioned implementation surfaces:

• The pinned DeepEP V2 README exposes a normal training/prefill path and a decode path under one dispatch/combine interface, with handles and completion events. Its benchmark numbers remain source-reported.
• The pinned SGLang EP documentation distinguishes normal throughput-oriented prefill and low-latency graph-compatible decode modes and recommends automatic switching. That is one release’s integration contract, not a universal API.
• The pinned Megatron inference dispatcher separates equal-count graph and variable-count behavior, uses stable per-step metadata tensors, and includes a variable-count graph-oriented path under stated hardware constraints.

Loop 7 will bind exact classes, methods, arguments, and source lines. Loop 6 uses only the common state contract: stable addresses, mutable valid counts, masked padding, exact capacity admission, and semantic-preserving fallback.

0x05H Phase And Graph Failure Lab

Same-shape mistake	Violated contract	Consequence
cached-context length used as decode $T$	current rows define MoE work	route and capacity ledgers are inflated
graph padding receives valid router ids	padding has no semantic identity	inactive slots execute and combine
route counts change but valid-count tensors do not	mutable graph state describes this step	stale rows survive replay
placement changes without graph-key change	graph binds physical generation	captured pointers address wrong experts
oversize step is clipped to its bucket	fallback preserves semantic rows	accepted tokens/routes disappear
mixed rows are merged without phase provenance	graph admission and objective are per execution unit	decode latency and prefill throughput policies are conflated
stable buffer address is changed after capture	replay uses captured addresses	graph reads or writes unrelated storage
one graph submission is called one kernel	host and device execution coordinates are conflated	launch accounting is false
fewer control items per route is called lower latency	logical control count is not measured time	unsupported performance conclusion

0x06 A Real Runtime Trace: Dispatcher, Runner, Combine

Fast-moving runtime code must be treated as a versioned implementation, not as the definition of MoE. This section traces six public surfaces pinned in Loop 1:

Megatron token dispatcher       0896be0f... / blob 256f38cc...
Megatron inference dispatcher   0896be0f... / blob 1a2ae10b...
DeepEP V2 README                01dc3aaa... / blob 078a1514...
DeepEP legacy compatibility     01dc3aaa... / blob e4f92abe...
SGLang EP documentation         e76d0acd... / blob 26e38cf0...
SGLang DeepEP dispatcher        e76d0acd... / blob f33c74f7...

The trace is source evidence. No GPU path was executed, and comments or type signatures do not prove latency, throughput, memory use, or a private production runtime.

0x06A Six Surfaces, Not Six Implementations Of One Interface

The public artifacts expose different ownership layers:

Surface	What the pin exposes	What it does not establish
Megatron standard all-to-all	layer-level preprocess, transport, local expert ordering, reverse transport, and final unpermute	one explicit portable dispatch-handle object
Megatron flex + DeepEP manager	layer wrapper around fused dispatch/combine and manager-owned state	the same ABI as standard all-to-all
Megatron inference NCCL	equal-count graph and unequal-count padded/compact all-gather paths	universal decode eligibility
Megatron inference NVLS	on-device variable-count metadata and persistent symmetric buffers	support without Hopper+, NVLink, and symmetric-memory prerequisites
DeepEP README + compatibility wrapper	communication primitives, mode-specific handles, counts, and completion events	full router, expert runner, scheduler, or current ElasticBuffer source internals
SGLang DeepEP integration	typed normal/low-latency outputs, runner boundary, stage machine, and mode resolution	DeepEP’s internal kernel schedule or a universal serving architecture

The common statement is deliberately narrower:

An already-selected route becomes transport state, destination-local expert rows, expert output, reverse transport state, and origin-local output. The state created for dispatch must remain valid through its matching combine.

0x06B The Eight-State Lifetime Contract

Loop 7 names the lifecycle independently of any method signature:

0 router_output
    produces topk_ids, topk_weights

1 dispatch_metadata_ready
    produces counts, layout, placement_generation

2 dispatch_in_flight
    produces dispatch_handle, dispatch_event

3 expert_rows_ready
    dispatch_event has completed

4 expert_output_ready
    local runner has produced expert_output

5 combine_in_flight
    consumes expert_output and dispatch_handle

6 origin_rows_ready
    combine_event has completed

7 dispatch_state_releasable
    origin output is visible; paired state may retire

The lifetime invariant is

$$G_{\mathrm{route}} =G_{\mathrm{dispatch}} =G_{\mathrm{combine}} =G_{\mathrm{placement}},$$

for the generation bound to one in-flight invocation. The source libraries do not all call this value a generation. M6.6 requires it because Loop 2 proved that an old placement cannot retire while its route state is still live.

0x06C Megatron Standard All-To-All: State Lives In The Dispatcher

At 0896be0f..., MoETokenDispatcher splits the layer into six abstract operations rather than one dispatch() call:

dispatch_preprocess
token_dispatch
dispatch_postprocess
expert runner                    # outside dispatcher
combine_preprocess
token_combine
combine_postprocess

The standard MoEAlltoAllTokenDispatcher consumes

hidden_states   [T_local, d]
routing_map     [T_local, E] bool
probs           [T_local, E]

and derives variable split state from expert counts. For the EP axis,

$$\sum_{j=0}^{P-1}\texttt{input\_splits}[j]=R_{\mathrm{local}}, \qquad \sum_{i=0}^{P-1}\texttt{output\_splits}[i]=R_{\mathrm{recv}}.$$

The pinned source then executes this exact physical order:

preprocess routing_map
    -> num_local_tokens_per_expert
    -> input_splits / output_splits

permute 1
    hidden [T_local,d]
    -> route-major local send rows [R_local,d]
    save reversed_local_input_permutation_mapping

token_dispatch
    all_to_all(hidden, output_splits, input_splits)
    all_to_all(probs,  output_splits, input_splits)

dispatch_postprocess
    optional TP gather
    optional sort_chunks_by_idxs into local-expert order
    -> [R_recv,d], tokens_per_expert, route probabilities

expert runner

combine_preprocess
    reverse local-expert sort
    optional TP reduce-scatter

token_combine
    all_to_all(hidden, input_splits, output_splits)

combine_postprocess
    unpermute with saved inverse map and routing_map
    reshape to original hidden shape

There is no single EPHandle in this path. The paired state is distributed over instance fields such as input_splits, output_splits, routing_map, probs, and reversed_local_input_permutation_mapping. Calling the API “stateless” because it lacks a handle argument would therefore be wrong.

The source also shows a real design choice: token rows and probability rows are separate all-to-all operands. A fused transport may package those coordinates differently, but it must preserve their route association.

0x06D Megatron Flex And Inference Paths Change The State Contract

The same Megatron revision also contains a flex dispatcher whose DeepEP manager uses fused communication. _DeepepManager.setup_metadata converts the dense route/probability surface to top-k indices and probabilities. fused_dispatch returns destination rows, destination indices, probabilities, expert counts, and a manager-owned handle. fused_combine consumes that handle, after which the source clears the handle and dispatched metadata.

standard A2A:
    explicit permute -> A2A -> local sort
    reverse state spread across dispatcher fields

flex DeepEP manager:
    setup_metadata -> fused_dispatch
    reverse state held by _DeepepManager.handle
    fused_combine -> clear handle and dispatched metadata

Neither representation changes the selected logical route. The flex manager’s reconstruction of top-k indices is a format conversion over upstream probabilities and dropped-route sentinels, not permission to choose a different expert policy.

Megatron’s inference source exposes two more contracts:

Dispatcher	Per-step row contract	Persistent state	Qualification
NCCLAllGatherDispatcher graph path	equal token counts per EP rank	stable _valid_tokens_tensor	source says decode-only graphs enforce equal counts at this pin
NCCLAllGatherDispatcher non-graph path	unequal rank counts; pad to max, all-gather, compact	_local_tokens_per_rank plus valid-count state	host count synchronization occurs; source says this path does not run under graph capture
NVLSAllGatherVDispatcher	variable counts with device-side metadata	_step_metadata, real-count tensor, hidden/route/probability/return symmetric buffers	requires Hopper+, NVLink, and symmetric-memory support

The NVLS path allocates worst-case buffers once, updates [valid_tokens, rank_token_offset, ep_max_tokens] in place, and pre-masks route storage so graph-padding rows cannot reach the expert runner. That source contract is the implementation analogue of Loop 6’s stable addresses plus mutable valid state; it is not evidence that the Loop 6 teaching buffer schema matches Megatron byte for byte.

0x06E DeepEP: Communication Handle And Event, Not A Full MoE Layer

The pinned DeepEP V2 README describes one ElasticBuffer surface for high-throughput and low-latency operations. Its training/prefill example returns

recv_x
recv_topk_idx
recv_topk_weights
EPHandle
EventOverlap

and the matching combine consumes the handle. The event may be waited on before the runner reads receive rows. The decode example additionally shows a cached handle path, qualified by the condition that the routing layout remains valid.

The separately pinned deep_ep/buffers/legacy.py is explicitly a compatibility surface. It must not be presented as the V2 ElasticBuffer implementation. It is still useful source evidence for two physical layouts:

normal dispatch:
    variable received rows
    recv_topk_idx / recv_topk_weights
    num_recv_tokens_per_expert_list
    mode-specific tuple handle
    EventOverlap

low-latency dispatch:
    expert-major capacity tensor
    packed receive counts
    low-latency handle
    invalid slots represented separately from useful tokens
    EventOverlap or receive hook

The normal and low-latency handles are not interchangeable tuples. Their matching combine methods unpack different state. A generic tutorial should therefore type the handle by runtime revision and mode rather than serialize it as an undocumented list of tensors.

0x06F SGLang Makes Mode And Stage State Explicit

At e76d0acd..., SGLang’s public EP documentation separates TopK, Dispatcher.dispatch, MoeRunner.forward, and Dispatcher.combine. The pinned DeepEP dispatcher then defines different typed transport boundaries:

DeepEPNormalDispatchOutput:
    hidden_states, hidden_states_scale,
    topk_ids, topk_weights,
    num_recv_tokens_per_expert

DeepEPLLDispatchOutput:
    hidden_states, hidden_states_scale,
    topk_ids, topk_weights,
    masked_m, expected_m

Normal mode first obtains a dispatch layout, calls buffer.dispatch, stores self.handle, waits on the event when configured for asynchronous finish, and passes a normal-format object to the runner. Its combine consumes the stored handle and clears it after completion.

Low-latency mode calls buffer.low_latency_dispatch, retains both packed_recv_count and self.handle, and passes a masked expert-major contract to the runner. low_latency_combine consumes the handle and gate state; the source clears both retained fields after launching the return path.

The wrapper also enforces an explicit state machine:

INITIAL
    -> AFTER_DISPATCH_A
    -> AFTER_DISPATCH_B
    -> AFTER_COMBINE_A
    -> INITIAL

Calling combine before dispatch_b, dispatching twice without combine, or mixing normal output with low-latency combine input violates a typed integration contract even if the hidden tensors have compatible shapes.

In auto mode, _get_impl() resolves normal versus low-latency from batch phase at this pin. That is SGLang behavior at one revision, not a universal definition of prefill and decode.

0x06G Binding The Loop 6 Profiles To Public Source Surfaces

The source oracle makes only five phase bindings:

Loop 6 profile	Pinned surface	Binding strength
56-row prefill	SGLang normal DeepEP integration	documentation and source at e76d0acd...
five-request decode	SGLang low-latency integration	documentation and source at e76d0acd...
five-request decode	Megatron equal-count NCCL graph path	conditional pinned-source contract
five-request decode	Megatron variable-count NVLS graph path	conditional on stated hardware and symmetric-memory support
oversize decode	uncaptured or larger valid path	M6.6 correctness boundary, not one library API promise

DeepEP’s README supports training/prefill and decode communication surfaces, but the Loop 6 numeric profiles are not DeepEP benchmark reproductions. Mixed-batch splitting remains an M5 scheduler decision consumed by M6.6.

0x06H A Portable Contract Without A Fictional Portable API

The safest integration pseudocode names state ownership while leaving each runtime’s method decomposition intact:

def execute_routed_layer(hidden, routes, phase, runtime):
    generation = runtime.acquire_placement_generation()

    dispatch_state = runtime.prepare_dispatch(
        hidden=hidden,
        immutable_routes=routes,
        phase=phase,
        generation=generation,
    )

    received = runtime.start_dispatch(dispatch_state)
    runtime.wait_dispatch_ready(received.completion)

    expert_output = runtime.run_local_experts(
        received.rows,
        received.runner_metadata,
    )

    returned = runtime.start_combine(
        expert_output,
        paired_dispatch_state=dispatch_state,
    )
    runtime.wait_combine_ready(returned.completion)

    assert returned.generation == generation
    assert returned.output.shape == hidden.shape
    runtime.release_dispatch_state(dispatch_state)
    runtime.release_placement_generation(generation)
    return returned.output

For Megatron standard all-to-all, dispatch_state is an explanatory wrapper around instance fields. For DeepEP it corresponds more directly to a handle and event. For SGLang it also includes mode-specific typed state and stage-machine position. The pseudocode is a contract adapter, not copy-paste API code.

0x06I Source-To-Instruction Failure Lab

Mistake	Why shapes can still look valid	Failure
standard Megatron path is described as handle-free	inverse state lives in dispatcher fields	second invocation overwrites state needed by the first combine
flex handle is reused after combine	source clears it after paired return	stale routing metadata or use-after-release
NVLS path is enabled without its hardware contract	tensor shapes do not encode fabric support	initialization failure or unsupported transport
graph padding is not pre-masked	buffers are capacity-shaped	invalid rows acquire expert work
legacy.Buffer is called the V2 ElasticBuffer implementation	both expose dispatch/combine language	documentation and source claims are conflated
normal DeepEP handle is passed to low-latency combine	both are tuple-like state	fields are interpreted under the wrong layout
SGLang normal output enters the low-latency runner path	hidden width can match	counts/masks and combine format disagree
combine() runs before dispatch completion	asynchronous return exists	expert runner reads incomplete receive rows
placement generation changes while state is live	source APIs may not name generation	return path addresses another placement
source comment is promoted to a latency result	no timing is required to compile code	unsupported performance claim

The executable evidence records six artifacts, 17 line spans, 80 symbol references, six source surfaces, five phase bindings, and twenty rejected one-field mutations. It leaves private production code, compiled kernel order, unnamed allocator state, profile measurements, and post-pin API changes public unknowns.

0x07 Topology-Aware Routing, Hierarchical Exchange, And Process Groups

Topology is not a performance annotation attached after dispatch. It changes which coordinate indexes a split vector, which paths consume intra-node or inter-node transport, where relay buffers live, and which communicator may legally execute the collective. It must nevertheless leave the Loop 3 route identity unchanged.

The executable Loop 8 fixture reuses all eight routes from Loop 3 under placement-v17. It adds two nodes, two NIC identities, four process groups, and a deliberately nontrivial expert-parallel peer order. No collective or network path was executed.

0x07A Freeze Physical Domains Before Estimating Cost

The fixture topology is:

Global rank	Node	GPU-local rank	Intra-node domain	NIC	Node relay
0	node-0	0	nvlink-domain-0	nic-0, direct	yes
1	node-0	1	nvlink-domain-0	nic-0, via rank 0	no
2	node-1	0	nvlink-domain-1	nic-1, direct	yes
3	node-1	1	nvlink-domain-1	nic-1, via rank 2	no

NVLink and RDMA are teaching transport classes here, not measured links. The pinned DeepEP README distinguishes NVLink for intra-node communication and RDMA for inter-node communication, but it does not prove this fixture’s path, bandwidth, rail choice, or relay schedule.

A topology record needs at least:

global rank
node and GPU-local rank
intra-node fabric domain
NIC identity and locality
node relay, if the schedule uses one
process-group identity and collective epoch
peer-slot order inside every process group
placement and topology versions

The topology version is separate from placement-v17. Moving an unchanged expert replica to another rank changes placement. Rewiring rank-to-NIC or process-group membership changes topology or communicator state. A graph or dispatch handle may need both identities.

0x07B Global Rank Is Not Communicator Peer Slot

The fixture’s expert-parallel group has the peer-slot order:

peer slot:   0  1  2  3
global rank: 2  0  3  1

Let the process-group map be

$$\pi_G(i)=\text{global rank stored at peer slot }i.$$

For a global-rank count matrix $C$, source rank $u$ must construct the split vector in peer-slot order:

$$s_u[i]=C_{u,\pi_G(i)},$$

and destination rank $v$ expects receive splits:

$$r_v[i]=C_{\pi_G(i),v}.$$

Loop 3’s global-rank matrix is still:

             destination global rank
source          0  1  2  3
rank 0         [1, 2, 1, 0]
rank 1         [1, 1, 1, 1]
rank 2         [0, 0, 0, 0]
rank 3         [0, 0, 0, 0]

The resulting source-0 split vector is not [1,2,1,0]. It is:

global destination order: [0,1,2,3] -> [1,2,1,0]
peer-slot order:           [2,0,3,1] -> [1,1,0,2]

All four send vectors are:

rank 0: [1,1,0,2]
rank 1: [1,1,1,1]
rank 2: [0,0,0,0]
rank 3: [0,0,0,0]

and destination rank 1’s source splits are [0,2,0,1], not its global-source column [2,1,0,0]. Confusing global rank with peer slot can preserve the total row count while sending every nonuniform segment to the wrong peer.

Zero-count ranks remain in every vector. Removing them changes the coordinate system even though they contribute no rows to this step.

0x07C One Route Relation, Three Physical Classes

For route row $r$ with source rank $o_r$ and destination rank $d_r$, define:

$$\kappa(r)= \begin{cases} \text{same-gpu}, & o_r=d_r,\\ \text{same-node-remote-gpu}, & N(o_r)=N(d_r),\ o_r\ne d_r,\\ \text{different-node}, & N(o_r)\ne N(d_r). \end{cases}$$

The eight immutable routes classify as:

same GPU:              r00, r07                  -> 2 rows
same node, remote GPU: r01, r02, r05             -> 3 rows
different node:        r03, r04, r06             -> 3 rows

Therefore:

$$f_{gpu}=\frac{2}{8}, \qquad f_{intra}=\frac{3}{8}, \qquad f_{inter}=\frac{3}{8}, \qquad f_{gpu}+f_{intra}+f_{inter}=1.$$

With $d=3$ BF16 values, one teaching row contains six activation bytes. The flat dispatch carries 18 intra-node bytes and 18 inter-node bytes; combine returns the same logical payload:

Flat path	Dispatch rows	Dispatch bytes	Round-trip bytes	Directed pairs per direction
same-node remote GPU	3	18	36	2
different node	3	18	36	3

These are logical activation values only. They exclude route metadata, count exchange, alignment, protocol, retransmission, relay storage, and link-layer transactions. They are not fabric counters.

0x07D A Logical Expert Group Is Not A Physical Domain

M6.2 owns group-limited or device-limited route selection. Its released-model trace showed that a contiguous expert-ID group is a logical eligibility group until a physical manifest proves otherwise. Loop 8 consumes the already-selected logical expert and resolves its physical destination.

The forbidden shortcut is:

$$\text{logical group ID} \not\equiv \text{device ID} \not\equiv \text{node ID} \not\equiv \text{process-group peer slot}.$$

A model can use group-limited routing while experts in one logical group are spread across nodes. Conversely, a runtime can colocate several logical groups inside one node. The route policy and placement policy can coordinate, but neither namespace may be inferred from the other.

0x07E Hierarchical Exchange Preserves Payload And Adds Relay Work

The fixture’s teaching schedule uses rank 0 as node-0 relay and rank 2 as node-1 relay. Inter-node dispatch becomes:

1. source rank -> source-node relay
2. source relay -> destination-node relay
3. destination relay -> final expert rank

For r03, rank 0 already is the source relay and rank 2 is the final expert rank, so only stage 2 carries a row. For r04, rank 1 first stages to rank 0, then rank 0 sends to rank 2, then rank 2 forwards to rank 3. For r06, only the source staging and inter-node stages are needed.

The exact dispatch ledger is:

source -> source relay:        2 rows
source relay -> dest relay:    3 rows
dest relay -> final rank:      1 row

Combine reverses source and destination responsibilities:

expert rank -> expert relay:   1 row
expert relay -> origin relay:  3 rows
origin relay -> origin rank:   2 rows

Thus the inter-node payload is preserved:

$$M^{hier}_{inter,roundtrip} = M^{flat}_{inter,roundtrip} =36\ \text{bytes},$$

while the teaching hierarchy adds six intra-node relay-row movements:

$$M^{hier}_{relay,added} =(2+1+1+2)\cdot 6 =36\ \text{bytes}.$$

What changes is the inter-node message-pair coordinate: flat dispatch plus combine use six directed source/destination rank pairs, while the relay schedule uses two directed relay-node pairs. Fewer inter-node pairs can enable coalescing; it does not prove fewer network bytes or lower latency.

0x07F Why Hierarchical Can Win Or Lose

A teaching decomposition is:

$$C_{flat} =C_{intra}+C_{inter}+C_{sync},$$ $$C_{hier} =C_{gather}+C_{inter}^{coalesced}+C_{scatter} +C_{extra\ sync}+C_{relay\ contention}.$$

Hierarchy can help when coalescing reduces small-message overhead, maps traffic to a better NIC rail, or avoids many cross-node peers. It can lose when relay copies, HBM traffic, barriers, queueing, or relay hot spots dominate. Peak link bandwidth cannot decide between the schedules by itself.

A valid comparison must hold constant:

route IDs and gates
final expert destinations
origin ranks for combine
payload and metadata schema
placement and checkpoint generation
process-group membership and collective epoch
warmup, batch phase, skew, and measurement window

Loop 11 owns measured flat-versus-hierarchical latency. Loop 8 proves only coordinates, conservation, and logical byte ledgers.

0x07G Process-Group Construction Is Part Of The Plan

The fixture binds four groups:

Group	Kind	Peer-slot order	Purpose
ep-group-0	full expert parallel	[2,0,3,1]	flat route transpose
ep-node-0	node-local EP	[1,0]	gather/scatter inside node 0
ep-node-1	node-local EP	[3,2]	gather/scatter inside node 1
ep-relays	inter-node relays	[2,0]	node-coalesced exchange

Every group uses collective epoch 12. Equal member sets with different peer orders are different coordinate systems. A placement snapshot must not retain an old peer-slot map after group reconstruction.

def build_topology_plan(routes, topology, process_groups):
    ep = process_groups.require("ep-group-0", epoch=topology.collective_epoch)
    assert set(ep.global_ranks) == set(topology.global_ranks)

    count_global = count_by_global_source_destination(routes)
    send_splits = {
        source: [count_global[source, ep.global_rank(peer)]
                 for peer in range(ep.size)]
        for source in topology.global_ranks
    }

    for node in topology.nodes:
        local = process_groups.require(node.local_group_id, epoch=ep.epoch)
        assert set(local.global_ranks) == set(node.member_ranks)
        assert node.relay_rank in local.global_ranks

    relays = process_groups.require("ep-relays", epoch=ep.epoch)
    assert set(relays.global_ranks) == {node.relay_rank for node in topology.nodes}

    return preserve_route_identity(
        classify_and_stage(routes, topology, send_splits)
    )

The helper intentionally receives immutable routes. It may stage an inter-node row through relays, but the final physical expert slot and inverse-combine identity remain those frozen in Loops 2 and 3.

0x07H What The Pinned Runtimes Establish

Pinned surface	Topology/process-group evidence	Scope limit
DeepEP README 01dc3aaa..., lines 71-72 and 131-151	distinguishes NVLink intra-node and RDMA inter-node communication; buffer binds a distributed process group	documentation and source-reported capabilities, not this relay schedule
Megatron dispatcher 0896be0f..., lines 81-89 and collective call sites	separates EP, expert-TP, and TPxEP process groups and indexes collectives through explicit group objects	does not make global rank equal group-local rank
Megatron inference dispatcher at the same pin	NVLS path requires an eligible NVLink and symmetric-memory domain	not a generic multi-node path
SGLang EP docs e76d0acd..., lines 73-75	backend construction and supported EP/TP relations are configuration-specific	compatibility changes after the pin
DeepSeek-V3 report	source-reports restricted communication fanout and overlap design	does not expose every production topology or packet schedule

The source contracts justify explicit topology and process-group identities. They do not justify claiming that every runtime uses one relay per node, that RDMA always means inter-node, or that a logical router group maps directly to a physical fabric domain.

0x07I Topology And Process-Group Failure Lab

Same-shape mistake	Violated contract	Consequence
global rank indexes a peer-slot split vector	communicator order owns split coordinates	valid total rows reach wrong peers
zero-count peer axes are removed	every participant uses the same group shape	collective split disagreement
logical expert group is treated as node ID	routing and placement namespaces differ	fanout or locality is misreported
one rank belongs to two nodes	topology domains form a partition	bytes and relay ownership are double-counted
node relay is not a node member	relay stages are node-local	cross-node path appears in an intra-node stage
node-local group crosses a node boundary	group membership must match topology	local collective uses the wrong fabric
relay group includes a nonrelay rank	one named relay per node	coalesced stage addresses a rank without relay state
collective epoch changes under a live handle	split vectors bind group generation	old peer slots index a rebuilt communicator
hierarchical stage changes final expert rank	staging cannot reroute semantics	another replica or expert executes the row
inter-node payload reduction is assumed	coalescing changes pairs, not useful rows	byte ledger silently drops accepted routes
added relay traffic is omitted	hierarchy materializes extra local stages	HBM and intra-node costs are understated
fewer pairs is called lower latency	time includes copies, sync, queueing, and contention	unsupported performance conclusion
NVLink/RDMA labels are treated as measurements	transport class is not a counter	source capability becomes fabricated bandwidth

The executable oracle freezes four ranks, two nodes, two NICs, four process groups, eight routes, three transport classes, flat and hierarchical byte ledgers, and twenty rejected one-field mutations. Actual collective algorithms, packetization, switch paths, relay buffers, contention, and latency remain public unknowns.

0x08 EPLB Replication, Rebalancing, And State Migration

Router balancing changes the demand presented by logical experts. Expert-parallel load balancing changes physical supply: how many compatible copies exist, where they reside, and which placement generation new work may use. The distinction is strict. Replicating logical expert $e$ does not add another term to the MoE sum. One accepted route still executes exactly one compatible copy of $E_e$.

The visual and executable fixture use the same one-layer load vector $[4,2,1,1]$, four ranks, two nodes, two expert groups, eight target physical slots, and immutable checkpoint-0042. They transcribe the public EPLB solver at commit d52c72d5...; migration and generation fencing are additional M6.6 runtime contracts, not behavior claimed from that solver.

0x08A Demand Is A Matrix With Provenance

For $L$ MoE layers and $E$ logical experts, a placement solver consumes:

$$W\in\mathbb{R}_{\ge 0}^{L\times E}, \qquad W_{l,e}=\text{estimated load of logical expert }e\text{ in layer }l.$$

The matrix needs a provenance record:

load unit: accepted rows, FLOPs, time, or weighted cost
observation window and phase: prefill, decode, or mixed
aggregation: count, mean, quantile, or moving estimate
router/checkpoint version
topology and placement version that produced the observation
timestamp or iteration interval

The Loop 9 teaching matrix is not predicted traffic. It is the exact accepted route count from the Loop 2 semantic handoff:

$$W=[4,2,1,1], \qquad \sum_e W_e=8.$$

Row counts are a useful first coordinate because they reconcile to route identity. They are not automatically equal to expert time. Different expert widths, dtypes, segment sizes, padding, and kernel tactics can give equal row counts different costs.

0x08B Why Replication Divides Load

Let $m_e\ge 1$ be the number of physical copies of logical expert $e$. If a separate replica selector distributes $W_e$ evenly, the ideal copy load is:

$$\ell_e(m_e)=\frac{W_e}{m_e}.$$

Adding one redundant copy changes that ideal from $W_e/m_e$ to $W_e/(m_e+1)$. The pinned replicate_experts routine therefore allocates each next physical slot to:

$$e^*=\arg\max_e\frac{W_e}{m_e}, \qquad m_{e^*}\leftarrow m_{e^*}+1.$$

This is a greedy discrete heuristic. It is understandable because it attacks the largest current ideal copy load. It is not a proof of a globally optimal placement once node constraints, indivisible routes, kernel shape, memory, migration, and tie behavior are included.

Tie behavior is observable. The pinned PyTorch max selects the first maximum index. For node-0 loads $[4,2]$ and two redundant copies:

Allocation state	$m_0$	$m_1$	ideal copy loads	next choice
initial	1	1	$[4,2]$	expert 0
after first copy	2	1	$[2,2]$	expert 0 by first-index tie
final	3	1	$[4/3,2]$	stop

The final maximum $2$ is higher than $4/3$. The example is not a bug in the transcription; it exposes why deterministic tie policy and greedy order belong to the placement artifact.

0x08C The Pinned Solver Has Three Hierarchical Stages

The public EPLB implementation applies three transformations when the number of expert groups is divisible by the number of nodes.

0x08C1 Pack Logical Groups To Nodes

For contiguous group $g$ with expert set $\mathcal{E}_g$:

$$G_g=\sum_{e\in\mathcal{E}_g}W_e.$$

balanced_packing sorts groups by descending $G_g$ and greedily assigns each group to the currently lightest node while enforcing exactly the same number of groups per node. In the teaching fixture:

group 0 = {E0,E1}, load 6 -> node 0
group 1 = {E2,E3}, load 2 -> node 1

Because there is one group per node, cardinality fixes that assignment. Group locality is preserved, but the node loads remain unequal.

0x08C2 Replicate Experts Inside Each Node

The solver applies the greedy ratio rule independently to each node’s logical experts. With four physical slots per node:

node 0: [E0,E1] loads [4,2] -> replica counts [3,1]
node 1: [E2,E3] loads [1,1] -> replica counts [2,2]

The physical-copy weight used by the next stage is:

$$\widehat{W}_q = \frac{W_{\phi(q)}}{m_{\phi(q)}}.$$

0x08C3 Pack Physical Copies To GPUs

Within each node, balanced_packing assigns the physical copies to equal-size GPU packs. With two copies per GPU, the exact target map is:

Global rank	Target slots	phy2log	replica ranks	ideal rank load
0	p0,p1	$[E1,E0]$	$[0,2]$	$2+4/3=10/3$
1	p2,p3	$[E0,E0]$	$[0,1]$	$4/3+4/3=8/3$
2	p4,p5	$[E2,E2]$	$[0,1]$	$1/2+1/2=1$
3	p6,p7	$[E3,E3]$	$[0,1]$	$1/2+1/2=1$

The source returns three related coordinates:

phy2log[physical_slot] -> logical expert
phyrank[physical_slot] -> rank among that logical expert's copies
logcnt[logical_expert] -> number of physical copies

It then constructs a padded inverse map:

log2phy[E0] = [p2, p3, p1]
log2phy[E1] = [p0, -1, -1]
log2phy[E2] = [p4, p5, -1]
log2phy[E3] = [p6, p7, -1]

-1 is padding, not a physical slot. A route-to-replica selector must use logcnt[e] or a validity mask before indexing log2phy[e].

0x08D Hierarchical And Global Are Different Contracts

The pinned entry point selects hierarchical placement when:

$$N_{groups}\bmod N_{nodes}=0.$$

Otherwise it invokes the same hierarchical routine with one group and one logical node, which yields global replication and GPU packing. For the exact same $W=[4,2,1,1]$ and eight physical copies, the global counterfactual is:

logcnt = [4,2,1,1]

rank 0: [E0,E0] -> ideal load 2
rank 1: [E1,E0] -> ideal load 2
rank 2: [E2,E1] -> ideal load 2
rank 3: [E3,E0] -> ideal load 2

Define the ideal rank-load ratio:

$$\Gamma_{rank} = \frac{\max_p L_p} {\frac{1}{P}\sum_p L_p}.$$

The teaching fixture gives:

Policy	replica counts	rank loads	$\Gamma_{rank}$	What the number omits
hierarchical	$[3,1,2,2]$	$[10/3,8/3,1,1]$	$5/3$	route-to-replica policy, locality benefit, migration, kernel time
global counterfactual	$[4,2,1,1]$	$[2,2,2,2]$	$1$	group-locality loss and resulting communication

Global is more balanced in this fixture’s ideal row-load coordinate. That does not prove lower communication or latency. Hierarchical preserves expert-group node locality. A complete decision requires both coordinates and measurement. The EPLB README’s prefill/decode guidance is source-specific guidance, not a universal phase law.

0x08E Source-Faithful Solver Trace

def source_faithful_rebalance(weight, Q, groups, nodes, gpus):
    # eplb.py lines 150-156: policy selection.
    if groups % nodes == 0:
        group_load = sum_contiguous_expert_groups(weight)
        group_to_node = balanced_packing(group_load, nodes)

        node_local_weight = reorder_by_group_pack(weight, group_to_node)
        phy2local, replica_rank, local_count = replicate_experts(
            node_local_weight,
            Q // nodes,
        )

        copy_weight = gather(node_local_weight / local_count, phy2local)
        copy_to_gpu = balanced_packing(copy_weight, gpus // nodes)
        phy2log = undo_group_and_copy_permutations(phy2local, copy_to_gpu)
    else:
        phy2log, replica_rank, local_count = hierarchical_rebalance(
            weight, Q, groups=1, nodes=1, gpus=gpus
        )

    log2phy = padded_inverse(phy2log, replica_rank, local_count)
    return phy2log, log2phy, local_count

The transcription preserves source ordering and fixed pack cardinality. It does not silently add a topology-aware route selector, migration planner, allocator, or measured cost model.

0x08F A Placement Map Is Not A Migration Plan

The solver output names what placement-v19 should contain. The runtime must still compare it with active placement-v18, reserve memory, materialize missing state, verify compatible parameters, publish the new generation, and drain old handles.

The toy expert is a BF16 SwiGLU with $d=2$ and $d_{ff}=3$. Its three matrices contain:

$$N_{expert} =3d d_{ff} =3\cdot2\cdot3 =18\ \text{elements},$$

so one teaching copy carries:

$$M_{expert}=18\cdot2=36\ \text{B}.$$

This is parameter payload only. It excludes allocator metadata, checkpoint framing, transport protocol, optimizer state, and temporary copy buffers.

The exact placement-v18 -> placement-v19 plan is:

Target rank	Target slot	Expert	Action	Source	Payload
0	p0	E1	same-node network copy	rank 1 q2	36 B
0	p1	E0	retain compatible storage	rank 0 q0	36 B retained
1	p2	E0	same-node network copy	rank 0 q0	36 B
1	p3	E0	same-GPU duplicate after p2	rank 1 target state	36 B local
2	p4	E2	retain compatible storage	rank 2 q4	36 B retained
2	p5	E2	same-GPU duplicate after p4	rank 2 retained state	36 B local
3	p6	E3	retain compatible storage	rank 3 q5	36 B retained
3	p7	E3	same-GPU duplicate after p6	rank 3 retained state	36 B local

Therefore:

retained copies:              3 -> 108 B state, no transfer
same-node network copies:     2 ->  72 B logical transfer payload
same-GPU duplicate copies:    3 -> 108 B local copy payload
old copies retired later:     3 -> 108 B state

The source and target resident ledgers are:

$$M_{v18}=6\cdot36=216\ \text{B}, \qquad M_{v19}=8\cdot36=288\ \text{B}.$$

If retained copies are immutable and shared by both generation records, peak unique parameter storage during overlap is:

$$M_{peak} =M_{v18}+M_{v19}-M_{retained} =216+288-108 =396\ \text{B}.$$

Adding old and new resident bytes without subtracting shared retained storage would double-count. Retiring old metadata before its handles drain would be a correctness failure, even when the underlying parameter storage is shared.

0x08G Publication Is A Generation State Machine

placement-v18 ACTIVE, one handle acquired
    -> collect load
    -> solve placement-v19 candidate
    -> reserve storage for five non-retained target copies
    -> materialize two network and three local copies
    -> verify logical expert, checkpoint, and parameter digest for all slots
    -> publish placement-v19 for new dispatches
    -> mark placement-v18 DRAINING
    -> reject retirement while refcount(v18) = 1
    -> combine and release dispatch-h18-0099
    -> retire placement-v18 metadata and unreferenced old copies

The transition can be written as:

candidate = solve_eplb(observed_load, topology, replica_budget)
reservation = reserve_nonretained_state(candidate, active_generation)
materialize_and_verify(candidate, reservation, checkpoint="checkpoint-0042")

registry.publish(candidate, generation="placement-v19")
registry.mark_draining("placement-v18")

assert not registry.can_retire("placement-v18")  # live h18-0099
combine_with_generation(handle="dispatch-h18-0099", generation="placement-v18")
registry.release("dispatch-h18-0099")
assert registry.can_retire("placement-v18")
registry.retire("placement-v18")

Publication before every target digest verifies can route new work into stale or partial expert state. In-place replacement is worse: an old dispatch handle can combine with new inverse maps while every tensor shape still matches.

0x08H Replica Selection Is A Separate Runtime Decision

EPLB returns replica sets. It does not assign each route row to one member of that set. For accepted route $r$ targeting logical expert $e_r$, a runtime selector must choose exactly one compatible target:

$$A_v(r)\in\rho_v(e_r), \qquad \sum_{q\in\rho_v(e_r)}\mathbf{1}[A_v(r)=q]=1.$$

def select_physical_replica(route, generation, predicted_rank_load):
    candidates = generation.log2phy[route.logical_expert_id]
    candidates = candidates[: generation.logcnt[route.logical_expert_id]]
    candidates = [q for q in candidates if generation.compatible(route, q)]
    q = min(candidates, key=lambda slot: (
        remote_class(route.origin_rank, generation.rank_of(slot)),
        predicted_rank_load[generation.rank_of(slot)],
        slot,
    ))
    return q  # logical expert and combine gate remain unchanged

Locality-first, least-load, round-robin, hash-based, and queue-aware selectors can produce different physical traffic from the same EPLB map. Their policy, state, and update cadence must be named before post-placement bytes are claimed.

0x08I Alternative Objectives And Load Estimators

The pinned solver is one design point. Alternatives change the owned object:

Choice	Changed contract	Benefit	New failure surface
greedy redundant copies	discrete $m_e$ by $W_e/m_e$	simple deterministic allocation	tie/order sensitivity; no joint migration objective
hierarchical group placement	group-to-node locality before GPU balance	can reduce cross-node group fanout	constrains node balancing
global placement	ignores expert groups	larger balancing search space	can increase inter-node routes
exact min-max integer solve	jointly minimizes a declared load objective	auditable optimum for small instances	solver cost and objective mismatch
topology-aware multi-objective	adds traffic and domain penalties	can price locality	requires route distribution and link model
migration-aware optimization	penalizes state movement and peak overlap	avoids churn	can retain a temporarily poor placement
phase-specific plans	separate prefill and decode estimates	matches different row distributions	transition and cache/graph complexity

A general teaching objective is:

$$J = \lambda_1\max_p L_p +\lambda_2 M_{inter-node} +\lambda_3 M_{migration} +\lambda_4 M_{resident} +\lambda_5 C_{churn}.$$

This is an M6.6 scheduler formulation, not an EPLB paper equation. Its terms cannot be compared until units and normalization are declared.

Load estimation alternatives include trailing counts, exponentially weighted counts, phase-specific histograms, kernel-time estimates, and predictive models. EPLB consumes the matrix but does not predict it. A moving estimate:

$$\widehat{W}^{(t)} = \beta\widehat{W}^{(t-1)} +(1-\beta)W^{(t)}$$

adds lag and update cadence. A burst can make an otherwise exact placement solve obsolete before migration finishes.

0x08J Pinned Source Contract

Source span at d52c72d...	Exact use here	Explicit non-claim
balanced_packing, lines 5-41	descending greedy packing with equal item count	globally optimal weighted partition
replicate_experts, lines 44-71	repeated weight / logcnt maximum and replica rank	route assignment or measured speedup
rebalance_experts_hierarchical, lines 74-129	group-to-node, node-local replication, copy-to-GPU pipeline	weight migration or process-group rollout
rebalance_experts, lines 131-162	policy condition and phy2log/log2phy/logcnt outputs	load prediction, generation fencing, or rollback
EPLB README at the same commit	redundant-expert motivation and source-specific hierarchical/global guidance	a theorem that one policy wins each phase

The local oracle is dependency-free and transcribes only those integer and greedy operations. It compares the hierarchical output with an explicit global counterfactual, then adds a separately labeled teaching migration state machine. No GPU kernel, distributed collective, allocator, or benchmark is executed.

0x08K Replication And Migration Failure Lab

Same-shape mistake	Violated contract	Consequence
load matrix loses router/checkpoint provenance	placement input is versioned evidence	stale workload drives a valid but irrelevant solve
row count is called kernel time	load unit is explicit	equal counts hide shape-dependent stragglers
EPLB map is treated as route assignment	placement and replica selection are separate	post-placement traffic is fabricated
log2phy padding -1 is indexed	logcnt owns validity	invalid slot reaches dispatch metadata
replica rank is confused with global rank	namespaces differ	compatible copy maps to wrong device
tie policy changes silently	greedy order is artifact state	replica counts or pack order drift
global balance number ignores group locality	objective is incomplete	lower ideal load can raise communication
new generation publishes before digest verification	all slots must be compatible	requests execute stale or partial expert state
retained bytes are counted as network traffic	state and transfer ledgers differ	migration traffic is overstated
retained storage is counted twice at peak	aliases are explicit	memory admission falsely rejects candidate
source copy is retired before materialization completes	copy lifetime is incomplete	target digest is corrupt or unavailable
old map is overwritten in place	live handles bind old generation	shape-correct combine returns wrong expert rows
old generation retires at nonzero refcount	drain fence is absent	use-after-free of maps or weights
replica means ensemble member	physical copies preserve one function	expert output is multiplied or averaged incorrectly
computed payload bytes are labeled measured	analytical and observed evidence collapse	unsupported bandwidth or latency claim

The executable fixture rejects twenty one-field mutations across upstream hashes, source spans, load provenance, divisibility, parameter state, migration identity, generation order, and claim scope.

0x09 Communication-Compute Overlap Is A Dependency Schedule

Overlap is not a boolean optimization flag. It is a schedule over tasks, streams, buffers, and completion events. A diagram that merely draws communication and compute on top of each other has not yet proved that the consumer sees complete data, that two tasks do not collide on one serialized resource, or that a buffer remains live until its final consumer.

0x09A Pinned Interfaces Own Events, Not This Teaching Timeline

The pinned DeepEP README at revision 01dc3aaa exposes dispatch and combine operations that return handles and events; consumers wait before using the corresponding results. It also documents BF16 input or an FP8 data-and-scale tuple at that revision. The pinned Megatron dispatcher at revision 0896be0f contains explicit waits around shared-expert FC1/FC2 and routed all-to-all work.

Those source spans prove that event ordering, handles, shared-path overlap, and precision metadata are concrete runtime concerns. They do not report the durations below. The Loop 10 oracle uses integer analytical time units so the dependency arithmetic can be replayed without presenting a GPU or network benchmark.

0x09B From The MoE Dataflow To An Event DAG

For a task $i$ with start $s_i$, end $e_i$, and declared duration $d_i$,

$$e_i-s_i=d_i, \qquad s_i\ge \max_{j\in\operatorname{pred}(i)}e_j.$$

Tasks sharing the same teaching stream may not overlap. Tasks on different streams may overlap only after their own predecessors finish. The exact fixture has eleven tasks:

Task	Units	Stream	Must wait for
route plan	1	control	nothing
pack 0, pack 1	1 each	pack	route plan
dispatch 0, dispatch 1	3 each	communication	corresponding pack
shared path	5	compute	route plan
expert 0, expert 1	4, 2	compute	corresponding dispatch
combine 0, combine 1	2 each	communication	corresponding expert
reduce	1	compute	both combines and shared path

The dependencies preserve the runtime meaning:

route metadata -> pack_j -> dispatch_j -> expert_j -> combine_j
route metadata -> shared path -----------------------------> reduce
combine_0 + combine_1 ------------------------------------> reduce

dispatch_0 does not authorize expert_1; expert_0 does not authorize combine_1; and completion of one route slot does not authorize the token reduction while another route slot is still outstanding.

0x09C Serialized And Overlapped Schedules, Derived

Placing every task one after another gives

$$T_{\mathrm{serialized}} =\sum_i d_i =25\ \text{analytical units}.$$

The executable alternative is:

Task	Serialized $[s,e)$	Overlapped $[s,e)$
route plan	[0,1)	[0,1)
pack 0	[1,2)	[1,2)
pack 1	[2,3)	[2,3)
dispatch 0	[3,6)	[2,5)
dispatch 1	[6,9)	[5,8)
shared path	[9,14)	[1,6)
expert 0	[14,18)	[6,10)
expert 1	[18,20)	[10,12)
combine 0	[20,22)	[10,12)
combine 1	[22,24)	[12,14)
reduce	[24,25)	[14,15)

The overlapped makespan is the latest task end:

$$T_{\mathrm{overlap}}=\max_i e_i=15, \qquad H_{\mathrm{optimistic}}=25-15=10.$$

The union of compute intervals intersects the union of communication intervals for eight units. This is not the same as saying that all ten hidden critical path units are communication time. Pack/control movement and changed idle gaps also affect the schedule.

Most importantly, $25\rightarrow15$ is not a measured speedup. It is the result of one exact dependency fixture under fixed integer durations. On real hardware, communication can consume SMs, HBM bandwidth, copy engines, network injection credits, and workspace that the expert kernels also need. Concurrent tasks can both become slower.

0x09D Buffers Have A Longer Contract Than Their Producers

An event does not merely release a successor. It also closes a buffer lifetime. The overlapped fixture keeps nine half-open lifetimes:

Buffer	Allocated	Last consumer ends	Release
send 0, send 1	1, 2	5, 8	5, 8
receive 0, receive 1	5, 8	10, 12	10, 12
expert output 0, 1	10, 12	12, 14	12, 14
shared output	6	15	15
combined segment 0, 1	12, 14	15, 15	15, 15

The producer’s launch is not a release fence. For example, send_1 remains live until dispatch_1 ends at unit 8, and shared_out remains live from unit 6 through the final reduction at unit 15. Early reuse can produce a shape-correct but nondeterministic wrong answer.

0x09E Alternative Overlap Designs

The same semantics permit several physical schedules:

Design	Unit of readiness	Advantage	New burden
bulk all-to-all	whole rank payload	simplest completion contract	no expert starts early
peer/chunk pipeline	peer segment or chunk	earlier compute and return	more events, buffers, and small-message risk
shared-path overlap	routed dispatch vs always-on expert	independent useful work	shared and routed paths may contend for SM/HBM
microbatch pipeline	communication for batch $m+1$ vs compute for $m$	coarse reusable schedule	extra activation state and pipeline bubbles

No option dominates without the phase, topology, row histogram, resource profile, and tail-latency objective measured together.

0x09F Overlap Failure Lab

Same-shape mistake	Violated contract	Consequence
expert starts when dispatch is launched	completion, not launch, owns readiness	expert reads partially written rows
two communication tasks overlap on one serialized stream	stream exclusion	fixture invents impossible concurrency
send buffer releases after pack	dispatch is still a consumer	payload is overwritten in flight
shared output releases when FC2 ends	final reduction still consumes it	routed/shared sum is corrupted
optimistic units are called milliseconds	analytical and measured evidence differ	fictitious latency result
timeline overlap is called speedup	resource contention is omitted	deployment regresses despite a prettier trace

0x0A Precision Is Data Plus Metadata

Precision is not one dtype attached to the layer. Router decisions, dispatched activation data, scale metadata, expert arithmetic, gate weights, returned rows, and token accumulation can each use different representations. Every representation change adds an owner, shape, lifetime, and error boundary.

0x0AA One Frozen Payload Scenario

To avoid silently changing traffic after Loop 9 replication, the precision fixture deliberately returns to the pre-rebalance placement-v17 scenario: six remote route rows, eight useful local-plus-remote rows, and physical-slot counts

$$R_{\mathrm{remote}}=6, \qquad \mathbf n=[2,2,1,1,1,1].$$

This is a counterfactual payload ledger for those six rows. It is not the post-placement-v19 route traffic, because the article has not defined a new route-to-replica assignment for that generation.

0x0AB Deriving BF16 And FP8 Payloads

With activation width $d$, BF16 element width $b_{16}=2$ bytes, FP8 data width $b_8=1$ byte, and FP32 scale width $b_s=4$ bytes, pure BF16 data is

$$M_{\mathrm{BF16}} =R_{\mathrm{remote}}d b_{16}.$$

One FP32 scale per route row gives

$$M_{\mathrm{FP8,row}} =R_{\mathrm{remote}}d b_8 +R_{\mathrm{remote}}b_s.$$

With one scale per $g=128$ elements per row,

$$M_{\mathrm{FP8,block128}} =R_{\mathrm{remote}}d b_8 +R_{\mathrm{remote}}\left\lceil\frac{d}{128}\right\rceil b_s.$$

The exact logical payload results are:

Width $d$	BF16 data	FP8 + one scale/row	FP8 + one scale/128 values/row
3	36 B	42 B	42 B
16	192 B	120 B	120 B
7,168	86,016 B	43,032 B	44,352 B

The $d=3$ row is intentional: low-precision data does not guarantee a smaller total logical payload. Six FP32 scales cost 24 bytes, so 18 FP8 data bytes plus scales exceed 36 BF16 data bytes. Real transports may add alignment, route metadata, protocol headers, packetization, allocator state, and physical link transactions; none is included above.

0x0AC Scale Tensor Ownership And Row Trace

For the per-row contract, quantization produces a pair rather than one tensor:

source activation rows      X_remote_fp8   [6, d]
source scale rows           S_remote_fp32  [6, 1]
stable route order          shared by X and S

dispatch(X_remote_fp8, S_remote_fp32, route_handle)
    -> destination X_fp8 segment [n_q, d]
    -> destination S_fp32 segment [n_q, 1]

dequantize with the scale at the same route row
    -> expert arithmetic
    -> BF16 return row in this teaching contract

If payload rows are permuted by destination but scales remain token-major, all shapes still match while every dequantized row can use the wrong scale. Scale shape, permutation, destination split, and dispatch handle therefore belong to one generation.

0x0AD Alignment Is Local Executed Work, Not Remote Traffic

Let the local expert runner require row alignment $a=4$. Each physical slot executes

$$\widetilde n_q =a\left\lceil\frac{n_q}{a}\right\rceil, \qquad R_{\mathrm{executed}}=\sum_q\widetilde n_q.$$

For $[2,2,1,1,1,1]$, all six segments round to four rows:

$$R_{\mathrm{useful}}=8, \qquad R_{\mathrm{executed}}=24, \qquad R_{\mathrm{align\ waste}}=16.$$

Those sixteen fringe rows are local runner work in this fixture. They are not sixteen additional remote network rows. Padding before transport is a different design and would require a different payload oracle.

0x0AE Router, Gate, Combine, And Accumulation Dtypes

Stage	Loop 10 teaching representation	Why it is separate
router logits and top-k comparison	FP32	perturbations can change discrete expert identity
dispatch activation	BF16 or FP8 plus FP32 scales	transport payload and quantization error
expert arithmetic	backend-owned	weight/activation/accumulator policy is a kernel contract
returned expert row	BF16	the pinned interface permits a higher-precision combine path
route gate	FP32	preserves the declared mixture coefficient
token accumulation	FP32	sums multiple returned route slots

This table is one explicit policy, not a universal recommendation. It does prevent an invalid algebraic shortcut. The MoE output is

$$y_t=\sum_{j=1}^{k}g_{t,j}E_{e_{t,j}}(h_t).$$

The gate may be fused with the expert output before return or with the origin reduction after return, subject to finite-precision ordering. It cannot generally be moved to the expert input: for a nonlinear expert, $E(gh)\ne gE(h)$. The counterexample $E(x)=x^2$ gives $E(gx)=g^2E(x)$, not $gE(x)$.

0x0AF Precision Failure Lab

Same-shape mistake	Violated contract	Consequence
FP8 bytes omit scales	representation is data plus metadata	traffic is understated
scales use token-major order after expert-major permutation	scale and row share route identity	finite but corrupted expert input
$d=3$ still assumes FP8 is smaller	metadata can dominate tiny rows	false universal compression claim
alignment rows are added to remote-row count	local execution and transport differ	network payload is fabricated
gate is applied before nonlinear expert	mixture algebra changes	different model is executed
router precision change is judged only by logit error	top-k is discrete	small error can reroute the token
lower-precision mode is called quality-safe	interface support is not model validation	unsupported quality claim

0x0B Parallel Axes Change Physical Ownership

Parallel folding is not merely a different set of group sizes. EP determines which rank group owns expert identities; ETP determines whether one physical expert copy is complete on one rank or sharded across several ranks. Changing ETP therefore changes the unit named by the placement table.

0x0BA Two Exact Four-Rank Layout Manifests

The fixture has global ranks [0,1,2,3]. Its active layout keeps complete expert copies on rank slots:

active placement-v19
attention: TP=2, CP=1, DP=2, PP=1
MoE:       ETP=1, EP=4, EDP=1, PP=1

EP group:  [0,1,2,3]
ETP groups:[0] [1] [2] [3]
physical expert unit: one complete copy on one rank slot

The candidate shards each physical expert over two ranks:

candidate placement-v20-required
attention: TP=2, CP=1, DP=2, PP=1
MoE:       ETP=2, EP=2, EDP=1, PP=1

rank coordinates (EP, ETP):
    r0=(0,0)  r1=(0,1)  r2=(1,0)  r3=(1,1)
EP groups:   [0,2] [1,3]
ETP groups:  [0,1] [2,3]
physical expert unit: one copy spanning one ETP group

For these declared coordinate systems,

$$W_{\mathrm{attention}} =TP\cdot CP\cdot DP\cdot PP =2\cdot1\cdot2\cdot1=4,$$ $$W_{\mathrm{MoE,active}} =ETP\cdot EP\cdot EDP\cdot PP =1\cdot4\cdot1\cdot1=4,$$ $$W_{\mathrm{MoE,candidate}} =2\cdot2\cdot1\cdot1=4.$$

Equal products prove only that both manifests cover four ranks. They do not prove that their ownership, traffic, or checkpoint shards are compatible.

0x0BB Why placement-v19 Cannot Be Reused

In the active layout, physical_slot_id=p4 can name one complete expert copy on one destination rank. In the candidate, the analogous copy has two weight shards, two ETP coordinates, an intra-expert collective contract, and a group rather than a single destination rank.

old identity:
    (logical expert, replica rank, destination rank, complete parameter digest)

new identity:
    (logical expert, replica rank, ETP group, shard coordinate,
     per-shard digest, collective generation)

Reusing placement-v19 would preserve an integer slot id while changing what that slot physically means. The candidate must materialize, verify, publish, drain, and retire as a new placement-v20 generation. Loop 9’s generation state machine still applies; only the parameter-state schema becomes richer.

0x0BC Redistribution Is A Tensor Contract, Not A Product Formula

Attention and MoE can use different rank coordinates. The hidden rows crossing their boundary may need an all-gather, reduce-scatter, all-to-all, local view, or fused redistribution depending on which tensor dimensions each rank owns. The axis products above cannot determine the byte count.

To derive redistribution bytes, a runtime needs at least:

per-rank input tensor shape and strides
source shard dimension and coordinate
destination shard dimension and coordinate
replication vs reduction semantics
dtype and scale metadata
padding, sequence, and microbatch shape
process-group peer order and topology

The Loop 10 result is therefore explicit:

attention-to-MoE redistribution bytes:
    unknown_without_per_rank_shard_manifest

This is stronger than guessing zero from equal world factors or guessing one full hidden tensor from group sizes alone.

0x0BD Training, Prefill, And Decode Integration

Phase	Axis pressure	Runtime consequence
training	optimizer/gradient ownership plus expert parameter shards	EDP and ETP affect reductions, checkpointing, and migration state
prefill	many routed rows, larger expert GEMMs	EP can aggregate useful work; redistribution and all-to-all volume grow
one-token decode	few rows per request, repeated latency path	large EP/ETP groups can expose tiny collectives and synchronization tails
mixed serving	prefill and decode share ranks	one fixed folding can be poor for one phase; phase split needs explicit state transfer

Alternatives include complete experts with larger EP, sharded experts with ETP, node-local EP plus cross-node data replication, phase-specific pools, or a fixed global layout that sacrifices one operator to avoid redistribution. Each choice changes placement, process groups, checkpoint shards, and release evidence; none is a free configuration toggle.

0x0BE Parallel-Axis Failure Lab

Same-shape mistake	Violated contract	Consequence
axis products equal four, so layouts are called equivalent	factorization does not define ownership	wrong process group or shard is selected
EP and ETP groups both use adjacent-rank assumption	each group has its own coordinate rule	tokens or shards cross the wrong peers
placement-v19 reused after ETP 1 to 2	physical expert unit changed	complete-copy digest is applied to a shard group
route handle omits collective generation	combine must invert the same groups	output returns through incompatible communicator
redistribution bytes inferred from group sizes	per-rank tensor manifest is absent	fabricated traffic ledger
one folding is declared optimal for all phases	row shape and cadence differ	decode tail or prefill throughput regresses

The executable Loop 10 fixture rejects twenty one-field mutations spanning upstream identity, source revision, task duration and dependency, stream collision, buffer release, precision metadata, alignment, dtype, parallel factorization, placement generation, and forbidden performance claims.

0x0C A Runtime Is Released As One Compatibility Object

Loops 2-10 isolated identities, tensors, traffic, kernels, phase modes, source interfaces, topology, placement, and precision. A deployment cannot select those coordinates independently. A graph compiled for one placement and a handle produced by another can both have valid shapes while encoding different expert ownership.

The release object is therefore a tuple, not a binary:

$$\mathcal{R} = (C,P,T,G,A,D,B,Q),$$

where $C$ is checkpoint identity, $P$ placement generation, $T$ topology, $G$ process-group generation, $A$ route and handle ABIs, $D$ dtype roles, $B$ phase/backend selection, and $Q$ graph manifests. A change to any coordinate requires either compatibility evidence or a new release identity.

0x0CA The Exact Release Fixture

The checked release is m66-placement-v19-canary-01. It binds checkpoint-0042, placement-v19, topology-v3, groups-v19-ep4-etp1, BF16 dispatch/combine, FP32 gate/accumulation, and three phase paths:

Phase path	Dispatcher	Expert runner	Graph state
variable prefill	throughput-variable	grouped GEMM	eager
fixed-bucket decode	low-latency	padded and masked	decode.bucket4.v19
oversize decode	low-latency eager	grouped GEMM	explicit fallback

Every digest in the teaching fixture is synthetic, but every field is a real compatibility question. Loop 11 also hashes all nine Loop 2-10 reports. A release built after one report changes is a different evidence object even if its human-readable configuration is unchanged.

@dataclass(frozen=True)
class MoEReleaseBundle:
    release_id: str
    checkpoint_version: str
    checkpoint_digest: str
    placement_generation: str
    placement_digest: str
    topology_version: str
    process_group_generation: str
    route_abi: str
    dispatch_handle_abi: str
    precision_roles: dict[str, str]
    phase_backends: dict[str, BackendManifest]
    graph_manifest_digest: str
    upstream_report_digests: tuple[str, ...]

placement-v20-required is deliberately not the candidate. Loop 10 changes EP4/ETP1 into EP2/ETP2, so one physical expert becomes an ETP-sharded object. Without a per-rank checkpoint-shard manifest, the redistribution bytes and the target weight identity are both unknown. Equal world-size products do not make that alternative releasable.

0x0CB Runtime Objects Carry The Release Coordinates

@dataclass(frozen=True)
class PlacementSnapshot:
    release_id: str
    placement_generation: str
    checkpoint_version: str
    process_group_generation: str
    phy2log: Tensor
    log2phy_offsets: Tensor
    log2phy_values: Tensor
    physical_rank: Tensor

@dataclass
class DispatchHandle:
    release_id: str
    placement_generation: str
    process_group_generation: str
    origin_token: Tensor
    route_slot: Tensor
    inverse_permutation: Tensor
    send_counts: Tensor
    recv_counts: Tensor
    expert_offsets: Tensor
    gate_weights: Tensor
    dispatch_ready: Event
    combine_ready: Event

Compression of these fields is allowed; loss of their identity is not. The minimum cross-stage condition is

$$G_{dispatch}=G_{handle}=G_{expert}=G_{combine}, \qquad \mathcal{R}_{dispatch}=\mathcal{R}_{combine}.$$

0x0CC Forward Blueprint With Admission And Fallback

def moe_forward(hidden, runtime, phase):
    release = runtime.release_registry.acquire_active()
    placement = release.placement.acquire_snapshot()
    try:
        backend = release.select_backend(phase, hidden.shape[0])
        ids, gates = runtime.router(hidden)
        plan = runtime.plan_routes(hidden, ids, gates, placement)
        assert_route_conservation(plan)

        if not backend.graph_key_matches(release, plan):
            backend = release.eager_fallback_for(phase)

        recv_rows, handle = backend.dispatch(plan, release)
        assert handle.release_id == release.release_id
        assert handle.placement_generation == placement.placement_generation
        assert handle.process_group_generation == release.process_group_generation
        handle.dispatch_ready.wait()

        expert_out = backend.run_experts(recv_rows, handle.expert_offsets)
        output = backend.combine(expert_out, handle)
        handle.combine_ready.wait()
        assert output.shape == hidden.shape
        return output
    finally:
        release.placement.release_snapshot(placement)
        runtime.release_registry.release(release)

The eager path is not a silent approximation. It must preserve route identity, gate ownership, checkpoint state, and placement generation. Only the execution representation changes.

0x0CD Route And Combine Assertions

expected_routes = hidden.shape[0] * topk
assert plan.accepted_routes + plan.dropped_routes == expected_routes
assert sum(plan.send_counts) == plan.accepted_routes
assert sum(handle.recv_counts) == recv_rows.shape[0]
assert handle.inverse_permutation.numel() == plan.accepted_routes

def combine(returned_rows, handle, local_token_count, d):
    token_major = returned_rows[handle.inverse_permutation]
    weighted = token_major * handle.gate_weights[:, None]
    out = zeros([local_token_count, d], dtype=weighted.dtype)
    scatter_add_(out, handle.origin_token, weighted)
    return out

If a runner already applies the gate to each expert output, combine must not apply it again. The release ABI therefore owns weighting location as well as tensor shape.

0x0D Evaluation Requires Cells, Denominators, And Evidence Classes

A list of metric names is not an evaluation protocol. Each result needs a release ID, workload cell, numerator, denominator, observation window, quantile convention, and evidence class. Loop 11 uses four deterministic synthetic cells only to execute this accounting contract:

Cell	Steps	Attempted / accepted / dropped routes	Remote / inter-node	Graph replay / fallback
prefill uniform	40	320 / 320 / 0	240 / 120	0 / 0
decode fixed bucket	100	200 / 200 / 0	120 / 60	96 / 4
decode hot expert	80	160 / 158 / 2	120 / 70	70 / 10
mixed burst	20	120 / 120 / 0	90 / 50	10 / 10

These are fixture observations, not a benchmark. They let the oracle derive five non-interchangeable ratios:

$$\begin{aligned} r_{drop} &= \frac{N_{dropped}}{N_{attempted}}, &r_{remote} &= \frac{N_{remote}}{N_{accepted}},\\ r_{inter} &= \frac{N_{inter-node}}{N_{accepted}}, &r_{graph} &= \frac{N_{replay}}{N_{replay}+N_{fallback}},\\ r_{fallback} &= \frac{N_{fallback\ steps}}{N_{steps}}. \end{aligned}$$

For the exact aggregate, $N_{attempted}=800$, $N_{accepted}=798$, and $N_{dropped}=2$. Therefore

$$r_{drop}=0.0025, \quad r_{remote}=\frac{570}{798}\approx0.714286, \quad r_{inter}=\frac{300}{798}\approx0.375940,$$

while $r_{graph}=176/(176+24)=0.88$ and $r_{fallback}=24/240=0.10$. Dividing remote routes by attempted routes would answer a different question. Omitting the denominator would make the ratio unreproducible.

0x0DA Tail Statistics Need A Convention

The fixture includes twenty positive synthetic iteration units. For sorted observations $x_{(1)}\le\cdots\le x_{(n)}$, it uses nearest rank:

$$Q_q=x_{(\lceil qn\rceil)}.$$

That gives p50=11, p95=23, and max=24 analytical units. These values are not milliseconds, TTFT, TPOT, or GPU timing. A production evaluation must name clock boundaries, warmup, synchronization, concurrency, sample count, and the hardware/software manifest before publishing latency quantiles.

0x0DB Telemetry Schema And Cardinality

Metric	Numerator	Denominator	Required bounded labels
moe_route_drop_ratio	dropped routes	attempted routes	release, phase
moe_remote_route_ratio	remote routes	accepted routes	release, phase
moe_inter_node_route_ratio	inter-node routes	accepted routes	release, phase, topology
moe_graph_replay_ratio	graph replays	replay + fallback	release, phase, graph
moe_fallback_step_ratio	fallback steps	all steps	release, phase, reason
route-output mismatch counter	mismatches	counter	release, phase
stale-handle acceptance counter	accepted stale handles	counter	release, placement generation
count-conservation counter	failures	counter	release, process-group generation

Request IDs, token IDs, and raw prompt strings do not belong in metric labels. Per-expert diagnostics need bounded expert IDs or sampled traces; otherwise the observability system becomes a second unbounded routing workload.

0x0DC Promotion Is A Conjunction, Not A Score

The exact teaching gate is

$$\begin{aligned} Promote={}&HashOK\land ManifestOK\land FaultsOK\land RollbackOK\\ &\land N_{shadow}\ge100\land N_{canary}\ge200\\ &\land r_{drop}\le0.005\land r_{fallback}\le0.15\\ &\land N_{route\ mismatch}=N_{stale\ accept}=N_{count\ failure}=0. \end{aligned}$$

The fixture passes that logical gate. It does not prove production performance, availability, or quality. A real release must add named latency, throughput, memory, numerical-parity, quality, and SLO gates from measured artifacts.

The benchmark matrix must still cross phase, token count, uniform/hot/bursty routing, one-node/multi-node topology, EP/ETP layout, payload dtype, runner, placement, and graph mode. One balanced prefill batch cannot authorize decode or a hot-expert production distribution.

0x0E Fault Injection And Rollback Close The Runtime Contract

Happy-path parity does not exercise ownership failures. The Loop 11 oracle requires ten faults and one declared safe result for each:

Injected fault	Required safe outcome
Loop 2-10 report hash drifts	reject build
process-group generation differs	reject load
peer route counts disagree	abort before payload exchange
inverse permutation is corrupt	reject output before publish
stale handle reaches new active generation	reject handle and replan
graph key differs	explicit eager fallback
FP8 scale count differs	reject payload before expert compute
runner parity differs	block canary
completion dependency is absent	reject schedule
active candidate faults	roll back to prior generation

“Abort before payload exchange” requires a control agreement before ranks enter incompatible collectives. A local assertion after one rank has entered all-to-all cannot rescue the group.

0x0EA Rollback Is Ordered State Transfer

The candidate trace is

BUILT -> VERIFIED -> LOADED -> SHADOW -> CANARY -> ACTIVE
       -> DRAINING -> RETIRED

The exact fault path is:

stop new candidate dispatch
    -> activate loaded placement-v18 for new dispatch
    -> drain three placement-v19 handles to zero
    -> release placement-v19 graph resources
    -> retire placement-v19
    -> rollback complete

The lifetime rules are

$$N^{new}_{dispatch}(P_{candidate})=0 \quad\text{after the stop fence},$$ $$RefCount(P_{candidate})=0 \quad\text{before graph release and retirement}.$$

Activating the prior generation before draining the candidate keeps new work flowing while old handles complete against the snapshot they acquired. The two generations may coexist only if their weights, maps, process groups, graph resources, and refcounts remain separately owned.

def rollback(candidate, prior, registry):
    registry.stop_new_dispatch(candidate.release_id)
    assert prior.loaded
    assert prior.checkpoint_version == candidate.checkpoint_version
    registry.activate_for_new_dispatch(prior.release_id)

    registry.wait_for_refcount(candidate.release_id, expected=0)
    candidate.graph_pool.release_all()
    registry.retire(candidate.release_id)

Retiring first is use-after-free. Draining first without activating fallback is an availability outage. Reusing a placement-v19 graph under placement-v18 is an ABI violation even when the graph bucket shape matches.

0x0EB Failure Closure Table

Same-shape mistake	Violated release contract	Consequence
logical expert ID used as GPU ID	semantic and physical namespaces collapsed	wrong owner or out-of-range slot
dispatch and combine use different maps	handle generation not immutable	correct shape, wrong route output
expert-major output indexed by token ID	inverse metadata absent	route rows permute silently
alignment rows counted as drops	execution and semantic ledgers collapsed	false quality alarm
graph fallback clips oversize routes	fallback is not semantic parity	missing expert contributions
gate applied before and after expert output	weighting owner duplicated	systematic scale error
replica publishes before digest agreement	release manifest incomplete	request-dependent parameters
candidate accepts work after stop fence	acquisition fence violated	rollback never drains
graph pool releases at nonzero refcount	graph and handle lifetimes diverge	in-flight use-after-free
synthetic units labeled latency	evidence class changed	fabricated benchmark claim

The executable fixture rejects twenty one-field mutations across upstream hashes, release identity, EP/ETP ownership, graph digest, blocked-v20 state, route conservation, telemetry denominators, sample gates, fault outcomes, rollout order, drain state, and claim scope. It executes no collective or GPU kernel and reports no production latency, throughput, quality, or availability.

0x0F Claim Status And Evidence

Claim	Status	Evidence or qualification
expert parallelism requires token dispatch to expert-owning ranks and a return/combine path	source-local and externally verified	A21/A29, DeepSpeed-MoE, Megatron Core documentation
variable expert loads create irregular local matrix shapes	externally verified	MegaBlocks and grouped-GEMM runtime designs
block-sparse execution can support dropless MoE without global capacity padding	externally verified	MegaBlocks; performance results remain source-reported
all-to-all can be a major MoE cost	externally verified	DeepSpeed-MoE and Megatron documentation; exact fraction is workload-specific
current runtimes expose separate throughput and low-latency dispatch paths	externally verified, version-pinned	current DeepEP/SGLang documentation checked 2026-07-15
redundant experts can balance hot logical experts	externally verified	official EPLB repository and DeepSeek runtime practice
hierarchical placement always wins for prefill and global placement always wins for decode	not universal	EPLB README gives scenario guidance, not a theorem
router precision can affect discrete expert identity	technically motivated and implementation-supported	Megatron Core guidance; sensitivity is model-specific
quantized dispatch always improves end-to-end latency	uncertain if generalized	depends on conversion, scale traffic, topology, and quality constraints
overlap always reduces latency	false if generalized	contention and dependency shape can reverse gains

0x10 Handoffs

M6.6 closes the path from router output to distributed expert execution. It also exposes questions that belong elsewhere:

How should the router learn or constrain load?
    -> M6.1 and M6.2

How should low-rank updates be designed and served?
    -> M6.3, M6.4, and M6.5

How do expert and adapter states interact in routed adaptation?
    -> later M6 routed-adapter treatment

How should payload, expert weights, and deployment artifacts be quantized?
    -> M8

How do scheduler and KV lifetimes interact with decode?
    -> M5

0x11 Publish Gate

Check	Status
Logical expert identity is separated from physical placement and replicas	pass
Semantic, conservation, identity, and placement-version invariants are explicit	pass
Routed-row tensors, counts, offsets, permutation, and inverse mapping are traced	pass
Communication bytes distinguish total routed rows from remote routed rows	pass
Padded, grouped-GEMM, and block-sparse execution are compared	pass
Capacity, alignment, and block padding are distinct	pass
Prefill and decode runtime modes are treated separately	pass
Dispatch and combine are represented as a paired stateful operation	pass
Current engine APIs are version-qualified rather than universalized	pass
Topology-aware, group-limited, and hierarchical exchange choices are explicit	pass
Expert replication includes memory, load, mapping, and transition costs	pass
Communication-compute overlap includes dependencies and contention	pass
Router, dispatch, combine, and accumulation precision are separated	pass
EP, ETP, TP, DP, CP, and PP composition is acknowledged	pass
Runtime data structures and route-accounting assertions are concrete	pass
Evaluation covers phase, skew, topology, precision, placement, and tail latency	pass
Failure injection includes stale maps, count mismatch, metadata corruption, and graph fallback	pass

0xFF References

• Transformer Series A21 - MoE
• Transformer Series A29 - DeepSeek MoE
• M6.1 - Routed Capacity
• M6.2 - Expert Parallelism and DeepSeek MoE
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Switch Transformers
• DeepSpeed-MoE
• Tutel: Adaptive Mixture-of-Experts at Scale
• MegaBlocks: Efficient Sparse Training with Mixture-of-Experts
• FastMoE: A Fast Mixture-of-Expert Training System
• DeepSeekMoE
• DeepSeek-V3 Technical Report
• Megatron Core MoE documentation, version 0.17.0
• Megatron Core inference token dispatcher, version 0.17.0
• DeepEP README at 01dc3aaa
• EPLB README at d52c72d5
• SGLang expert-parallelism documentation at e76d0acd