A tensor-to-cluster study of MoE execution: logical and physical experts, variable-size dispatch, all-to-all communication, grouped and block-sparse kernels, prefill/decode modes, expert replication, topology, overlap, precision, observability, and failure recovery.
A tensor-level study of routing outside FFN experts: Mixture of Attention Heads, SwitchHead, Mixture-of-Depths, causal route prediction, per-layer KV consequences, MoD+MoE composition, and staged multimodal sparse training.
A cost- and shape-oriented study note on Mixture-of-Experts layers, dense FFN baselines, top-k routing, token dispatch, expert capacity, overflow, and total-vs-active parameter ledgers.
A systems-oriented study note on scaled Mixture-of-Experts routing, load balancing, expert parallelism, all-to-all communication, shared experts, fine-grained experts, device-limited routing, and DeepSeek-style MoE.