#all-to-all • Tags • Transformer System

2 posts · Transformer System

Tag: #all-to-all

M6.6 - MoE Runtime Systems: Placement, Dispatch, Kernels, and Topology

🗓 2026-07-15 • Transformer System • ⏱ 94 min read

A tensor-to-cluster study of MoE execution: logical and physical experts, variable-size dispatch, all-to-all communication, grouped and block-sparse kernels, prefill/decode modes, expert replication, topology, overlap, precision, observability, and failure recovery.

#numinens #transformer #moe #expert-parallelism #all-to-all #deepep

Read →

M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing

🗓 2026-07-09 • Transformer System • ⏱ 87 min read

A systems-oriented study note on scaled Mixture-of-Experts routing, load balancing, expert parallelism, all-to-all communication, shared experts, fine-grained experts, device-limited routing, and DeepSeek-style MoE.

#numinens #transformer #moe #deepseek #routing #load-balancing

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #all-to-all

Tag: #all-to-all

M6.6 - MoE Runtime Systems: Placement, Dispatch, Kernels, and Topology

M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing