#experts • Tags • Transformer System

2 posts · Transformer System

Tag: #experts

M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch

🗓 2026-07-09 • Transformer System • ⏱ 100 min read

A cost- and shape-oriented study note on Mixture-of-Experts layers, dense FFN baselines, top-k routing, token dispatch, expert capacity, overflow, and total-vs-active parameter ledgers.

#numinens #transformer #moe #mixture-of-experts #routing #experts

Read →

M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing

🗓 2026-07-09 • Transformer System • ⏱ 87 min read

A systems-oriented study note on scaled Mixture-of-Experts routing, load balancing, expert parallelism, all-to-all communication, shared experts, fine-grained experts, device-limited routing, and DeepSeek-style MoE.

#numinens #transformer #moe #deepseek #routing #load-balancing

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #experts

Tag: #experts

M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch

M6.2 - Expert Parallelism and DeepSeek MoE: Load Balancing, Shared Experts, and Fine-Grained Routing