#capacity • Tags • Transformer System

2 posts · Transformer System

Tag: #capacity

M3.6 — Dense and Gated MLPs: Channel Mixing, Training, and Runtime

🗓 2026-07-13 • Transformer System • ⏱ 55 min read

A derivation- and implementation-oriented study of Transformer MLPs: position-wise channel mixing, activation and GLU variants, SwiGLU width matching, backward flow, fused execution, knowledge interpretations, and memory alternatives.

#numinens #transformer #ffn #mlp #feed-forward-network #swiglu

Read →

M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch

🗓 2026-07-09 • Transformer System • ⏱ 100 min read

A cost- and shape-oriented study note on Mixture-of-Experts layers, dense FFN baselines, top-k routing, token dispatch, expert capacity, overflow, and total-vs-active parameter ledgers.

#numinens #transformer #moe #mixture-of-experts #routing #experts

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #capacity

Tag: #capacity

M3.6 — Dense and Gated MLPs: Channel Mixing, Training, and Runtime

M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch