|

M6.8 - Conditional Compute Beyond FFN Experts: Attention, Depth, and Modalities

A tensor-level study of routing outside FFN experts: Mixture of Attention Heads, SwitchHead, Mixture-of-Depths, causal route prediction, per-layer KV consequences, MoD+MoE composition, and staged multimodal sparse training.