A tensor-level study of routing outside FFN experts: Mixture of Attention Heads, SwitchHead, Mixture-of-Depths, causal route prediction, per-layer KV consequences, MoD+MoE composition, and staged multimodal sparse training.
A mechanism-level study of mixtures of LoRA adapters: routing granularity, joint versus post-hoc training, HydraLoRA factor sharing, router collapse, static and dynamic composition, continual low-rank state, and multi-adapter runtime integration.
A cost- and shape-oriented study note on Mixture-of-Experts layers, dense FFN baselines, top-k routing, token dispatch, expert capacity, overflow, and total-vs-active parameter ledgers.
A systems-oriented study note on scaled Mixture-of-Experts routing, load balancing, expert parallelism, all-to-all communication, shared experts, fine-grained experts, device-limited routing, and DeepSeek-style MoE.