X1.1 - Unitization and Geometry Metadata Across Modalities
A derivation- and implementation-oriented study of how images, documents, audio, video, tools, and actions become addressable units with aligned spatial, temporal, and structural metadata.
Tag: #tensor-shapes
M3.6 — Dense and Gated MLPs: Channel Mixing, Training, and Runtime
A derivation- and implementation-oriented study of Transformer MLPs: position-wise channel mixing, activation and GLU variants, SwiGLU width matching, backward flow, fused execution, knowledge interpretations, and memory alternatives.
M3.7 — Residual Stream, Normalization, and Block Assembly
A derivation- and implementation-oriented reconstruction of residual learning, LayerNorm and RMSNorm geometry, backward flow, Pre-LN/Post-LN ordering, depth scaling, dropout, and Transformer block variants.
M3.8 — Output Head and Decoding Policy
A derivation- and implementation-oriented reconstruction of vocabulary projection, logits and loss gradients, weight tying, score processors, temperature, top-k/top-p, beam search, and local token-selection policy.
M6.3 - LoRA Foundations: Low-Rank Updates, Gradients, and Target Modules
A derivation- and implementation-oriented study of LoRA: low-rank update geometry, tensor shapes, forward and backward passes, initialization, scaling, optimizer state, target-module selection, and the limits of the low-rank assumption.
M3.2 — Multi-Head and Cross-Attention: Projection, Layout, and Variants
A code-, shape-, and systems-oriented study of packed multi-head attention, target-by-source cross-attention, head composition, conditional head routing, concatenation, and output projection.