M4.1 - Transformer Cost Model: Parameters, Activations, FLOPs, and Memory Traffic
A code- and shape-oriented study note on Transformer cost: parameters, activations, FLOPs, memory traffic, prefill, decode, and why FLOPs alone do not predict runtime.
A code- and shape-oriented study note on Transformer cost: parameters, activations, FLOPs, memory traffic, prefill, decode, and why FLOPs alone do not predict runtime.