#loss • Tags • Transformer System

Press Esc to close

2 posts · Transformer System

Tag: #loss

M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation

🗓 2026-07-24 • Transformer System • ⏱ 100 min read

A source-audited study of neural scaling experiments: measurement contracts, power laws, joint model-data loss surfaces, compute-optimal allocation, paper-contract comparison, fitting, and uncertainty.

#numinens #transformer #scaling-laws #compute-optimal #training #inference

Read →

M3.9 — Training Execution: Teacher Forcing, Masks, Loss, and Parallelism

🗓 2026-06-21 • Transformer System • ⏱ 94 min read

A derivation- and code-path reconstruction of Transformer training: shifted supervision, causal/padding/packed/loss masks, cross-entropy and smoothing, accumulation, Adam, Noam scheduling, mixed precision, distributed parallelism, checkpoints, and reproducibility.

#numinens #transformer #training #teacher-forcing #causal-mask #padding-mask

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #loss

Tag: #loss

M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation

M3.9 — Training Execution: Teacher Forcing, Masks, Loss, and Parallelism