#multi-token-prediction • Tags • Transformer System

2 posts · Transformer System

Tag: #multi-token-prediction

M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime

🗓 2026-07-13 • Transformer System • ⏱ 45 min read

A formula-, tensor-, code-, and runtime-level study of DeepSeek-V3 multi-token prediction modules, shared embeddings and heads, teacher forcing, released weight structure, KV ownership, and speculative serving.

#numinens #transformer #deepseek #deepseek-v3 #mtp #multi-token-prediction

Read →

M7.4 - Multi-Token Prediction Foundations: Objectives, Gradients, and Prediction Heads

🗓 2026-07-09 • Transformer System • ⏱ 37 min read

A derivation- and implementation-oriented study of multi-token prediction objectives, shifted labels, shared-trunk gradients, prediction-head designs, training memory, and the boundary between denser supervision and faster inference.

#numinens #transformer #mtp #multi-token-prediction #training #prediction-heads

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #multi-token-prediction

Tag: #multi-token-prediction

M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime

M7.4 - Multi-Token Prediction Foundations: Objectives, Gradients, and Prediction Heads