M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime

M7.6 - DeepSeek MTP: Sequential Modules, Teacher Forcing, and Speculative Runtime

0x00 Overview And Background

M7.4 derived independent future-token heads. M7.5 followed a post-hoc feature drafter around a frozen target.

DeepSeek-V3 uses a third design:

main-model hidden state
    + shifted ground-truth token embedding during training
    -> normalize both paths
    -> concatenate
    -> project back to model width
    -> run an additional Transformer block
    -> use the shared output head
    -> predict one additional future token

The owner question is:

How does this module preserve a complete causal prediction chain, how are its tensors shifted and trained, and what extra weights, caches, and verification steps appear if the module is retained at inference?

Background knowledge: DeepSeek MTP makes future prediction a causal depth chain

Independent MTP heads share one trunk state but do not consume one another’s selected tokens. EAGLE uses a post-hoc drafter around target features. DeepSeek-V3 integrates a different structure during pretraining: each additional MTP depth receives the previous depth’s hidden state and the shifted token embedding needed to continue the causal chain.

main model:
    predicts the next token from the ordinary hidden sequence

MTP depth k:
    previous-depth state + token x_(i+k) -> state h_i^k

shared head:
    maps h_i^k to the prediction for x_(i+k+1)

training:
    ground-truth shifted embeddings provide teacher-forced depth inputs

This gives richer pretraining supervision, but inference acceleration is optional. Retaining the modules introduces their own weights, draft state, candidate policy, target verification, and commit rules.

0x00A Historical Lineage And Owner Contract

Multi-token prediction did not begin with DeepSeek-V3. Blockwise parallel decoding trained future predictors and validated the longest acceptable prefix; classical MTP later attached several independent future-token heads to a shared trunk; EAGLE made the proposal path autoregressive in target-feature space. DeepSeek-V3 takes another step: it puts a sequence of shallow causal modules inside pretraining, and each depth receives the token that closes the preceding prediction edge.

That history matters because the phrase “predict several tokens” hides three different factorizations:

The owner contract is therefore narrower than “all multi-token prediction”:

this article owns:
    DeepSeek's sequential module equation
    depth and token-coordinate alignment
    paper-exact MTP objective and gradient ownership
    teacher-forced training versus self-fed rollout
    released V3 module count and checkpoint aliases
    discard-at-inference and self-speculative state contracts

this article consumes:
    M2.2 embedding/output-head identity
    M6.2 DeepSeekMoE routing and logical traffic
    M6.6 physical expert placement, collectives, and kernels
    M7.1 exact verification and correction semantics
    M7.4 general MTP supervision

this article defers:
    EAGLE/HASS/EAGLE-3 derivations to M7.5
    general proposal topology to M7.7
    production admission, rollout, and rollback policy to M7.8
    MLA derivations to M5.8
    MoE routing and logical traffic to M6.2; physical execution to M6.6

The DeepSeek-V3 report motivates MTP as denser supervision and possible representation pre-planning. It reports benchmark improvements in controlled small- and large-MoE ablations while discarding MTP at inference, and separately reports speculative results when the one additional prediction is retained. Those are two different empirical contracts. Neither proves that every sequential MTP design improves quality or serving speed.

0x01 Claim Status And Scope

The equations below describe the general depth-$D$ architecture. Source-reported DeepSeek-V3 values are called out separately so a general design is not confused with the released configuration.

0x01A Immutable Architecture, Checkpoint, And State Manifest

Before compiling a batch or loading a checkpoint, freeze the identities that give every tensor its meaning:

The state rows identify the round-entry coordinate. In §0x09, output is P + [f], target K/V covers only P, and parent_hidden belongs to the end of P. Publication advances these state fields together; architecture and checkpoint identities do not change with that advancement.

For the open DeepSeek-V3 weights, D_ckpt = 1, main layers are 0..60, and extra layer 61 belongs to MTP. The weight documentation reports embed_tokens and shared_head as shared; a runtime may materialize local module objects, but it must restore the documented identity or an explicitly equivalent read-only serving arrangement.

@dataclass(frozen=True)
class MTPManifest:
    target_checkpoint: str
    num_main_layers: int
    num_checkpoint_mtp_layers: int
    mtp_layer_ids: tuple[int, ...]
    embedding_owner: str
    output_head_owner: str
    storage_dtype: str
    compute_dtype: str
    cache_layout: str
    source_commit: str

def validate_manifest(m):
    assert m.num_checkpoint_mtp_layers == len(m.mtp_layer_ids)
    assert m.mtp_layer_ids == tuple(
        range(m.num_main_layers,
              m.num_main_layers + m.num_checkpoint_mtp_layers)
    )
    assert m.embedding_owner == "main.embed_tokens"
    assert m.output_head_owner == "main.norm+lm_head"

The official DeepSeek inference demo is an important negative observation: its Transformer iterates the configured target layers and applies the ordinary final norm and vocabulary head; it does not construct an MTP proposer. The weights document MTP, but the demo is not evidence that speculative MTP is automatically active. Runtime claims below use the pinned vLLM implementation trace where an actual MTP module path is required.

0x02 Start From The Main Model State

Let the token sequence be:

$$x_{1:T}$$

The main model produces hidden representations:

$$h_i^{(0)} = \operatorname{MainModel}(x_{\le i})_i, \qquad h_i^{(0)} \in \mathbb{R}^{d}$$

The ordinary shared output head predicts the next token:

$$p_{i+1}^{(0)} = \operatorname{softmax} \left(W_U \operatorname{Norm}_{out}(h_i^{(0)})\right)$$

The first MTP module predicts one token farther. It must know two things:

1. what the main model represented at position $i$;
2. which token occupies the next position $x_{i+1}$.

During training, $x_{i+1}$ is ground truth. The first MTP module consumes $h_i^{(0)}$ and the embedding of $x_{i+1}$, then predicts $x_{i+2}$.

Generalizing to prediction depth $k$:

input representation:
    h_i^(k-1)

shifted token input:
    x_(i+k)

output representation:
    h_i^k

predicted label:
    x_(i+k+1)

That one-index gap is the complete causal-chain contract.

0x03 Derive One MTP Module

Source: DeepSeek-V3 Technical Report, Figure 3. Each MTP module normalizes and concatenates the previous-depth state with the shifted token embedding, projects back to model width, applies one Transformer block, and reuses the shared embedding and output head.

The $k$-th module normalizes the previous-depth hidden state and shifted token embedding separately:

$$\bar h_i^{(k-1)} = \operatorname{RMSNorm}_{h,k} \left(h_i^{(k-1)}\right)$$ $$\bar e_{i+k} = \operatorname{RMSNorm}_{e,k} \left(E(x_{i+k})\right)$$

It concatenates the two $d$-wide vectors:

$$c_i^{(k)} = \left[ \bar h_i^{(k-1)}; \bar e_{i+k} \right] \in \mathbb{R}^{2d}$$

A learned projection returns to model width:

$$u_i^{(k)} = M_k c_i^{(k)}, \qquad M_k \in \mathbb{R}^{d \times 2d}$$

The MTP Transformer block preserves causal sequence dependencies:

$$H_{1:T-k}^{(k)} = \operatorname{TransformerBlock}_k \left(U_{1:T-k}^{(k)};\ M_{causal}\right)$$

The shared output head predicts the additional token:

$$p_{i+k+1}^{(k)} = \operatorname{softmax} \left( W_U \operatorname{Norm}_{out} (h_i^{(k)}) \right)$$

Renderer-safe chain:

h_bar = hnorm_k(h_i^(k-1))
e_bar = enorm_k(embed(x_(i+k)))
u_i^k = eh_proj_k(concat(h_bar, e_bar))
h_i^k = transformer_block_k(u_<=i^k)
p_(i+k+1)^k = softmax(shared_head(h_i^k))

0x03A Why Normalize Before Concatenation

The previous-depth hidden state and token embedding come from different pathways and may have different activation scales. Separate normalization prevents one half of the concatenation from dominating the fusion projection solely because of magnitude.

It also gives the module separate learned scale parameters:

hnorm:
    calibrates the representation path

enorm:
    calibrates the shifted-token path

The official DeepSeek weight documentation names these parameters hnorm and enorm.

0x03B Why Project $2d \rightarrow d$

Passing a (2d)-wide residual stream into an ordinary model-width Transformer block would change every attention and MLP matrix shape. The eh_proj matrix localizes the width expansion to one fusion operation:

$$[d] + [d] \xrightarrow{concat} [2d] \xrightarrow{eh\_proj} [d]$$

The projection has:

$$P_{eh\_proj}=2d^2$$

before bias, if present.

For $d=7168$, a dense $d \times 2d$ matrix contains about 102.8 million parameters. This is only one component; the additional Transformer block dominates or shares the architecture-specific attention/MoE costs.

0x03C Derive The Full Module Schedule

RMSNorm is not an informal “make the scales similar” operation. For a vector $z\in\mathbb{R}^{d}$ and learned gain $g$:

$$\operatorname{RMSNorm}_{g}(z) = g\odot \frac{z}{\sqrt{d^{-1}\sum_{j=1}^{d}z_j^2+\epsilon}}$$

The two branches use different gains because their distributions and gradient roles differ. The projected state can be expanded coordinate by coordinate:

$$u_{i,r}^{(k)} = \sum_{j=1}^{d}M_{k,rj}\bar h_{i,j}^{(k-1)} + \sum_{j=1}^{d}M_{k,r,d+j}\bar e_{i+k,j}$$

This makes two facts visible. Concatenation does not mix information by itself; $M_k$ performs the first cross-path mixing. And the projection is not required to preserve either input as an identity path. Any residual connection begins inside TRM_k, after fusion.

For a pre-norm decoder block, a conceptual schedule is:

$$a^{(k)} = u^{(k)}+\operatorname{Attention}_k \left(\operatorname{Norm}_{a,k}(u^{(k)});M_{causal}\right)$$ $$h^{(k)} = a^{(k)}+\operatorname{FFN/MoE}_k \left(\operatorname{Norm}_{f,k}(a^{(k)})\right)$$

The official weight layout says the extra hidden layer has the same structure as a main hidden layer. For DeepSeek-V3 that means its MLP path is not safely approximated as a small dense head: it inherits architecture-specific MLA/MoE state and routing. M5.8 owns MLA’s internal derivation; M6.2 owns routing and logical traffic, while M6.6 owns physical placement, collectives, kernels, and measured runtime.

The numerical contract is also layered. The released weights are FP8 with block scales, while the report keeps embeddings, output head, normalization, attention, master weights, gradients, and optimizer state in higher precision during training. Therefore dtype = FP8 is not an executable MTP contract:

checkpoint storage:
    FP8 weights plus scale metadata where documented

fusion inputs:
    dequantized/compute representations selected by the runtime

RMS reductions and gains:
    higher-precision contract

attention, routing, output head:
    architecture/runtime-specific mixed precision

loss reduction and optimizer state:
    training-framework contract, not inferred from checkpoint bytes

0x03D Alternative Fusion And Depth Designs

The DeepSeek equation is one design choice, not a mathematical necessity:

DeepSeek-V3’s paper defines depth-specific modules in the general case, while the released depth-one checkpoint cannot empirically distinguish depth-specific from recurrent weight reuse. A runtime that repeatedly applies the single released MTP layer is implementing a recurrent speculative rollout policy; it is not revealing additional checkpoint depths that do not exist.

0x04 Tensor Alignment Across Prediction Depths

Assume tokens: [B,S] and main_hidden: [B,S,d].

At depth (k=1):

previous hidden:
    main_hidden[:, 0:S-1, :]        [B, S-1, d]

shifted tokens:
    tokens[:, 1:S]                  [B, S-1]

MTP-1 output:
    hidden_1                        [B, S-1, d]

labels:
    tokens[:, 2:S]                  [B, S-2]

One output row at the end has no $x_{i+2}$ label. A clean implementation can trim inputs to S-2 before the block or compute S-1 rows and mask the final loss row.

At general depth $k$:

previous-depth rows used for labeled loss:
    h^(k-1)[:, 0:S-k-1, :]          [B, S-k-1, d]

shifted token input:
    tokens[:, k:S-1]                [B, S-k-1]

labels:
    tokens[:, k+1:S]                [B, S-k-1]

This convention directly aligns every module output row with one valid label.

0x04A Concrete Sequence Trace

For:

tokens = [A, B, C, D, E, F]

Depth 2 does not independently read the main-model representation. It reads the depth-1 representation for the same base position and the next shifted token.

Source-local visual reused from A33. Colors follow one base-token row across the main objective and two pedagogical MTP depths; the released V3 checkpoint itself has one MTP module.

0x04B Shape-Checked Forward

Here shared_head means normalization then vocabulary projection, not a bare LM projection. This teaching interface returns logits; the pinned vLLM SharedHead.forward instead returns normalized hidden state. The decoder block below returns residual-completed, pre-final-norm state for the paper-style depth equations.

class MTPOutputHead(nn.Module):
    def __init__(self, norm, head):
        super().__init__()
        self.norm = norm
        self.head = head

    def forward(self, pre_final_norm):
        return self.head(self.norm(pre_final_norm))


class MTPModule(nn.Module):
    def __init__(self, model_block, shared_embedding, shared_head, d):
        super().__init__()
        self.embed_tokens = shared_embedding
        self.enorm = RMSNorm(d)
        self.hnorm = RMSNorm(d)
        self.eh_proj = Linear(2 * d, d, bias=False)
        self.block = model_block
        self.shared_head = shared_head

    def forward(self, previous_hidden, shifted_token_ids, positions, mask, kv=None):
        # previous_hidden:  [B, L, d]
        # shifted_token_ids:[B, L]
        e = self.enorm(self.embed_tokens(shifted_token_ids))
        h = self.hnorm(previous_hidden)
        fused = self.eh_proj(concat([h, e], dim=-1))
        current_hidden, kv = self.block(fused, positions, mask, kv=kv)
        logits = self.shared_head(current_hidden)
        return current_hidden, logits, kv

Assertions:

assert previous_hidden.shape[:2] == shifted_token_ids.shape
assert e.shape == previous_hidden.shape
assert fused.shape == previous_hidden.shape
if manifest.requires_physical_training_aliases:
    assert storage_id(mtp.embed_tokens.weight) == storage_id(main_model.embed_tokens.weight)
    assert storage_id(mtp.shared_head.norm.weight) == storage_id(main_model.norm.weight)
    assert storage_id(mtp.shared_head.head.weight) == storage_id(main_model.lm_head.weight)
else:
    # A frozen serving replica may be semantically shared but physically copied.
    assert tensor_fingerprint(mtp.embed_tokens.weight) == manifest.embedding_fingerprint
    assert tensor_fingerprint(mtp.shared_head.norm.weight) == manifest.output_norm_fingerprint
    assert tensor_fingerprint(mtp.shared_head.head.weight) == manifest.head_fingerprint

Training-time physical sharing should be checked by object/storage identity, not only equal values. A serving runtime that intentionally materializes read-only replicas must instead prove checkpoint/version equality and account for the extra memory; it must not claim the training memory optimization.

0x04C Compile Semantic Rows Before Gathering Tensors

Shape arithmetic alone does not prove alignment. A packed sequence can produce three equal-length tensors while gathering a shifted token or label from the next document. In this compact layout, base is a stable semantic address in one packed stream; previous_depth_row is a physical row in the preceding depth’s tensor. Carry previous_bases[j] beside that tensor so row j keeps its identity even after compaction or reordering. Never infer the next row index from the original token position.

Compile state support before selecting loss rows. A state needs every teacher-forced input from base through base + depth to be active and in the same document occurrence. Its next label may be unavailable: retain that state with loss_valid=False, rather than deleting context because it has no loss. Here valid means input availability, not a task-specific supervision mask; any additional loss-only mask is applied after the module without pruning state support.

def compile_mtp_rows(tokens, segment_ids, valid, depth, previous_bases):
    # One packed stream. Segment IDs identify document occurrences, not classes.
    n = len(tokens)
    if len(segment_ids) != n or len(valid) != n:
        raise ValueError("unaligned packed inputs")
    if type(depth) is not int or depth < 1:
        raise ValueError("depth must be positive")
    if any(type(b) is not int or not 0 <= b < n for b in previous_bases):
        raise ValueError("invalid previous base")
    if len(set(previous_bases)) != len(previous_bases):
        raise ValueError("duplicate previous base")
    previous_row = {base: row for row, base in enumerate(previous_bases)}

    rows = []
    for base in range(n):
        shifted = base + depth
        if shifted >= n:
            continue
        # State support: the whole teacher-forced input chain must exist.
        if not all(valid[j] and segment_ids[j] == segment_ids[base]
                   for j in range(base, shifted + 1)):
            continue
        if base not in previous_row:
            raise ValueError("required predecessor state is missing")
        label = shifted + 1
        loss_valid = (label < n and valid[label]
                      and segment_ids[label] == segment_ids[base])
        rows.append({
            "base": base,
            "previous_depth_row": previous_row[base],
            "shifted_token_index": shifted,
            "label_index": label if loss_valid else None,
            "loss_valid": bool(loss_valid),
            "position": base,
            "segment": segment_ids[base],
        })
    return rows

Only after this compilation should the program gather:

previous_hidden <- h^(k-1)[previous_depth_row]  # physical row lookup
shifted_token   <- tokens[shifted_token_index]  # original packed address
position        <- position_ids[position]     # semantic position, not row index
label           <- tokens[label_index]         # only for loss_valid rows

For two four-token segments, the state rows at depth one have bases [0,1,2,4,5,6]; their supervised subset is [0,1,4,5]. At depth two the state bases are [0,1,4,5], gathered from physical rows [0,1,3,4], while the supervised bases are only [0,4]. After every module, carry its output row identities forward as state_bases = [row["base"] for row in rows].

The old loss-only layout had depth-one bases [0,1,4,5]: treating base 4 as its physical row 4 was out of range. Fixing that one number is insufficient; the explicit mapping must survive every depth and any permitted row ordering. A missing predecessor is an error, never permission to fall back to a different row. The interval check also prevents a row from reappearing across a padding hole or an intervening segment. Loss-only masking must not create such holes.

Zeroing an invalid row’s loss after the forward pass is insufficient if that row already attended to another packed segment. The segment rule must affect both row selection and the causal attention graph.

The paper uses one-based inclusive notation and may write an EOS-like $t_{T+1}$ target. The executable article fixture uses zero-based, half-open arrays. The conversion is:

0x05 Training Objective And Gradient Paths

For depth $k$, define the set of valid semantic rows $\mathcal{V}_k$ after padding, packing, and boundary checks. The source paper writes its per-depth objective with a fixed $1/T$ denominator:

$$\mathcal{L}_{MTP,paper}^{(k)} = -\frac{1}{T} \sum_{i\in\mathcal{V}_k} \log p_{i+k+1}^{(k)}(x_{i+k+1})$$

This is a paper contract, not the only possible software reduction. A masked-mean implementation often uses the following expression on nonempty support ($|\mathcal{V}_k|>0$):

$$\mathcal{L}_{MTP,valid}^{(k)} = -\frac{1}{|\mathcal{V}_k|} \sum_{i\in\mathcal{V}_k} \log p_{i+k+1}^{(k)}(x_{i+k+1})$$

The two are not equivalent when the number of valid rows shrinks with depth or varies across packed batches:

$$\mathcal{L}_{MTP,paper}^{(k)} = \frac{|\mathcal{V}_k|}{T} \mathcal{L}_{MTP,valid}^{(k)}$$

An empty support cannot be divided by its row count. Declare an empty-depth policy, such as a zero contribution retained in the $D$-depth average used by the training sketch below. Skipping an empty depth and renormalizing over only nonempty depths is a different weighting policy. The identity above applies only when the masked mean is defined.

Thus the fixed-$T$ form implicitly downweights deeper or heavily masked depths. Changing to a valid-row mean changes the objective unless $\lambda_k$ is adjusted. A reproducible training manifest records the numerator mask, the denominator, and whether reduction happens before or after data-parallel aggregation.

The MTP objective averages across $D$ additional depths and applies weight $\lambda$:

$$\mathcal{L}_{MTP} = \frac{\lambda}{D} \sum_{k=1}^{D} \mathcal{L}_{MTP}^{(k)}$$

The total training objective is:

$$\mathcal{L} = \mathcal{L}_{main} + \mathcal{L}_{MTP}$$

DeepSeek-V3 reports this source-specific schedule:

first 10T training tokens:
    lambda = 0.3

remaining 4.8T training tokens:
    lambda = 0.1

This schedule is an externally verified DeepSeek-V3 setting, not a universal MTP default.

Renderer-safe objective:

paper per-depth:
    L_mtp[k] = -(1 / T) * sum_(i in V_k) log p_k(i+k+1)[x_(i+k+1)]

overall:
    L = L_main + (lambda / D) * sum_(k=1..D) L_mtp[k]

alternative masked mean:
    divide by |V_k|; this changes depth weighting and must be declared

0x05A Dense Supervision Is Not Free

At depth $k$, the number of supervised rows is $|\mathcal{V}_k|$. Across a general depth-$D$ chain:

$$N_{MTP\ labels} = \sum_{k=1}^{D}|\mathcal{V}_k|$$

For one unpadded length-$T$ sequence without an explicit EOS extension, $|\mathcal{V}_k|=\max(T-k-1,0)$. Exhausted horizons have zero labels, not a negative row count. In general:

$$N_{MTP\ labels} = \sum_{k=1}^{D}\max(T-k-1,0)$$

When $1\le D\le T-1$, the nonnegative terms reduce to the polynomial:

$$N_{MTP\ labels} = D(T-1)-\frac{D(D+1)}{2}$$

For example, T=2, D=3 has counts [0, 0, 0]; the unrestricted polynomial would incorrectly give -3. Packed segments and padding instead use the compiler’s actual valid-label sets, not this single-sequence shortcut.

Dense labels add useful supervision, but every depth also executes fusion, one decoder block, the shared vocabulary projection, saved activations or recomputation, and backward. A first-order training ledger is:

$$C_{train} \approx C_{main,fwd+bwd} + \sum_{k=1}^{D} C_{MTP_k,fwd+bwd} + C_{shared\ head,extra}$$

The shared head saves parameters; it does not remove its extra matrix multiplications over MTP rows. For a very large vocabulary, those logits and their cross-entropy path can be a meaningful part of training cost.

The MTP gradient reaches:

MTP-local parameters:
    enorm, hnorm, eh_proj, Transformer block

shared parameters:
    token embedding
    output head

main-model parameters:
    through h_i^(0), unless intentionally detached

For a general $D$-depth chain, deeper loss gradients pass through preceding MTP modules:

$$\frac{\partial \mathcal{L}_{MTP}^{(D)}} {\partial h^{(0)}} = \frac{\partial \mathcal{L}_{MTP}^{(D)}} {\partial h^{(D)}} \prod_{k=1}^{D} \frac{\partial h^{(k)}} {\partial h^{(k-1)}}$$

This is different from independent parallel heads, whose future losses branch directly from the same trunk state.

0x05B Shared Embedding And Output-Head Gradients

Sharing saves parameters but combines gradient sources:

$$\nabla_{W_U}\mathcal{L} = \nabla_{W_U}\mathcal{L}_{main} + \frac{\lambda}{D} \sum_k \nabla_{W_U}\mathcal{L}_{MTP}^{(k)}$$

The same applies to the shared embedding. A pipeline implementation must ensure that shared gradients are accumulated exactly once and synchronized across the physical stages that use them.

The DeepSeek-V3 report describes placing the shallowest and deepest layers on the same pipeline-parallel rank so the embedding and output head can be physically shared with the MTP path.

The complete gradient ledger separates local, shared, and upstream ownership:

$$\nabla_{\theta_{main}}\mathcal{L} = \nabla_{\theta_{main}}\mathcal{L}_{main} + \sum_{k=1}^{D} \frac{\lambda}{D} \frac{\partial \mathcal{L}_{MTP}^{(k)}}{\partial h^{(0)}} \frac{\partial h^{(0)}}{\partial \theta_{main}}$$ $$\nabla_{\theta_j}\mathcal{L} = \sum_{k=j}^{D} \frac{\lambda}{D} \frac{\partial \mathcal{L}_{MTP}^{(k)}}{\partial \theta_j}$$

The second sum starts at $k=j$: a depth-$j$ module receives its own loss and all deeper losses that traverse it. A detached previous_hidden_states tensor would intentionally remove the MTP-to-main term and define a different training program.

0x05C Physical Sharing Under Pipeline Parallelism

The report’s memory optimization is stronger than checkpoint name reuse. By placing the shallowest and deepest layers on the same pipeline rank, the embedding and output head can physically share parameters and gradients between main and MTP paths. Three implementations that look equivalent in a state dictionary are not operationally identical:

At training time, the alias group must be established before optimizer construction. Restoring aliases after optimizer state is created can leave duplicate momentum or variance tensors even when the model parameters later share storage.

0x06 Teacher Forcing And The Train/Serve Gap

During training, module $k$ receives ground-truth token $x_{i+k}$:

$$e_{i+k} = E(x_{i+k}^{gold})$$

During speculative inference, that token is produced by the main model or an earlier MTP depth:

$$\hat x_{i+k} \sim p^{(k-1)}$$

The module then consumes:

$$E(\hat x_{i+k})$$

If $\hat x_{i+k} \ne x_{i+k}^{gold}$, the inference path enters a context not used by ordinary teacher-forced training.

This tradeoff is deliberate:

teacher forcing:
    stable parallel training
    exact shifted inputs
    no autoregressive sampling in the training loop

self-fed training:
    closer to speculative runtime
    serial or specially packed rollout
    accumulated errors and harder optimization

The target verifier protects output correctness, but low-quality self-fed paths reduce acceptance and speedup.

The mismatch suggests alternatives, each with a different cost:

DeepSeek-V3 documents teacher-forced MTP; the alternatives are design choices, not claims about its training recipe.

0x06A Training Loop

# Single packed stream: main_hidden [N,d]; token metadata [N].
# main_loss is supplied by the separately masked main next-token objective.
# T > 0 is the declared fixed denominator; D is the configured module count.
state = main_hidden
state_bases = list(range(len(tokens)))
mtp_losses = []
assert len(mtp_modules) > 0

for k, module in enumerate(mtp_modules, start=1):
    assert state.shape[0] == len(state_bases)
    rows = compile_mtp_rows(
        tokens, segment_ids, valid, depth=k, previous_bases=state_bases,
    )
    if not rows:
        # Local teaching policy: keep this depth's zero term in the D-way mean.
        mtp_losses.append(main_hidden.sum() * 0.0)
        state, state_bases = state[:0], []
        continue

    previous = gather_rows(state, rows, "previous_depth_row")
    shifted = gather_tokens(tokens, rows, "shifted_token_index")
    current, logits, _ = module(
        previous,
        shifted,
        positions=gather_positions(positions, rows),
        mask=segment_causal_mask(rows),
    )
    # The module preserves input row order; masks use semantic segment/position.
    assert current.shape[0] == logits.shape[0] == len(rows)
    state = current
    state_bases = [row["base"] for row in rows]

    loss_rows = [j for j, row in enumerate(rows) if row["loss_valid"]]
    if loss_rows:
        labels = gather_tokens(
            tokens, [rows[j] for j in loss_rows], "label_index",
        )
        mtp_losses.append(token_loss_sum(logits[loss_rows], labels) / T)
    else:
        mtp_losses.append(logits.sum() * 0.0)

loss = main_loss + mtp_weight * mean(mtp_losses)
loss.backward()

The sketch above uses one compact packed row axis, unlike the dense [B,L,d] interface in 0x04B; a backend adapter must preserve the row identities, semantic positions and segment-causal visibility. It must not interpret a compacted row number as a token position. The main-model forward and its masked next-token objective are inputs to this sketch, not redefined here.

State-only rows still pass through the module and may supply causal context; only loss_rows select logits for the objective. An empty depth contributes zero without being removed from the configured depth mean. Skipping a module is a local teaching convention, not a distributed implementation: synchronized collectives and unused-parameter/gradient participation need their own explicit policy. This example does not demonstrate production DeepSeek training.

In distributed training, reduce numerator sums and denominator counts under the same convention before dividing. Averaging already-normalized per-rank losses is wrong when ranks have different valid-row counts.

0x07 General Chain Versus DeepSeek-V3 Configuration

The architecture is naturally described with $D$ sequential MTP modules. DeepSeek-V3 itself uses:

multi-token prediction depth:
    D = 1

ordinary main prediction:
    x_(t+1)

one additional MTP prediction:
    x_(t+2)

The official open-weight documentation also states:

num_nextn_predict_layers:
    1

MTP unique parameters:
    11.5B, excluding shared embedding and output head

MTP activated parameters:
    2.4B, including shared 0.9B embedding and 0.9B output head

main model:
    671B total parameters
    36.7B activated including embedding and output head

The repository overview rounds the downloadable package to 685B total, consisting of the 671B main model plus approximately 14B of MTP-associated weights.

These numbers are source-reported properties of the official release. They should not be recomputed as though every MTP parameter were active in its MoE block.

0x07A Released Weight Structure

The official weight documentation exposes:

With 61 main hidden layers, the MTP layer is packaged after them as layer 61 in the documented layout.

0x07B Load-Time Alias Contract

This loader takes already decoded, dense state dictionaries: quantization scales, sharding, and packed-weight conversion belong to the checkpoint adapter before this step. Its explicit checkpoint_order records the fusion columns. The equations and MTPModule use [hidden, embedding] ("he"); the pinned vLLM path uses [embedding, hidden] ("eh"). Conversion swaps the two column blocks, not input meanings or output rows. Unknown order is an error; shape equality alone cannot identify it.

def projection_for_he(weight, d, checkpoint_order):
    if weight.shape != (d, 2 * d):
        raise ValueError("fusion weight must have shape [d, 2d]")
    if checkpoint_order == "he":
        return weight
    if checkpoint_order == "eh":
        return concat([weight[:, d:], weight[:, :d]], dim=-1)
    raise ValueError("declare checkpoint fusion order: he or eh")


def load_deepseek_mtp(main, mtp_checkpoint, *, checkpoint_order):
    mtp = MTPModule(
        model_block=load_extra_layer(mtp_checkpoint),
        shared_embedding=main.embed_tokens,
        shared_head=MTPOutputHead(main.norm, main.lm_head),
        d=main.hidden_size,
    )
    load_local(mtp.enorm, mtp_checkpoint["enorm"])
    load_local(mtp.hnorm, mtp_checkpoint["hnorm"])
    weight = projection_for_he(
        mtp_checkpoint["eh_proj"]["weight"],
        main.hidden_size,
        checkpoint_order,
    )
    load_local(mtp.eh_proj, {"weight": weight})
    return mtp

For the pinned runtime-order adapter, call this with checkpoint_order="eh". Do not swap raw quantized storage without also transforming scale/layout metadata. The wrapper aliases both main.norm and main.lm_head; it neither copies parameters nor adds final normalization inside the decoder block.

Duplicating shared weights instead of aliasing them wastes memory and risks divergent training updates. Check parity with unequal hidden/embedding paths, asymmetric projection columns, and nontrivial output normalization. Identity weights or shape checks can conceal either a column swap or missing norm. Compare pre-final-norm states and logits separately.

0x07C Pinned Official And Runtime Source Trace

The source contracts were checked against:

DeepSeek-V3 repository:
    commit 9b4e9788e4a3a731f7567338ed15d3ec549ce03b

DeepSeek-V3 report:
    arXiv:2412.19437v2

vLLM runtime trace:
    commit 755513d9a24fb639a8985172e645c7fe5a263154
    file vllm/model_executor/models/deepseek_mtp.py

At that vLLM revision, the runtime path provides the following concrete map:

Source-local annotated visual reused from A33. The numbered path connects the paper equation to normalization, concatenation, projection, Transformer block, and output head; implementation details are pinned separately below.

The forward path also zeros token embeddings at position zero, applies exactly one final norm for logits, and may gather sequence-parallel MoE rows before returning them. Those are implementation details of the pinned revision, not parts of Equations 21-25 in the paper.

def pinned_vllm_mtp_step(inputs_embeds, previous_hidden, positions, layer):
    inputs_embeds = where(positions[:, None] == 0, 0, inputs_embeds)
    token_path = layer.enorm(inputs_embeds)
    hidden_path = layer.hnorm(previous_hidden)
    fused = layer.eh_proj(concat([token_path, hidden_path], dim=-1))
    block_out, residual = layer.mtp_block(
        positions=positions, hidden_states=fused, residual=None
    )
    pre_final_norm = residual + block_out
    recycled = layer.shared_head(pre_final_norm)
    return pre_final_norm, recycled


def pinned_vllm_mtp_logits(pre_final_norm, layer, logits_processor):
    return logits_processor(
        layer.shared_head.head,
        layer.shared_head(pre_final_norm),
    )

The pinned SharedHead call performs only final normalization; the logits processor applies the vocabulary head. The recycled state is post-final-norm, a separate output for that runtime’s recurrent rollout. Passing it into the logits path would normalize twice. The paper-style MTPModule instead returns its pre-final-norm depth state and already projected logits. These interfaces are not interchangeable just because hidden shapes agree. See the pinned implementation.

The official DeepSeek demo at the pinned repository revision does not expose this path; it builds only the target Transformer. That absence is an implementation boundary, not a contradiction with the weight package.

0x07D Loader And Release Compatibility Checks

def validate_loaded_mtp(manifest, loaded_names, main, mtp):
    expected_layers = set(manifest.mtp_layer_ids)
    observed_layers = spec_layers_from_names(loaded_names)
    assert expected_layers <= observed_layers
    assert mtp.eh_proj.weight.shape == (main.hidden_size, 2 * main.hidden_size)
    assert mtp.enorm.weight.shape == mtp.hnorm.weight.shape == (main.hidden_size,)
    assert mtp.vocab_size == main.vocab_size
    assert mtp.rope_contract == main.rope_contract
    assert mtp.expert_layout == manifest.expert_layout
    assert mtp.quantization == manifest.quantization

Quantizing a checkpoint and dropping layer 61 can still produce a target model that loads and generates normally. It cannot produce a valid MTP proposer. A release artifact must report target-only completeness and MTP completeness as separate gates.

0x08 Inference Option 1: Discard MTP

The DeepSeek report is explicit: the MTP modules can be removed after training.

loaded weights:
    main model only

decode:
    ordinary next-token loop

possible benefit:
    representation improvements learned during MTP pretraining

direct speculative speedup:
    none

This makes MTP partly analogous to an auxiliary training head, despite its richer module architecture.

0x09 Inference Option 2: Self-Speculative Decoding

For one proposal from the released one-module configuration, choose a carried-frontier round. At entry, output already contains P + [f], but target K/V covers only P. The incoming token f was emitted once by bootstrap or the preceding round; it is not a new output of this round. The saved target hidden state at the end of P, together with E(f), seeds one MTP proposal y for the position after f.

Bootstrap is separate: prefill the prompt, sample its first target token, and emit it once as f. Do not enter another round after EOS or exhaustion of the output budget.

1. MTP proposes y from normalized distribution q, retaining the actual sampling distribution for verification.
2. One causal target pass processes [f, y] after cached prefix P. Its row for f predicts y; its row for y predicts a possible bonus.
3. Accept y with the exact target/draft ratio. On rejection, emit residual correction c; on acceptance, emit y and target-sampled bonus b.
4. Commit the target row for f in both branches and the row for y only on acceptance. Neither correction nor bonus has target K/V merely because it was sampled. Carry it as the next pending frontier.

This selector uses vocabulary-indexed distributions and an independent uniform draw u. The callable sample samples the supplied normalized distribution. remaining counts available new output slots, excluding incoming f.

def select_mtp_one(p, q, y, bonus_p, u, sample, *, remaining, eos=None):
    from math import isclose, isfinite

    if type(remaining) is not int or remaining < 1:
        raise ValueError("at least one new output slot is required")
    for distribution in (p, q, bonus_p):
        if (len(distribution) != len(p) or len(distribution) == 0
                or any(not isfinite(v) or v < 0 for v in distribution)
                or not isclose(sum(distribution), 1.0, rel_tol=0, abs_tol=1e-12)):
            raise ValueError("aligned normalized distributions required")
    if type(y) is not int or not 0 <= y < len(q) or q[y] <= 0:
        raise ValueError("candidate must be sampled from positive draft support")
    if not 0 <= u < 1:
        raise ValueError("uniform draw must be in [0, 1)")

    accepted = u < min(1.0, p[y] / q[y])
    if accepted:
        output = [y]
        if remaining > 1 and y != eos:
            output.append(sample(bonus_p))
        keep_rows = 2  # input rows for f and y
        pending = output[-1] if len(output) == 2 else None
    else:
        residual = [max(pi - qi, 0.0) for pi, qi in zip(p, q)]
        mass = sum(residual)
        if mass <= 0:
            raise ValueError("rejection must have positive residual mass")
        correction = sample([v / mass for v in residual])
        output = [correction]
        keep_rows = 1  # row for f only; never the rejected y row
        pending = correction

    return {
        "accepted": accepted,
        "new_tokens": output,
        "target_rows_to_keep": keep_rows,
        "pending_frontier": pending,
        "stopped": len(output) == remaining or output[-1] == eos,
    }

The serving integration below uses teaching interfaces, not a particular engine’s API. Its entry state must satisfy the carried-frontier invariant:

q, mtp_delta = propose_one_from_frontier(
    previous_target_hidden=state.parent_hidden,
    shifted_token=state.frontier,
    mtp_kv=state.mtp_kv,
)
y = sample(q)
verified = target_verify(
    input_tokens=[state.frontier, y],
    target_kv=state.target_kv,
)
decision = select_mtp_one(
    verified.next_probs[0], q, y, verified.next_probs[1],
    uniform(), sample, remaining=request.remaining, eos=request.eos,
)
keep = decision["target_rows_to_keep"]
publish_target_rows(verified.rows[:keep])
publish_output_tokens(decision["new_tokens"])  # never emit incoming f again
discard_target_rows(verified.rows[keep:])
state.parent_hidden = verified.pre_final_norm_hidden[keep - 1]
state.frontier = decision["pending_frontier"]
repair_or_reseed_mtp(state, mtp_delta, decision)
if decision["stopped"]:
    finish_request_without_materializing_unused_frontier(state)

For a nonterminal round with at least two available output slots:

The rejection distribution is normalized positive target-minus-draft mass, not a fresh sample from the unmodified target distribution. The ratio applies only to y, not to f, c, or b. This is the one-proposal specialization of M7.1 and exact speculative sampling. EOS and budget limits truncate output before publication: an accepted EOS or last-budget proposal has no bonus. Its target row may already be materialized, but no later speculative token is published.

If a different trace counts freshly emitted f in the same interval and resolves y without a bonus, it emits two tokens, accepted or corrected. That interval must not reuse the carried-frontier 1 + acceptance_rate ledger. Schedule and accounting boundary must change together.

0x09A Cache Ownership

If the MTP Transformer block is served autoregressively, it has its own K/V cache:

target KV:
    canonical committed model state

MTP KV:
    draft-module state

verification branch KV:
    temporary target rows for proposed tokens

For a general $D$-module chain, distinct modules have distinct weights and therefore distinct K/V projections. Their caches are not interchangeable:

$$K^{(k)} = X W_K^{(k)}, \qquad V^{(k)} = X W_V^{(k)}$$

Even equal token prefixes do not make $K^{(k)}$ valid for module $j\ne k$.

0x09B Prefill Serialization

For (D>1):

main model prefill
    -> produces h^(0)

MTP module 1 prefill
    -> produces h^(1)

MTP module 2 prefill
    -> produces h^(2)

...

The dependency chain serializes module prefills. More prediction depth can improve proposal reach but adds module compute, cache, and scheduling complexity. DeepSeek-V3’s depth-1 choice avoids a multi-module chain in the released configuration.

0x09C Correction-Token State

The M7.1 correction rule still applies. If a candidate is rejected and a different correction token is emitted:

do not:
    commit target branch KV belonging to the rejected candidate
    treat MTP KV for the rejected path as canonical

do:
    commit only valid target-verified rows
    compute correction-token target state through a valid target path
    repair or rebuild MTP draft state from the committed prefix

MTP changes candidate generation, not the correctness rules for target state.

In the carried-frontier schedule, the correction is the first input of the next verification pass. There is no separate append before proposing from its saved parent hidden state. An eager-append runtime is also possible, but its extra target work must be charged and its seed convention declared. Pending output is not an already materialized target row.

0x09D One Checkpoint Depth Can Drive Several Draft Steps

D_ckpt = 1 and num_speculative_tokens > 1 are not contradictory. The pinned vLLM path selects:

current module index = spec_step_idx mod num_mtp_layers

With one checkpoint module, every draft step reuses that module’s weights while feeding back the previous draft hidden state and newly proposed token. This is a recurrent rollout over one learned transition:

$$\hat h_{t}^{(s+1)} = \operatorname{MTP}_{1} \left(\hat h_t^{(s)},E(\hat x_{t+s+1})\right)$$

It differs from the paper’s general depth-specific chain $\operatorname{MTP}_1,\ldots,\operatorname{MTP}_D$. The first rollout step is closest to the depth-one teacher-forced training coordinate. Later recurrent steps move farther from that coordinate and must be evaluated empirically.

0x09E Versioned Cache Transaction

Treat a speculative round as a transaction over four state families:

round = {
    "prefix_version": v,
    "frontier": target_frontier,
    "target_kv": canonical_target_kv.at(v),
    "mtp_kv": draft_kv.at(v),
    "target_branch_kv": empty_provisional_buffer(v),
    "proposal_tokens": [],
    "proposal_probabilities": [],
}

for step in range(requested_steps):
    token, q, hidden, mtp_delta = mtp_propose(round, step)
    round["proposal_tokens"].append(token)
    round["proposal_probabilities"].append(q)
    stage(round["mtp_kv"], mtp_delta)

verification = target_verify(round)
decision = exact_select(verification, round["proposal_probabilities"])

publish_target_rows(verification, decision.accepted_target_rows)
publish_output_tokens(decision.output_tokens)
discard_unpublished_target_rows(verification)

if decision.frontier_token_requires_append:
    carry_pending_frontier_with_parent_state(decision.frontier_token)
    require_next_target_pass_to_consume_frontier_first()

repair_or_reseed_mtp_from_committed_version(v + 1)

The target K/V generation is canonical. MTP K/V can be retained only when its ancestry exactly matches the committed output; otherwise it is discarded or recomputed. A draft cache that has consumed a rejected token cannot be repaired by changing only the token id metadata.

The figure’s frontier append is a dependency: in the carried schedule it is performed by the next verification pass, not an uncharged separate pass. Repair may retain verified-prefix draft rows and reseed from target features, but may not reuse a row that consumed the rejected candidate.

0x09F MoE And MLA State Are Module-Local

The extra MTP decoder layer follows DeepSeek’s hidden-layer structure, but its state remains owned by the MTP layer:

MLA cache:
    projected by the MTP layer's attention weights
    not interchangeable with any target layer cache

MoE route:
    selected from the MTP layer's hidden state
    can touch experts different from the target verification pass

expert communication:
    belongs in C_mtp and in per-rank traffic telemetry

target verification:
    produces canonical target rows under target layers only

This is why a “one extra token” acceptance rate cannot be converted directly into speedup without placement, communication, and memory measurements.

0x09G Integrated Depth-Two Oracle

A small oracle catches the contracts that shape tests miss. For tokens [A,B,C,D,E,F], two general MTP depths, and no packed boundary:

depth 1 rows:
    (base A, teacher B, label C)
    (base B, teacher C, label D)
    (base C, teacher D, label E)
    (base D, teacher E, label F)

depth 2 rows:
    (base A, teacher C, label D)
    (base B, teacher D, label E)
    (base C, teacher E, label F)

The executable evidence fixture then proves:

1. every row stays inside one segment;
2. module fusion has $d+d\rightarrow2d\rightarrow d$ shapes;
3. paper-fixed and valid-row mean losses differ at depth two;
4. the shared head receives main and all MTP loss contributions;
5. layer IDs 61..61+D_ckpt-1 reconcile with the checkpoint manifest;
6. target, MTP, and branch cache generations cannot alias;
7. rejection never publishes a draft-owned or sibling target row.

0x0A Runtime Cost Ledger

Let:

C_main:      ordinary target cost for one output-token step
C_round:     extra target work outside verification (zero for this steady round)
C_mtp:       MTP proposal plus required draft-state repair/reseed cost
C_verify:    one target pass over incoming frontier f and proposed token y
R:           newly emitted output tokens; excludes the already emitted f
M_mtp_kv:    MTP cache bytes
M_branch_kv: temporary target branch bytes

A dimensionally explicit speedup estimate is:

$$S = \frac{\mathbb{E}[R]C_{main}} {C_{round}+C_{mtp}+C_{verify}+C_{pack}+C_{sync}}$$

For the carried-frontier, one-proposal schedule in 0x09, away from EOS and output-budget limits, rejection emits one correction and acceptance emits the proposal plus one bonus. Let $a$ be proposal acceptance probability conditioned on the incoming state. Then:

$$\mathbb{E}[R]=1+a$$

This is $(1-a)\cdot 1+a\cdot 2$, not “one fresh target token plus an optionally emitted proposal.” Both branches emit something at the rejected/accepted position. One bootstrap output is counted once per request, outside these rounds; stopping can reduce progress below this expression.

The costs above are disjoint per-round means when branch-dependent. C_verify includes both target input rows, output distributions, and their provisional K/V even when y is rejected. The incoming frontier is already included in that pass: do not charge it again through C_round. Prefill, bootstrap, termination, and any alternative eager append belong in the request-level ledger or a separately declared extra-target-work term.

Thus $1+a$ is expected output progress under this schedule, not guaranteed wall-clock speedup. Taking the report’s source-measured $a\in[0.85,0.90]$ as the mean acceptance rate of this schedule gives a progress term of $1.85$-$1.90$ tokens per nonterminal round; that value is conditional on the reported rate, not a measurement of this schedule. The 1.8x TPS (tokens per second) is likewise a source result from DeepSeek’s deployment, not an algebraic consequence of the progress term, and neither number proves that this teaching schedule or another deployment achieves those timings.

Renderer-safe ledger:

nonterminal carried-frontier round:
    reject: emit correction                         R = 1
    accept: emit proposal + bonus                   R = 2
    incoming frontier was already emitted           contributes 0 to R
    E[R] = 1 + acceptance_rate
    C_round = 0  # no target work outside the two-input verification pass

speedup:
    S = E[R] * C_main
        / (C_round + C_mtp + C_verify + C_pack + C_sync)

win condition:
    E[R] * C_main > total speculative-round cost

request output accounting:
    bootstrap outputs + sum(actual new outputs per round)
    stop/EOS truncation uses actual outputs, not the nonterminal expectation

Memory is:

$$M_{served} = M_{main\ weights} + M_{MTP\ unique\ weights} + M_{target\ KV} + M_{MTP\ KV} + M_{branch\ KV} + M_{workspaces}$$

DeepSeek-V3 reports an 85%-90% acceptance rate for its additional predicted token across evaluated generation topics and reports 1.8x TPS. Those are source-reported measurements, not values to insert into an unrelated deployment model.

0x0A1 MoE Cost Interaction

The MTP block in DeepSeek-V3 follows the model’s hidden-layer architecture, including sparse expert behavior. Serving it is not equivalent to adding one small dense MLP head. Runtime cost can include:

router computation
routed expert weights
expert-parallel communication
shared expert compute
MTP attention and KV
shared output-head projection

Verification of two candidate positions can also widen the union of experts touched by the main model. M6.2’s routing and logical-traffic ledger still applies; M6.6 owns how physical placement, collectives, kernels, and topology turn that traffic into measured runtime.

0x0B Comparison With Related Designs

The closest conceptual relation is EAGLE’s complete causal draft chain, which the DeepSeek report acknowledges. The ownership and training regime remain different.

0x0B1 Paper-By-Contract Comparison

This table prevents an implementation from borrowing the loss of one paper, the recurrence of another, and the checkpoint names of a third while calling the result “DeepSeek MTP.”

0x0C Training And Runtime Integration Checklist

These are deployment requirements, not unfinished article tasks. The mechanism and the reason for each check are covered below; a training or serving team must still execute the checks against its exact checkpoint, parallel layout, engine, and workload before release.

Training integration contract:

Runtime integration contract:

0x0D Failure Lab

0x0E Observability

Training:

main loss
MTP loss by depth
MTP accuracy by depth
gradient norm from MTP into main trunk
shared embedding/head gradient contribution
MTP block expert load and communication
loss-weight schedule

Runtime:

MTP candidate probability
additional-token acceptance rate
MTP module time
target verification time
MTP KV bytes
temporary target branch-KV bytes
experts touched by MTP and verification
fallback rate to ordinary decode
end-to-end tokens/s and request latency

0x0F What The Mechanism Articles Hand Forward

M7’s first six articles establish six distinct acceleration contracts:

1. exact draft/verify sampling;
2. Medusa heads and tree attention;
3. Lookahead Jacobi and n-gram reuse;
4. MTP objectives and shared-trunk gradients;
5. EAGLE-family feature/token drafting and dynamic trees;
6. DeepSeek’s integrated sequential MTP module.

M7.7 adds proposal-source selection, dependent-head comparisons, exact multi-candidate selection, and hardware-aware topology. M7.8 then closes the module with continuous-batch control, K/V frontier state, evaluation, release, and rollback.

M8 can compare these with quantization, batching, and deployment placement, but should not erase their state differences into a single “speculative decoding” toggle.

0x10 Publish Gate

0xFF References

• Transformer Series A33 - DeepSeek MTP
• M7.1 - Speculative Decoding
• M7.4 - Multi-Token Prediction Foundations
• M7.5 - EAGLE Feature Drafting
• M7.7 - Draft Sources and Hardware-Aware Trees
• M7.8 - Speculative Serving
• DeepSeek-V3 Technical Report, arXiv v2
• DeepSeek-V3 official repository, audited commit
• DeepSeek-V3 official weight documentation, audited commit
• DeepSeek-V3 target-only inference demo, audited commit
• vLLM DeepSeek MTP runtime, audited commit
• Better & Faster Large Language Models via Multi-token Prediction
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty