M6.10 - Teacher-Student Distillation and Model Transfer

0x00 Why One Trained Model Is Not One Deployment Shape

A large model has been trained, evaluated, and found useful. It is also too expensive for the intended deployment. Perhaps it misses a latency target, cannot fit on the available device, or consumes too much memory when many replicas are required. A natural question follows:

Can the behavior of the large, well-trained model be transferred into a smaller model that is cheaper to run?

That question sounds similar to quantization, pruning, or low-rank adaptation, but the operation is different. Quantization changes how an existing model is represented. Pruning changes which parameters or structures remain. LoRA changes which small update state is trained. Distillation starts another training problem: one model becomes a source of supervision for another.

The established names are teacher and student. The teacher is normally the stronger model. The student is the model being optimized. In a compression setting the student is deliberately shallower, narrower, or otherwise cheaper, but the definition does not require that. A same-size student can be distilled, and a model can even be trained from another instance of the same architecture. Knowledge distillation is therefore a transfer procedure, not a file format and not a guarantee of compression.

The deployment chain is:

choose a student architecture
        |
        v
construct teacher-derived supervision
        |
        v
optimize the student under a distillation objective
        |
        v
evaluate and export the student by itself
        |
        v
measure the student-only deployment

The first step creates the possible runtime saving. The middle steps try to recover behavior within that smaller hypothesis class. The last step proves whether the proposed saving exists in the actual runtime.

0x00A The Object Being Transferred Kept Changing

Distillation did not begin as one canonical Transformer loss. It accumulated several different transfer contracts as researchers changed what the student could observe and what kind of model they wanted to produce.

The pre-Transformer lineage already separated two ideas that are still easy to conflate. Ba and Caruana showed that a shallow model could learn the class-probability function of a deeper model more successfully than the same shallow architecture learned from hard labels alone. FitNets then treated an intermediate teacher representation as a hint, adding a learned regressor when teacher and student hidden widths differed. The student architecture, the output target, and the intermediate alignment object were already separate choices.

Hinton, Vinyals, and Dean made softened teacher distributions and temperature scaling the standard vocabulary. That work also inherited an earlier ensemble-compression motivation: the deployed model should reproduce useful behavior without carrying every model that created the target distribution. Sequence-Level Knowledge Distillation changed the artifact again. Instead of retaining a full teacher distribution at every reference-prefix row, it decoded teacher sequences and trained the student on those discrete outputs. Attention Transfer supplied another representation-level branch: transfer can constrain relations or activation-derived maps even when it does not match raw hidden vectors.

Source: Kim and Rush, Sequence-Level Knowledge Distillation, arXiv:1606.07947v4, Figure 1, cropped from the revision-pinned PDF. The figure distinguishes row-wise teacher distributions, one decoded teacher sequence, and sequence interpolation. It does not include later student-generated on-policy prefixes and does not prove that these objectives are equivalent.

Transformer distillation widened the target set further. DistilBERT combined a shallower initialized student with output-distribution, masked-language-model, and cosine objectives. PKD transferred selected teacher layers. TinyBERT used general and task-specific stages with embedding, hidden, attention, and prediction targets. MiniLM moved from raw states toward attention and value-relation matrices. These papers did not merely improve one scalar KD formula; they changed architecture, layer mapping, target tensors, training stages, and state lifetime.

Autoregressive LLM work exposed a different mismatch. A student trained only on reference or teacher prefixes is not directly supervised on the prefixes created by its own mistakes. MiniLLM and GKD therefore made prefix occupancy, divergence direction, sampling, and rescoring part of the method contract. Recent pruning-plus-distillation and sparse-logit work adds two systems questions: whether distillation is recovery for an already changed student graph, and whether a compressed teacher-target artifact still defines the same gradient estimator.

The resulting lineage is not one sequence of replacements:

output function      -> softened token distributions
raw labels           -> teacher-decoded sequence artifacts
final output         -> hidden, attention, and relation targets
fixed prefixes       -> student-generated on-policy prefixes
online teacher       -> versioned full or sparse cached targets
unchanged student    -> deliberately smaller or structurally transformed graph

Each arrow changes a different contract. Later sections derive those contracts individually instead of treating every paper as a synonym for “KD.”

0x00B The Teacher Provides More Than One Correct Token

Ordinary supervised training often presents one observed target. At a next-token row, the loss may say only that token Paris should receive more probability than every alternative. A trained teacher can expose a richer local structure:

Paris          high probability
Lyon           lower, but still a plausible city token
France         grammatically possible in another continuation
London         semantically related but wrong for this context
punctuation    depends on whether the answer is already complete

The teacher distribution does not make these alternatives correct. It reveals how the teacher orders them. Hinton, Vinyals, and Dean formalized the common teacher-student recipe using softened output distributions: temperature exposes probability relationships that are almost invisible behind a confident argmax, while the student can still retain an ordinary hard-label objective.

This output-distribution view is only the beginning. Later work transfers different objects:

output logits and probabilities
selected hidden states
attention distributions
relations between token representations
teacher-generated output sequences
teacher feedback on prefixes generated by the student

These targets are not interchangeable. Logit matching requires compatible output support. Hidden-state matching may require a projection from student width to teacher width. Attention matching requires a policy for layers and heads. A generated sequence can be obtained through a black-box API, but it retains much less information than the full teacher distribution. On-policy distillation adds another cost because the student must generate the contexts on which the teacher supervises it.

Calling all of these methods “teacher-student training” is correct but not sufficient. The target tensor, the contexts on which it is evaluated, the available teacher interface, and the execution plan all change.

0x00C The Student Architecture Creates The Deployment Saving

Suppose a conventional decoder block has hidden width $d$, FFN width $d_{ff}$, and equal-width Q, K, V, and output projections. A rough projection-parameter ledger is:

$$P_{attention}\approx 4d^2$$

For a two-matrix FFN:

$$P_{FFN}\approx 2dd_{ff}$$

and for a three-projection gated FFN such as SwiGLU:

$$P_{gated\ FFN}\approx 3dd_{ff}$$

These are intentionally local ledgers. Embeddings, normalization parameters, biases, tied output heads, grouped-query attention, and runtime effects still need their own accounting. They are enough to expose the ownership boundary:

reduce layers
  -> fewer blocks execute

reduce d
  -> smaller attention and residual projections

reduce d_ff
  -> smaller FFN projections

change head layout
  -> different attention and KV state

reduce vocabulary
  -> smaller embedding/output interface

change only the loss
  -> no structural runtime saving is guaranteed

The toy ledger in the figure uses a 32-layer teacher with $(d,d_{ff})=(4096,11008)$ and a 16-layer student with $(d,d_{ff})=(3072,8192)$. Under the simplified $4d^2+3dd_{ff}$ gated-block count, block projection parameters fall from about $6.48$ billion to $1.81$ billion, or roughly $28\%$ of the teacher’s block projection count. That is not a latency prediction and it excludes embeddings and the output head. It proves a narrower point: the architecture decision creates the smaller executable graph. Distillation is one way to train that graph.

This distinction also explains why the same word appears in different parts of Transformer System. M8.7 uses teacher supervision while a student experiences fake quantization. M8.8 discusses method-specific transfer into native binary or ternary models. The live M6.11 pruning article uses a teacher as one optional recovery path after physical pruning. Those are applications of distillation inside other structural or representation changes; they do not redefine its general training contract.

0x00D The Teacher Is Training State, Not A Deployment Dependency

During online distillation, a training step may hold:

teacher weights and forward buffers
student weights, gradients, and optimizer state
teacher and student logits
optional teacher/student hidden or attention tensors
alignment projections used only by auxiliary losses
valid-token masks and loss-reduction state

The teacher is normally frozen, so it does not need gradient or optimizer state. That does not make it free. Its weights must be resident, loaded, offloaded, or served somewhere; its forward pass consumes compute; and intermediate-target methods can keep large activation tensors alive until the student loss is formed.

Offline distillation changes the state rather than removing it. Full logits, sparse logits, hidden states, or teacher-generated sequences become versioned artifacts. The teacher revision, tokenizer, temperature, generation policy, normalization contract, and sample identity now belong to the data lineage. Caching the output of an expensive teacher can save repeated forward passes, but a cache without this provenance is not a reproducible target.

The exported deployment should be loadable without the teacher, the distillation wrapper, or training-only alignment projections. If the claimed student cannot run and pass evaluation alone, the transfer pipeline has not produced a deployment artifact yet.

0x00E What This Article Owns

This article owns:

temperature-scaled logit distillation and its gradient scale
hard-label and teacher-distribution objective composition
causal token alignment, masks, and vocabulary support
hidden-state, attention, and relation transfer
student depth/width/head alignment and initialization
sequence-level and black-box distillation
fixed-prefix versus student-generated on-policy supervision
online, cached, hybrid, and distributed training execution
teacher/student artifacts, provenance, evaluation, and export boundaries
comparisons among major paper variants

It reuses M3.9’s training execution and M3.8’s output-head interface. It does not re-derive teacher forcing or the base vocabulary projection. It consumes the accounting and execution vocabulary established by M4.1 and M4.2, without claiming that a logical distillation ledger is a measured kernel result. It hands pruning to M6.11, and hands distilled draft-model use to M7.1, QAT-specific transfer to M8.7, native low-bit transfer to M8.8. When the student is composed with low-bit W/A/KV or a native low-bit representation, its existing local artifact/evaluation evidence becomes input to M8.9/M8.10. This does not assign every nonquantized student’s production promotion to M8.

The owner question is:

How do teacher logits, temperature, hidden or relational targets, generated sequences, student architecture, and cached or online supervision determine what behavior is transferred and what training/runtime state is required?

0x00F Immutable Identity, Evidence, And Loop Contract

“Teacher” and “student” are roles, not sufficient artifact identities. A replayable transfer candidate needs at least the following state before a loss is evaluated:

teacher:
    model artifact + exact revision + architecture + tokenizer
student:
    initialization artifact + exact revision + architecture + tokenizer
supervision:
    matched object + temperature/divergence + layer/head/width map
prefix/data:
    dataset revision + sample identity + generation/filtering policy
execution:
    online/offline/hybrid/on-policy plan + dtype + parallel ownership
release:
    student-only artifact identity + forbidden teacher/training-only state

Two runs that share the words “teacher,” “student,” and “KL” but differ in one of those fields are not the same experiment. A generated sequence belongs to a decoding configuration; a cached logit row belongs to a teacher, tokenizer, temperature, vocabulary, and sample position; an intermediate-state target belongs to a layer map and an alignment transform. The manifest is therefore part of the mathematical and runtime contract, not clerical metadata.

The evidence labels used throughout this article are deliberately separate:

Label	What it establishes	What it does not establish
Primary-paper claim	a revision-pinned paper states or derives the method claim	that the paper run was reproduced locally
Official-code contract	a pinned repository or versioned official interface implements a particular state path	that every framework or revision behaves identically
Source-local explanation	the earlier book/design material motivates an explanatory sequence	independent technical verification
Local derivation	the displayed algebra, tensor trace, or deterministic fixture is internally replayable	model quality, GPU performance, or production behavior
Source-reported result	a named paper reports a measurement under its declared setup	a transferable benchmark or deployment guarantee
Uncertain or runtime-specific	the claim depends on missing disclosure, moving software, hardware, or workload	permission to generalize it

Loop 1 freezes lineage, identities, sources, figures, and owner boundaries. It does not claim to have trained a teacher or student, reproduced a paper metric, executed a framework model or GPU kernel, validated a production scheduler, checked every live external URL, or deployed the student. The later loops are intentionally non-interchangeable:

Loop 2   classical logits, temperature, and gradient scale
Loop 3   causal token rows, support, and distributed loss
Loop 4   hidden, attention, relation, and layer-map transfer
Loop 5   architecture, initialization, capacity gap, and state cost
Loop 6   sequence and black-box artifacts
Loop 7   on-policy prefixes, divergences, MiniLLM, and GKD
Loop 8   supervision artifact schema, storage, filtering, and revisions
Loop 9   online/offline/hybrid distributed runtime
Loop 10  paper-by-contract and equal-budget comparison
Loop 11  evaluation, failures, release, and teacher independence
Loop 12  formula, source, rendering, accessibility, and boundary closure

The article can be publication-closed only after all twelve evidence bundles agree on the same candidate. Loop 1 proves the starting identities and claim ceiling; it does not pre-approve the later mechanisms.

0x01 Four Contracts Define A Distillation Method

A distillation method cannot be identified by its loss name alone. The same KL formula can train on reference prefixes or student-generated prefixes. The same teacher can expose only text, full logits, or every hidden state. Two systems can match identical targets while one runs teacher and student together and the other reads a multi-terabyte cache.

A useful method signature is:

K = (
  matched object,
  prefix distribution,
  teacher access,
  execution plan
)

This four-part signature prevents several false equivalences. DistilBERT and TinyBERT both use Transformer teachers, but their training objectives and intermediate-state contracts differ. Sequence-level KD and on-policy GKD both involve generated text, but the former commonly trains on teacher-produced sequences while the latter deliberately asks the teacher to score contexts created by the student.

0x01A Contract One: What Teacher Object Is Matched?

Matched object	Typical tensor/artifact	What it preserves	Alignment requirement
hard token	token id per valid row	observed target	student vocabulary contains target
soft token distribution	[B,S,V] logits/probabilities	teacher ordering over alternatives	compatible output support and token rows
hidden state	[B,S,d_T]	selected representation coordinates	layer map and often width projection
attention distribution	[B,H,S,S]	token-to-token read pattern	layer, head, mask, and token alignment
relation matrix	often [B,H,S,S]	pairwise structure rather than raw width	token and relation-head policy
generated sequence	token/text artifact	one or more teacher trajectories	generation and tokenizer provenance

The rows expose an information-access tradeoff. Output text can be collected from a black-box teacher but discards most of the teacher distribution. Full logits preserve far more local information but create vocabulary-sized compute, communication, and storage. Intermediate tensors can provide dense guidance, yet they couple the student to teacher layer semantics and greatly increase activation lifetime.

Patient Knowledge Distillation transfers multiple teacher-layer representations rather than relying only on the final output. TinyBERT combines embedding, hidden, attention, and prediction targets across general and task-specific stages. MiniLM instead emphasizes self-attention and value-relation transfer, avoiding a requirement that raw teacher and student hidden widths be identical. It still needs compatible token and relation layouts; relation matching is not alignment-free.

0x01B Contract Two: Which Prefix Distribution Is Supervised?

An autoregressive teacher predicts a distribution conditioned on a prefix. The source of that prefix changes the learning problem.

reference prefix:
    both models see contexts from the labeled or fixed corpus

teacher-generated prefix:
    the teacher decodes a sequence; the student trains on that artifact

student-generated prefix:
    the student visits its own contexts; the teacher scores those contexts

mixed prefix policy:
    training combines fixed and generated contexts under an explicit ratio

Fixed-prefix token KD is efficient because every target row can be produced in parallel under teacher forcing. It also leaves an exposure gap: at inference, the student conditions on its own earlier outputs, including mistakes not present in the reference corpus.

Sequence-Level Knowledge Distillation addresses sequence generation by producing target sequences with the teacher and training the student on those sequences. The generated corpus is useful even when only black-box text is available, but beam width, sampling temperature, stopping policy, and filtering become part of the transferred behavior.

On-Policy Distillation of Language Models, which introduces GKD, moves supervision onto student-generated outputs. The teacher and student are evaluated along prefixes the student actually visits. This is not merely a different divergence. It adds student generation to the training runtime and changes the distribution of contexts on which teacher feedback is consumed.

0x01C Contract Three: What Teacher Access Exists?

Teacher interface	Can support	Cannot provide by itself	Main systems cost
generated text only	sequence-level targets, filtering	full token uncertainty or hidden state	generation latency and corpus storage
top-k logits	sparse local distribution target	exact omitted tail without extra contract	indices, values, tail/normalization metadata
full logits	token KL/cross-entropy over shared support	internal representation targets	[B,S,V] compute, traffic, or storage
selected hidden/attention state	representation and relation losses	arbitrary unexposed layers	teacher activation lifetime and layer coupling
full white-box model	all of the above	nothing architectural in principle	maximum integration and residency burden

Direct token-wise KL normally assumes teacher and student probabilities refer to compatible vocabulary events at aligned token positions. If the tokenizers differ, equal tensor rank does not establish semantic correspondence. A system must introduce an explicit cross-tokenizer projection/alignment contract or transfer at another level, such as generated text. Silent truncation to the smaller vocabulary is not a valid general solution.

Teacher access also determines which claims can be reproduced. A hosted model that returns only generated strings can serve as a sequence teacher, but it cannot retroactively support a TinyBERT hidden-state loss. Conversely, access to every teacher tensor does not mean every tensor is a useful target; each additional match constrains the student and extends the training graph.

0x01D Contract Four: How Is Supervision Executed?

Execution plan	Live state	Persistent artifact	Failure boundary
online logits	teacher and student forwards	checkpoints and logs	teacher throughput/residency
online intermediate transfer	both models plus selected activations	checkpoints, layer map, projections	activation memory and hook correctness
cached full/sparse logits	student training plus cache reader	versioned logit shards	storage, stale cache, omitted-tail bias
offline teacher sequences	ordinary student training	generated corpus and decode provenance	narrow search/data coverage
on-policy scoring	student generation, teacher score, student update	prompts, generation policy, logs	high compute and scheduler complexity
distributed KD	sharded teacher/student and reductions	topology-aware checkpoints	vocabulary normalization and partition mismatch

Online and offline are not quality labels. Online execution preserves the ability to change contexts or teacher settings during training, but repeatedly pays for teacher inference. Offline execution amortizes teacher work, but fixes the teacher revision, examples, temperature, and output schema into the cache. A hybrid system can cache common examples and query the teacher on new student-generated prefixes; it must then prove that the two supervision paths use compatible distributions and normalization.

The execution contract also controls recovery. A resumable training run needs more than student optimizer state when data were generated on policy. It may need generation RNG state, teacher and tokenizer revisions, cache-shard positions, the fixed/on-policy mixture schedule, and training-only alignment parameters. Losing that state can change the objective after a restart even if the student checkpoint loads successfully.

0x01E Paper Names Are Coordinates, Not Complete Specifications

Method	Matched object	Prefix source	Teacher access	Execution sketch
Hinton KD	softened output distribution plus hard labels	fixed examples	teacher probabilities/logits	online or cached token targets
DistilBERT	output distribution, MLM labels, cosine representation target	pretraining corpus	logits and hidden state	online pretraining transfer
TinyBERT	embedding, hidden, attention, prediction targets	general and task-specific corpora	broad white-box access	multi-stage online transfer
MiniLM	attention and value relations	fixed corpus	relation-producing internal tensors	online relation transfer
Sequence-Level KD	teacher-decoded sequence	teacher generation	text output is sufficient	offline generated corpus
GKD	selectable divergence on teacher/student token distributions	fixed and student-generated	teacher scores selected prefixes	online generation and scoring

The table is a preview, not a verdict. These choices can be staged and combined. A student may be initialized from selected teacher layers, trained with hidden and logit objectives on fixed data, then refined on student-generated prefixes. Each stage changes the matched state, required teacher interface, stored artifacts, and runtime ledger.

That is the starting discipline for the rest of M6.10: before comparing loss values or benchmark results, identify the four contracts and the student graph that will actually be deployed.

0x02 Classical Logit Distillation

The simplest white-box transfer object is the teacher’s output distribution at one supervised row. Let the teacher and student produce vocabulary logits $z_T,z_S\in\mathbb{R}^{V}$ for the same prefix $c$. Temperature $\tau>0$ defines two distributions over the same vocabulary events:

$$p_{T,i}^{\tau} = \frac{\exp(z_{T,i}/\tau)} {\sum_{j=1}^{V}\exp(z_{T,j}/\tau)}, \qquad p_{S,i}^{\tau} = \frac{\exp(z_{S,i}/\tau)} {\sum_{j=1}^{V}\exp(z_{S,j}/\tau)}.$$

This is an ordinary softmax with rescaled logits. At $\tau=1$ it is the model’s usual output distribution. Raising $\tau$ reduces every pairwise log-odds gap:

$$\log\frac{p_i^{\tau}}{p_j^{\tau}} = \frac{z_i-z_j}{\tau}.$$

The ranking does not change, but the distribution becomes less concentrated. The teacher and student must use the same temperature inside the soft-target term. After training, the student normally returns to $\tau=1$ for ordinary inference; temperature is not baked into its weights as a deployment setting.

This loop deliberately freezes the problem at one already aligned causal row. Its contract is:

Field	Loop 2 value	Why it cannot remain implicit
teacher row	frozen $z_T\in\mathbb{R}^{V}$	the target must not receive student gradients
student row	trainable $z_S\in\mathbb{R}^{V}$	every derivative below is with respect to this row
event support	identical ordered vocabulary events	equal vector length alone does not establish semantic alignment
soft temperature	one positive $\tau$ used for teacher and student	unequal temperatures define a different matching problem
hard temperature	$1$	the deployed next-token objective is not softened accidentally
soft direction	$\operatorname{KL}(p_T^\tau\Vert p_S^\tau)$	reversing the arguments changes the expectation and gradient
reduction	sum over vocabulary for one row	causal shifts, token masks, and vocabulary shards enter in Loop 3

The contract is narrow on purpose. It lets us prove what temperature and $\tau^2$ do without hiding row-alignment or distributed-reduction errors inside the same derivation.

0x02A Temperature Exposes Relative Structure

The formula should be evaluated as a stable softmax, not as literal unbounded exponentials. With

$$a_i=\frac{z_i}{\tau}, \qquad m=\max_j a_j,$$

the equivalent implementation is

$$p_i^\tau = \frac{\exp(a_i-m)}{\sum_j\exp(a_j-m)}.$$

Subtracting $m$ changes neither numerator-to-denominator ratios nor the final probabilities, while preventing the largest exponential from overflowing. More generally, softmax discards a common logit offset:

$$p^\tau(z+c\mathbf{1})=p^\tau(z).$$

Temperature therefore acts on differences, not absolute logit levels. For finite $\tau>0$, it preserves ranking and divides every pairwise log-odds gap by $\tau$. If the maximum is unique,

$$\lim_{\tau\to0^+}p^\tau = \text{one-hot at }\arg\max_i z_i, \qquad \lim_{\tau\to\infty}p_i^\tau = \frac{1}{V}.$$

Neither limit creates knowledge. A small temperature discards almost all non-argmax structure; an arbitrarily large temperature eventually erases the differences it was meant to expose.

Consider one teacher row with four logits:

vocabulary events: [A, B, C, D]
teacher logits:    [8, 4, 1, -2]

Event	Logit	Probability at $\tau=1$	Probability at $\tau=4$
A	8	0.9811	0.6159
B	4	0.0180	0.2266
C	1	0.000895	0.1070
D	-2	0.0000445	0.0506

At $\tau=1$, a hard target and the teacher distribution both look almost like “choose A.” At $\tau=4$, the student can see that B is a much stronger alternative than C, and C is stronger than D. This relative structure among non-target events is what the classical distillation literature calls dark knowledge. It is not a hidden tensor or a claim that the teacher knows ground truth about every alternative. It is the teacher’s learned ordering, including any teacher mistakes and biases.

Hinton, Vinyals, and Dean motivate high temperature for exactly this reason: ratios among very small probabilities can describe which mistakes the teacher considers similar. Their result does not say that the largest possible temperature is best. At very high temperature, weakly constrained, very negative teacher logits can receive more influence. The useful temperature depends on the teacher, student capacity, data, and loss weighting.

0x02B One Row Contains Two Different Targets

Classical KD commonly combines two objectives:

1. A hard-label objective asks the student to predict the observed token using its ordinary $\tau=1$ distribution.
2. A soft objective asks the student to match the teacher distribution formed with $\tau>1$.

There are several superficially similar ways to construct the second term, but they do not preserve the same invariances or scalar meaning:

Target/objective	Row loss	What it preserves	Important difference
hard label	$-\log p^1_{S,y}$	observed target event	says nothing about teacher ordering among alternatives
raw-logit regression	$\lVert z_S-z_T\rVert_2^2$	every absolute logit coordinate	changes under independent teacher/student offsets unless logits are centered
centered-logit regression	$\lVert\widetilde z_S-\widetilde z_T\rVert_2^2$	relative logit geometry	is the high-temperature limit, not the exact finite-$\tau$ objective
soft-target cross-entropy	$-\sum_i p_{T,i}^\tau\log p_{S,i}^\tau$	teacher-weighted event support	includes constant teacher entropy in the reported scalar
forward KL	$\sum_i p_{T,i}^\tau\log(p_{T,i}^\tau/p_{S,i}^\tau)$	same student gradient as soft cross-entropy	reports zero exactly when the two distributions match

Ba and Caruana used output-function matching before the temperature recipe became standard. Hinton, Vinyals, and Dean then showed why high-temperature distribution matching approaches centered logit matching. This is a derivation-level relationship, not permission to swap finite-temperature KL for uncentered MSE without changing the method.

For the teacher row above, suppose the student produces $z_S=[6,5,0,-1]$. At $\tau=4$:

Event	Teacher $p_T^4$	Student $p_S^4$	Transfer signal
A	0.6159	0.4596	student underweights teacher’s top event
B	0.2266	0.3580	student overweights the closest alternative
C	0.1070	0.1026	distributions nearly agree
D	0.0506	0.0799	student overweights the weakest event

For this row, $\operatorname{KL}(p_T^4\Vert p_S^4)\approx0.0580$. Multiplication by $\tau^2=16$ makes the scaled soft contribution about $0.9285$ before token averaging and the mixture weight are applied. If A is also the observed hard target, the student’s ordinary $\tau=1$ probability for A is about $0.7293$, giving hard-label NLL $-\log 0.7293\approx0.3157$.

These scalar values should not be compared as if they had the same natural scale. The $\tau^2$ factor and mixture coefficient deliberately set the gradient balance. The useful information is the direction of both signals: the hard target rewards A, while the soft target also says how probability should move among B, C, and D.

0x02C Masked Hard And Soft Objectives

For batch index $b$, causal prediction row $s$, valid-target mask $m_{b,s}$, and $N=\sum_{b,s}m_{b,s}$, define:

$$L_{hard} = -\frac{1}{N}\sum_{b,s}m_{b,s} \log p_S^{1}(x_{b,s+1}\mid x_{b,\le s}),$$ $$L_{soft} = \frac{\tau^2}{N}\sum_{b,s}m_{b,s} \operatorname{KL}\!\left( p_T^{\tau}(\cdot\mid x_{b,\le s}) \Vert p_S^{\tau}(\cdot\mid x_{b,\le s}) \right),$$

and

$$L_{total}=(1-\alpha)L_{hard}+\alpha L_{soft}, \qquad 0\le\alpha\le1.$$

This article uses the forward direction $\operatorname{KL}(p_T\Vert p_S)$. Reversing the arguments creates a different objective and gradient. Soft-target cross-entropy, $-\sum_i p_{T,i}\log p_{S,i}$, gives the same student gradient because

$$\operatorname{KL}(p_T\Vert p_S) = H(p_T,p_S)-H(p_T),$$

and teacher entropy $H(p_T)$ is constant with respect to student parameters. The logged scalar is not the same, however. An implementation and experiment record must say whether it reports KL or soft cross-entropy, which reduction it uses, and whether $\tau^2$ is applied before or after reduction.

For one row, write $r_i=p_{S,i}^{1}$ and let $y_i$ be the one-hot hard target. The exact student-logit gradients of the two terms are

$$\frac{\partial \ell_{hard}}{\partial z_{S,i}} =r_i-y_i, \qquad \frac{\partial \ell_{soft}}{\partial z_{S,i}} =\tau\left(p_{S,i}^{\tau}-p_{T,i}^{\tau}\right),$$

where $\ell_{soft}=\tau^2\operatorname{KL}(p_T^\tau\Vert p_S^\tau)$. Consequently,

$$\frac{\partial \ell_{total}}{\partial z_{S,i}} =(1-\alpha)(r_i-y_i) +\alpha\tau\left(p_{S,i}^{\tau}-p_{T,i}^{\tau}\right).$$

This equation is the coefficient contract. Some codebases instead use $\ell_{hard}+\lambda\ell_{soft}$, or define $\alpha$ as the hard-loss weight. Those parameter names are not interchangeable. A run manifest must record the expanded scalar equation, not only alpha=0.5.

0x02D Deriving The $\tau^2$ Gradient Scale

The multiplier should not be memorized as a convention. It follows from how temperature enters both the softmax derivative and the difference between the soft distributions.

Let $p_i=p_{T,i}^{\tau}$ be a fixed teacher target and $q_i=p_{S,i}^{\tau}$ be the student distribution. For one-row soft-target cross-entropy,

$$C=-\sum_j p_j\log q_j.$$

The derivative of temperature-scaled log-softmax is

$$\frac{\partial\log q_j}{\partial z_{S,i}} = \frac{1}{\tau}(\mathbf{1}[i=j]-q_i).$$

Therefore:

$$\begin{aligned} \frac{\partial C}{\partial z_{S,i}} &= -\sum_j p_j \frac{\partial\log q_j}{\partial z_{S,i}} \\ &= -\frac{1}{\tau}\sum_j p_j (\mathbf{1}[i=j]-q_i) \\ &= \frac{q_i-p_i}{\tau}. \end{aligned}$$

That is the first factor of $1/\tau$. A second factor appears because the two soft distributions themselves become closer as temperature rises. Softmax is invariant to adding one constant to every logit, so center each teacher and student vector separately:

$$\widetilde z_i=z_i-\frac{1}{V}\sum_j z_j.$$

When $\tau$ is large relative to centered logit magnitudes, $\exp(\widetilde z_i/\tau)\approx1+\widetilde z_i/\tau$. Since centered logits sum to zero:

$$p_i \approx \frac{1}{V}+\frac{\widetilde z_{T,i}}{V\tau}, \qquad q_i \approx \frac{1}{V}+\frac{\widetilde z_{S,i}}{V\tau}.$$

Subtracting them and substituting into the gradient gives:

$$q_i-p_i \approx \frac{\widetilde z_{S,i}-\widetilde z_{T,i}}{V\tau},$$ $$\frac{\partial C}{\partial z_{S,i}} \approx \frac{\widetilde z_{S,i}-\widetilde z_{T,i}} {V\tau^2}.$$

One $1/\tau$ comes from differentiating the scaled softmax. The other comes from teacher and student probabilities converging toward the uniform distribution. Multiplying $C$ by $\tau^2$ approximately cancels this shrinkage:

$$\frac{\partial(\tau^2 C)}{\partial z_{S,i}} \approx \frac{\widetilde z_{S,i}-\widetilde z_{T,i}}{V}.$$

This also explains the centered-logit connection in the high-temperature limit. Distillation approaches matching centered teacher and student logits, not arbitrary raw logits: adding a common offset changes neither softmax distribution.

The loss itself has the same limit. Let

$$a_i=z_{T,i}-\overline z_T, \qquad b_i=z_{S,i}-\overline z_S, \qquad \sum_i a_i=\sum_i b_i=0.$$

Expanding the centered exponentials one order further gives

$$\exp(a_i/\tau) = 1+\frac{a_i}{\tau}+\frac{a_i^2}{2\tau^2} +O(\tau^{-3}),$$

and similarly for $b_i$. Substituting the normalized expansions into KL and collecting the first nonzero term yields

$$\operatorname{KL}(p_T^\tau\Vert p_S^\tau) = \frac{1}{2V\tau^2} \sum_i(a_i-b_i)^2 +O(\tau^{-3}),$$

so

$$\lim_{\tau\to\infty} \tau^2\operatorname{KL}(p_T^\tau\Vert p_S^\tau) = \frac{1}{2V} \lVert\widetilde z_T-\widetilde z_S\rVert_2^2.$$

This proves the loss-level relationship that the gradient argument suggests. It also shows why the factor is compensation, not normalization to a fixed finite-temperature value: the omitted terms still depend on $\tau$.

Temperature also changes curvature. For fixed teacher probabilities, the exact one-row Hessian of the unscaled soft cross-entropy is

$$\frac{\partial^2 C} {\partial z_{S,i}\partial z_{S,k}} = \frac{1}{\tau^2} q_i(\mathbf{1}[i=k]-q_k).$$

Multiplication by $\tau^2$ removes the explicit curvature factor, but the matrix still changes because $q$ itself changes with temperature. Therefore tau_squared does not make optimization exactly temperature-invariant.

For the teacher/student rows used above, the centered-logit limits are

$$\frac{1}{2V}\lVert\widetilde z_T-\widetilde z_S\rVert_2^2 =0.84375,$$

and

$$\left\lVert \frac{\widetilde z_S-\widetilde z_T}{V} \right\rVert_2 \approx0.649519.$$

The exact sweep makes the distinction visible:

$\tau$	unscaled KL	$\tau^2$ KL	$\lVert\nabla C\rVert_2$	$\lVert\nabla(\tau^2C)\rVert_2$
1	0.241712	0.241712	0.355075	0.355075
2	0.169050	0.676198	0.178497	0.713990
4	0.058029	0.928465	0.051572	0.825147
8	0.014729	0.942637	0.011934	0.763754
16	0.003536	0.905247	0.002761	0.706878
32	0.000856	0.876934	0.000661	0.677239
$\infty$	$0$	$0.843750$	$0$	$0.649519$

The non-monotonic scaled columns matter. At $\tau=4$ and $8$, the compensated loss is above its asymptotic value. The factor preserves order of magnitude; it does not guarantee the same loss landscape or the same best mixture weight at every temperature.

The approximation has boundaries. It is strongest when $\tau$ dominates the centered logit scale. At moderate temperature the exact gradient remains $(q_i-p_i)/\tau$. The $\tau^2$ multiplier belongs to this temperature-scaled softmax argument; it is not a universal multiplier for hidden-state MSE, attention transfer, sequence-level targets, or arbitrary divergences.

0x02E Executable One-Row Oracle

The derivation is small enough to check without a framework. The article-local oracle uses stable softmax, evaluates KL and soft cross-entropy independently, compares the analytical gradient to central finite differences, and sweeps temperature toward the centered-logit limit:

def audit_logit_kd(z_teacher, z_student, tau):
    p = stable_softmax(z_teacher, tau)
    q = stable_softmax(z_student, tau)

    teacher_entropy = -sum(pi * log(pi) for pi in p)
    soft_cross_entropy = -sum(pi * log(qi) for pi, qi in zip(p, q))
    forward_kl = sum(pi * (log(pi) - log(qi)) for pi, qi in zip(p, q))

    exact_gradient = [(qi - pi) / tau for pi, qi in zip(p, q)]
    scaled_gradient = [tau**2 * gi for gi in exact_gradient]

    assert_close(soft_cross_entropy - teacher_entropy, forward_kl)
    assert_close(exact_gradient, central_difference_gradient(...))
    assert_close(sum(exact_gradient), 0.0)
    return forward_kl, scaled_gradient

The checked fixture uses teacher logits [8, 4, 1, -2], student logits [6, 5, 0, -1], temperatures from $1$ through $1024$, and independent common offsets for the two rows. Its report proves:

probabilities sum to one                         exact
finite positive temperature preserves ranking  exact
teacher/student common-offset invariance        exact
soft CE - teacher entropy = forward KL          exact
analytical gradient = finite difference         within numeric tolerance
gradient coordinates sum to zero                within numeric tolerance
Hessian rows sum to zero                         within numeric tolerance
tau^2 KL -> centered-logit MSE / (2V)            observed at tau = 1024
tau^2 gradient -> centered-logit gradient        observed at tau = 1024

This is numerical evidence for the stated algebra only. It does not train a student, establish a best temperature, reproduce Hinton et al.’s experiments, or predict optimization behavior for a full Transformer.

0x02F Classical Logit-KD Failure Lab

Mistake	Shape check	What changes	Required correction
teacher uses $\tau=4$, student uses $\tau=1$	passes	different distributions are compared	use the same $\tau$ inside the soft term
hard loss also uses $\tau=4$	passes	deployed $\tau=1$ target is weakened	compute hard NLL from unscaled student logits
omit $\tau^2$ while tuning temperature	passes	soft gradient shrinks roughly as $1/\tau^2$	scale the intended soft term or retune with explicit rationale
use $\operatorname{KL}(p_S\Vert p_T)$ accidentally	passes	mode-seeking reverse-KL gradient replaces teacher cross-entropy	state and test divergence direction
apply softmax before a log-softmax API	often passes	avoidable underflow and changed API semantics	start from logits and use stable log-softmax
average batch means with different valid-token counts	passes	short sequences receive disproportionate weight	reduce valid-token sums and counts
treat high temperature as always better	passes	noisy low teacher logits gain influence	validate $\tau$, $\alpha$, and student capacity together
call $\tau^2$ exact normalization	passes	finite-temperature loss and curvature still vary	log the exact sweep and tune the mixture under the chosen $\tau$
regress uncentered logits as if it were KL	passes	arbitrary teacher/student offsets become supervision	center both rows or retain the finite-temperature distribution objective
compare alpha values across different coefficient conventions	passes	hard/soft gradient ratios differ despite the same metadata	persist the expanded objective equation and reductions

Loop 2 closes the scalar and one-row mathematics. It does not yet prove which Transformer row is supervised, how padding is removed, how a vocabulary shard contributes to the global denominator, or how teacher state is scheduled. Those are the tensor and execution contracts of Loop 3.

0x03 Token-Level KD In A Transformer

Loop 2 proved the loss for one already aligned row. A language-model batch adds four independent address spaces around that row: batch ownership, causal position, ordered vocabulary event, and distributed rank. A finite KL scalar does not prove that any of those addresses agree.

This section owns the fixed-prefix token objective. It consumes the generic teacher-forcing, shifting, mask, and global-token-mean derivations from M3.9 and the ordered output-event interface from M3.8. It adds what those articles intentionally do not own: the identity between a teacher row and a student row, vocabulary support shared by two checkpoints, and the two different reduction axes introduced by data and vocabulary parallelism.

Loop 3 owns	Consumes	Defers
teacher/student causal row identity	M3.9 autoregressive shift and mask semantics	hidden, attention, value-relation, and layer maps to Loop 4
exact valid-row hard and soft numerators	Loop 2 one-row objective and gradients	cached sparse/full targets and omitted tails to Loop 8
ordered vocabulary-event compatibility	M3.8 output event interface	teacher placement, collectives, overlap, and fused kernels to Loop 9
mathematical data/vocabulary-parallel reductions	a declared averaged-gradient DDP convention	framework and GPU execution evidence to Loop 9

The owner question is therefore narrower than “how do I run distillation?”:

Which teacher and student tensor entries denote the same next-token event, and how can their exact token objective survive masks and two orthogonal partition axes?

0x03A One Row Identity, Three Alignment Conditions

For token sequence $x_{b,0:S}$, a decoder-only model factorizes

$$p(x_{b,1:S}\mid x_{b,0}) = \prod_{s=0}^{S-2} p(x_{b,s+1}\mid x_{b,0:s}).$$

The output head materializes one logit row for each already-observed prefix:

$$z^M_{b,s,:} = \operatorname{Head}_M(h^M_{b,s}), \qquad z^M\in\mathbb{R}^{B\times S\times V_M}, \qquad M\in\{T,S\}.$$

Under one external shift, output row $s$ predicts input token $x_{b,s+1}$:

$$Z^M=z^M_{:,:-1,:}\in\mathbb{R}^{B\times(S-1)\times V_M}, \qquad Y=x_{:,1:}\in\{0,\ldots,V_S-1\}^{B\times(S-1)}.$$

If token position $j$ owns target supervision through indicator $a_{b,j}$, the prediction-row mask is shifted by the same address rule:

$$m_{b,s}=a_{b,s+1}, \qquad m\in\{0,1\}^{B\times(S-1)}.$$

This gives one row identity, three alignment conditions:

1. Prefix identity: teacher and student row $(b,s)$ receive the same token IDs, position IDs, segment/packing metadata, and allowed visible prefix.
2. Event identity: teacher and student vocabulary coordinate $v$ names the same output event, or an explicit transport maps both to a declared common event space.
3. Ownership identity: teacher KL, hard-label NLL, row mask, and reduction denominator all refer to the same $(b,s)$ address.

The teacher probability at $(b,s)$ and the hard token $x_{b,s+1}$ are two targets for the same student row. Shifting teacher logits, student logits, labels, or masks independently creates a different objective. A model API that shifts labels internally is valid; an external loss adapter is also valid. Two owners shifting once each is a two-position bug with perfectly plausible shapes.

0x03B A Complete Causal Address Trace

Use a padded batch with $B=2$, $S=5$, and the ordered vocabulary [BOS, A, B, C, EOS, PAD]:

input_ids[0] = [BOS, A, B, EOS, PAD]
input_ids[1] = [BOS, C, EOS, PAD, PAD]

attention_mask = target_mask =
    [[1, 1, 1, 1, 0],
     [1, 1, 1, 0, 0]]

prediction rows: logits[:, :-1, :]   [2,4,6]
hard targets:    input_ids[:, 1:]    [2,4]
row ownership:   target_mask[:, 1:]  [2,4]

Batch	Prediction row $s$	Prefix ends with	Hard target $x_{s+1}$	Row valid?	Teacher/student object
0	0	BOS	A	1	both [0,0,:] rows over six ordered events
0	1	A	B	1	both [0,1,:] rows
0	2	B	EOS	1	both [0,2,:] rows
0	3	EOS	PAD	0	excluded before probability normalization
1	0	BOS	C	1	both [1,0,:] rows
1	1	C	EOS	1	both [1,1,:] rows
1	2	EOS	PAD	0	excluded
1	3	PAD	PAD	0	excluded

The rectangular schedule contains $B(S-1)=8$ prediction rows. The objective contains only

$$C=\sum_{b=1}^{B}\sum_{s=0}^{S-2}m_{b,s}=5$$

valid rows. The denominator is five, not eight, two sequences, or the number of nonempty ranks.

The figure adds a teacher/student identity layer to M3.9’s generic causal training trace. It does not claim that KD changes the autoregressive factorization. It requires both models to evaluate that same factorization on the same fixed prefixes.

0x03C Attention Visibility Is Not Loss Ownership

Four state objects are often called “the mask,” but they act at different locations:

Object	Typical shape	Acts on	KD-specific invariant
key-padding mask	broadcast from [B,S]	attention scores	teacher and student cannot see padded keys
causal/segment visibility	logical [B,S,S] or packed metadata	attention scores	both models use the same permitted prefix topology
target ownership	[B,S], shifted to [B,S-1]	loss rows	hard and soft targets select the same rows
teacher mode/gradient state	model state, not a tensor mask	teacher forward and graph	deterministic frozen target, no teacher backward graph

In response-only SFT, prompt tokens remain visible to later answer rows while their own target ownership is zero. Reusing the target mask as attention visibility deletes context. Conversely, a correct attention mask does not remove prompt or padding rows from the loss automatically.

There is also an order-of-operations issue. A common dense implementation computes log-softmax for all $B(S-1)$ rows and multiplies the final row losses by zero. That works only if excluded rows are finite. IEEE arithmetic does not make 0 * NaN equal zero. Gather valid rows before probability normalization:

row_valid = target_mask[:, 1:].to(bool)          # [B,S-1]
row_index = nonzero(row_valid)                   # [C,2]

t_rows = teacher_logits[:, :-1, :][row_valid]    # [C,V]
s_rows = student_logits[:, :-1, :][row_valid]    # [C,V]
labels = input_ids[:, 1:][row_valid]              # [C]

assert t_rows.shape == s_rows.shape == (labels.numel(), V)

This selection also removes vocabulary-sized work on excluded rows in an unfused loss. A fused implementation may keep rectangular tensors internally, but its semantic oracle must produce the same selected-row objective.

Packed independent documents add one more condition. A physical next token at a segment boundary is not automatically a target for the preceding segment. The collator’s segment-aware target ownership must be shifted into $m$ before teacher and student rows are gathered. Loop 3 consumes that boundary policy from M3.9; it does not infer it from token adjacency.

0x03D Exact Masked Hard And Soft Numerators

For each valid row $(b,s)$ and shared event $v$, define

$$p_{b,s,v} = \operatorname{softmax}_v\!\left(\frac{Z^T_{b,s,:}}{\tau}\right)_v, \qquad q_{b,s,v} = \operatorname{softmax}_v\!\left(\frac{Z^S_{b,s,:}}{\tau}\right)_v.$$

The temperature-scaled soft row and ordinary hard row are

$$k_{b,s} = \tau^2\sum_{v=1}^{V} p_{b,s,v}\log\frac{p_{b,s,v}}{q_{b,s,v}},$$ $$h_{b,s} = -\log \operatorname{softmax}\!\left(Z^S_{b,s,:}\right)_{Y_{b,s}}.$$

Keep their numerators and count separate:

$$S_{soft}=\sum_{b,s}m_{b,s}k_{b,s}, \qquad S_{hard}=\sum_{b,s}m_{b,s}h_{b,s}, \qquad C=\sum_{b,s}m_{b,s}.$$

For the coefficient convention used in this article,

$$L = \frac{\alpha S_{soft}+(1-\alpha)S_{hard}}{C}, \qquad C>0.$$

This ratio is the objective contract. Averaging each component over a different mask, dividing soft KL by $V$, or averaging per sequence before mixing changes it. Logging only the final scalar loses the information needed to reconcile those differences.

The Loop 3 fixture uses $\tau=4$ and $\alpha=0.7$. Its five rows produce:

S_soft = 1.1543226735934238
S_hard = 2.9774370771735104
S_mix  = 0.7*S_soft + 0.3*S_hard = 1.7012569946674498
C      = 5
L      = S_mix / C = 0.34025139893349

State	Shape before selection	Shape after selection	Owner and lifetime
token IDs, positions	[B,S]	row metadata [C]	immutable batch/compiler state
attention/segment state	broadcast or packed descriptor	consumed by both forwards	must describe the same prefix topology
target ownership	[B,S]	Boolean [C] through index selection	shifted exactly once
teacher logits	[B,S,V_T]	[C,V]	frozen, detached, discardable after target formation
student logits	[B,S,V_S]	[C,V]	graph-retaining student state
teacher/student log-probabilities	optional [B,S-1,V]	preferably [C,V] or vocabulary chunks	stable accumulation dtype required
row KL and NLL	[C]	[C]	summed before global normalization
soft numerator, hard numerator, count	three scalars	three globally reducible scalars	telemetry and backward normalization

Teacher freezing does not remove teacher inference memory, its logits, or the loss temporaries. It only changes gradient ownership. Runtime liveness and placement remain Loop 9 work.

0x03E Dense Online Tensor And Gradient Trace

The following semantic path deliberately selects valid rows before softmax and keeps the local numerator distinct from the globally normalized backward scalar:

teacher.eval()
student.train()
teacher.requires_grad_(False)

with no_grad():
    t_full = teacher(
        input_ids=input_ids,
        attention_mask=attention_mask,
        position_ids=position_ids,
    ).logits                                      # [B,S,V]

s_full = student(
    input_ids=input_ids,
    attention_mask=attention_mask,
    position_ids=position_ids,
).logits                                          # [B,S,V]

valid = target_mask[:, 1:].to(bool)               # [B,S-1]
t_rows = t_full[:, :-1, :][valid]                 # [C_local,V]
s_rows = s_full[:, :-1, :][valid]                 # [C_local,V]
labels = input_ids[:, 1:][valid]                  # [C_local]

assert teacher_tokenizer_fingerprint == student_tokenizer_fingerprint
assert teacher_special_token_map == student_special_token_map
assert t_rows.shape == s_rows.shape

# Probability normalization is performed in a declared stable dtype.
t_logp = log_softmax(t_rows.float() / tau, dim=-1)
s_logp = log_softmax(s_rows.float() / tau, dim=-1)
t_prob = exp(t_logp)

soft_rows = tau**2 * sum(t_prob * (t_logp - s_logp), dim=-1)
hard_rows = cross_entropy_per_row(s_rows.float(), labels)

soft_sum = soft_rows.sum()
hard_sum = hard_rows.sum()
count = tensor(labels.numel(), dtype=int64)
mixed_sum = alpha * soft_sum + (1.0 - alpha) * hard_sum

global_count = all_reduce_sum_detached(count, data_group)
if global_count == 0:
    reject_global_empty_supervision()

# For a DDP implementation that averages parameter gradients by W:
backward_loss = data_world_size * mixed_sum / global_count
backward_loss.backward()

teacher.eval(), frozen parameters, and no_grad() remain three different controls. Evaluation mode fixes training-only behavior such as dropout; parameter flags define optimizer ownership; no-grad avoids a teacher activation graph. The student remains in training mode and owns every gradient.

A locally empty rank is not automatically a globally empty update. It may contribute zero numerator and zero count while still participating in collectives. Only $\sum_r C_r=0$ makes the global objective undefined. Exactly how a runtime keeps an empty rank in the graph and schedule is a Loop 9 integration question.

0x03F Ordered Vocabulary Support And Transport Alternatives

Direct token KL is an elementwise comparison over events, not array slots. Let teacher support be $\Omega_T$, student support be $\Omega_S$, and common comparison support be $\Omega_C$. A transport map sends model events to common events:

$$\phi_T:\Omega_T\rightarrow\Omega_C, \qquad \phi_S:\Omega_S\rightarrow\Omega_C.$$

The induced teacher distribution is the pushforward

$$p_T^C(c) = \sum_{u\in\Omega_T:\phi_T(u)=c}p_T(u),$$

with the analogous definition for the student. KL is defined only after both pushforwards are normalized on the same $\Omega_C$.

The ordered-event contract has four cases:

Case	Operation	Exact object preserved	What must be declared
identical tokenizer and order	compare same indices directly	original token-event distribution	tokenizer/revision hash and special-token map
same events, different order	apply known bijective permutation	exact original event distribution	immutable event-ID permutation
many-to-one map into common events	sum probability mass before KL	distribution on a coarser event space	common event definition and loss of within-group distinctions
no defensible event map	direct token KL is undefined	none	use sequence, byte/span, sampled-action, or intermediate targets

Equal widths do not establish the first case. Two vocabularies of size 32,000 can assign unrelated byte strings to index 17. Matching only embedding/output matrix shapes is likewise insufficient. BOS, EOS, PAD, unknown, and added special tokens are semantic events and must be included in the fingerprint.

The executable fixture permutes the teacher order for one row:

teacher order expressed as student IDs = [2, 0, 5, 1, 4, 3]
baseline aligned KL                 = 0.016081816165513345
naive same-index KL                 = 0.18528895854109495
KL after explicit bijective remap   = 0.016081816165513345

The remap recovers the baseline exactly; equal vector length alone differs by $0.1692071424$. Truncating to min(V_T,V_S) is therefore not a compatibility strategy.

Cross-tokenizer alternatives change the supervised object:

Alternative	Strength	New contract or limitation
byte/span-aligned event projection	can retain local distribution guidance	token events must be composed into common spans and timing may differ
common auxiliary vocabulary head	gives both models one declared support	adds trainable/frozen projection state and may not match deployed head
sampled sequence/action supervision	avoids dense same-index support	transfers only sampled events and generation policy becomes provenance
hidden or relational transfer	bypasses output vocabulary	width/layer/head/token alignment moves to Loop 4
sequence-level black-box transfer	works with token-only teacher access	generation, filtering, and artifact identity move to Loop 6/8

Cached top-$k$ targets do not solve support alignment and are not automatically equal to dense KL. Their omitted tail and normalizer belong to Loop 8.

0x03G Data Parallelism: Numerators, Counts, And Backward Scale

Suppose data-parallel rank $r$ owns valid-row set $\mathcal R_r$. Its mixed token-sum numerator and count are

$$S_r = \sum_{(b,s)\in\mathcal R_r} \left[\alpha k_{b,s}+(1-\alpha)h_{b,s}\right], \qquad C_r=|\mathcal R_r|.$$

The declared global token mean is

$$L = \frac{\sum_{r=1}^{W}S_r}{\sum_{r=1}^{W}C_r} = \frac{S}{C}.$$

The mean of rank means is a different objective:

$$L_{rank} = \frac{1}{W}\sum_{r=1}^{W}\frac{S_r}{C_r}.$$

It gives each nonempty rank weight $1/W$ regardless of how many valid rows it owns. In the fixture:

Rank	Valid rows $C_r$	Mixed numerator $S_r$	Local mean
0	3	1.0568925556	0.3522975185
1	2	0.6443644391	0.3221822195
global	5	1.7012569947	0.3402513989

The unweighted mean of local means is $0.3372398690$, already wrong on a two-sequence toy batch.

The backward scaling needs one additional step. Let $g_r=\nabla_\theta S_r$. Under the declared convention that DDP averages parameter gradients across $W$ replicas, backpropagating

$$\widetilde L_r = \frac{W}{C}S_r$$

produces, after DDP reduction,

$$\frac{1}{W}\sum_{r=1}^{W}\nabla_\theta\widetilde L_r = \frac{1}{W}\sum_r\frac{W}{C}g_r = \frac{\sum_r g_r}{C} = \nabla_\theta L.$$

DDP’s average cancels the explicit world-size factor. Backpropagating $S_r/C_r$ instead leaves each rank equally weighted. The Loop 3 oracle compares every student-logit gradient entry:

two-rank W/C-scaled DDP max error       = 1.39e-17
two-rank local-mean DDP max error       = 1.67e-02
three-rank probe with one empty rank    = 6.94e-18

This specializes M3.9’s global-token derivation to a mixed hard/soft KD numerator. PyTorch DDP 2.13 is versioned implementation evidence for averaged replicated gradients, not a universal law for every distributed runtime. The reduction convention belongs in the run manifest.

0x03H Vocabulary Parallelism: Global Distribution, Local Events

Data parallelism partitions rows. Vocabulary parallelism partitions the event axis of each row. A vocabulary shard owns events, not complete distributions. Let rank $r$ own event set $\mathcal V_r$ and temperature-scaled teacher values $a^T_v=Z^T_v/\tau$. Stable global normalization begins with

$$m^T_r=\max_{v\in\mathcal V_r}a^T_v, \qquad m^T=\max_r m^T_r,$$

then

$$u^T_r = \sum_{v\in\mathcal V_r}\exp(a^T_v-m^T), \qquad U^T=\sum_r u^T_r.$$

Each rank can now evaluate its exact local log probabilities:

$$\log p_v = a^T_v-m^T-\log U^T, \qquad v\in\mathcal V_r.$$

Repeat the maximum and shifted-sum reductions for the student. The local KL contribution is

$$K_r = \sum_{v\in\mathcal V_r} p_v(\log p_v-\log q_v),$$

and one SUM reduction gives

$$K=\sum_r K_r = D_{KL}(p\Vert q).$$

After Loop 2’s $\tau^2$ scaling, each aligned student shard owns gradient slice

$$\frac{\partial(\tau^2K)}{\partial Z^S_v} = \tau(q_v-p_v), \qquad v\in\mathcal V_r.$$

Five row-shaped reductions replace one vocabulary all-gather in this explicit construction:

teacher: MAX row maximum, SUM shifted exponential denominator
student: MAX row maximum, SUM shifted exponential denominator
loss:    SUM local KL contribution

For selected rows [C,V] split over $P_V$ ranks, local logits are [C,V/P_V]; each logical collective above carries a [C] row array rather than a full [C,V] tensor. Collective implementations can fuse or reorganize this schedule, but they must preserve the same global normalization and gradient.

def sharded_log_probs(local_logits, tau, vocab_group):
    local_scaled = local_logits.float() / tau       # [C,V_r]
    local_max = local_scaled.max(dim=-1).values     # [C]
    global_max = all_reduce_max(local_max, vocab_group)
    local_exp_sum = exp(local_scaled - global_max[:, None]).sum(dim=-1)
    global_exp_sum = all_reduce_sum(local_exp_sum, vocab_group)
    return local_scaled - global_max[:, None] - log(global_exp_sum[:, None])

t_logp_r = sharded_log_probs(t_logits_r, tau, teacher_vocab_group)
s_logp_r = sharded_log_probs(s_logits_r, tau, student_vocab_group)

assert local_teacher_event_ids == local_student_event_ids
local_kl = sum(exp(t_logp_r) * (t_logp_r - s_logp_r), dim=-1)  # [C]
row_kl = all_reduce_sum(local_kl, aligned_vocab_group)           # [C]

This is communication algebra, not a drop-in autograd kernel. The student normalizer needs an autograd-aware collective or fused loss; the teacher can be detached. The pinned Megatron vocabulary-parallel cross-entropy source upcasts logits, reduces global row maxima and exponential sums, and retains local softmax slices for backward. That hard-label primitive grounds the reduction pattern. It does not prove that Megatron implements the exact KD kernel derived here.

If teacher and student shard boundaries differ, same-rank arrays may represent different event IDs even when widths match. One side must be repartitioned by event ID, or the KL must run where aligned teacher and student shards coexist. Teacher placement, process groups, overlap, and measured traffic remain Loop 9.

0x03I Executable Token And Partition Oracle

The article-local pure-Python oracle evaluates the complete $B=2$, $S=5$, $V=6$ fixture without a model framework or process group. It independently computes dense rows, a nontrivial vocabulary permutation, uneven data ranks, an empty-rank gradient probe, and two vocabulary shards:

Contract	Oracle result
valid rows / rectangular rows	5 / 8
correct shifted hard-target IDs	[1, 2, 4, 3, 4]
dense mixed token mean	0.34025139893349
invalid-row logit perturbation error	0
naive permuted-vocabulary KL error	0.1692071423755816
explicitly remapped-vocabulary KL error	0
mean-of-rank-means loss error	0.0030115299004488394
exact DDP student-logit gradient max error	1.39e-17
vocabulary-parallel normalization error	2.22e-16
vocabulary-parallel KL max error	1.77e-16
vocabulary-parallel soft-gradient max error	3.33e-16

The checks are stronger than a single expected scalar. Mutating excluded rows cannot change the selected loss; a double shift changes the target trace; an explicit vocabulary bijection must recover the baseline while naive indexing must not; uneven and locally empty data ranks must reproduce the dense gradient; every sharded row must normalize and match its dense KL and gradient slices.

The oracle establishes tensor algebra and ownership only. It did not instantiate a teacher or student, execute PyTorch DDP, create a process group, run a GPU kernel, train a checkpoint, reproduce a paper result, or measure communication.

0x03J Token-KD Failure Lab And Loop Boundary

Failure	Why shape checks can pass	Observable consequence	Repair owner
teacher and student receive different position/segment metadata	both emit [B,S,V]	KL compares different prefixes	input manifest and row-identity assertion
model and loss adapter both shift	both become compatible [B,S-2,*] tensors	target trace is two positions ahead	one named shift owner
attention mask reused as target mask	both are Boolean and broadcastable	prompt rows gain loss or context disappears	separate visibility and ownership state
excluded rows normalized before selection	final row mask still has zeros	padding NaN survives as 0*NaN	gather finite valid rows first
local empty rank aborts before global count	local count is zero	valid rows on other ranks are discarded	reject only globally empty supervision
rank-local means are averaged	each rank emits one scalar	shorter ranks receive excess weight	reduce numerators/counts or apply exact $W/C$ scaling
DDP world-size factor is omitted	training remains finite	gradient is smaller by $W$ under token-sum formulation	record reduction convention and derive backward scale
equal vocabulary widths assumed sufficient	KL returns a scalar	unrelated token events are matched	tokenizer/special-token fingerprint or event transport
known vocabulary permutation ignored	support sets are equal	loss differs solely because coordinates moved	reorder by immutable event map
many-to-one map treated as a permutation	output width may be coerced	probability mass is duplicated or dropped	sum mass on declared common events
local vocabulary softmax used	each shard sums to one	global probabilities and KL depend on TP degree	global max and shifted-sum reductions
teacher/student shards aligned by rank number only	local widths match	same positions denote different events	align/repartition by event ID
hard-label CE source cited as fused KD proof	reduction pattern looks similar	unsupported implementation claim	mark KD construction as local derivation
synthetic oracle called distributed execution	arithmetic is exact	performance and backend failures remain unknown	defer runtime evidence to Loop 9

Renderer-safe contract summary:

prediction row:       (b,s) -> prefix x[b,0:s] -> target x[b,s+1]
valid row:            m[b,s] = target_owner[b,s+1]
soft numerator:       sum_valid tau^2 * KL(p_T || p_S)
hard numerator:       sum_valid NLL_student(target)
global objective:     [alpha*S_soft + (1-alpha)*S_hard] / C_global
DDP local backward:   world_size * S_rank / C_global   # if gradients average
vocab-sharded KL:     SUM_r local_KL_r after global MAX/SUM normalizers

Loop 3 therefore closes classical full-logit KD from one row through causal tensor addresses and mathematical distributed ownership:

same token IDs + positions + visible prefix
  -> teacher and student [B,S,V] logits
  -> exactly one causal shift
  -> gather the same valid [C,V] rows
  -> align ordered vocabulary events
  -> exact hard and soft token numerators
  -> reduce row counts across data partitions
  -> normalize events across vocabulary partitions
  -> student-only gradient of one declared global objective

Loop 4 may now add internal-state targets. It must consume this exact token and mask identity while introducing explicit width projections, layer maps, head maps, and token-pair support. It may not use a hidden-state tensor to repair an already-misaligned causal row or vocabulary objective.

0x04 Intermediate Representation Transfer

The token objective from 0x03 asks the student to reproduce a conditional distribution. It does not require the student to reach that distribution by the same internal route. Intermediate transfer adds a second kind of supervision: it identifies a teacher-side object, identifies a student-side object, maps their incompatible axes, and declares a reduction. That sentence contains four separate design decisions. Calling all of them “feature distillation” hides the part that usually breaks.

There is no single Transformer tensor called the representation:

residual/hidden state:       [B,S,d]
query, key, value vectors:   [B,H,S,d_h]
pre-softmax attention score: [B,H,S,S]
attention distribution:     [B,H,S,S]
value-value relation:        [B,H,S,S]
relation-head vectors:       [B,A_r,S,d_r]
output distribution:        [B,S,V]

The batch and token axes consume Loop 3’s immutable example, tokenizer, position, prefix, and mask identities. Loop 4 owns the new axes:

depth axis:       which teacher layer supervises which student layer?
width axis:       direct coordinates, learned projection, normalization, or relation?
head axis:        same attention heads, explicit head map, or independent relation heads?
pair axis:        which query/key pairs exist, and over which key set is a row normalized?
state axis:       which alignment tensors train, checkpoint, resume, and disappear at export?
schedule axis:    joint objective, stage-wise hinting, general KD, or task-specific KD?

The paper lineage makes those distinctions concrete. FitNets uses a trainable regressor and a separate hint-training stage. Patient Knowledge Distillation, or PKD, matches normalized [CLS] states at selected layers. TinyBERT defines embedding, hidden, unnormalized attention-score, and prediction targets inside general and task-specific stages. MiniLM matches last-layer attention and value relations. MiniLMv2 repartitions concatenated Q/K/V coordinates into relation heads so teacher and student attention-head counts no longer have to match. They are not aliases for one projected MSE.

0x04A One Phrase, Five Different Target Objects

Before deriving a loss, write its target contract as a tuple:

$$\mathcal{C}_{int} = (o_T,o_S,g,\phi_T,\phi_S,\mathcal{M},D,R,\Theta_A),$$

where:

Field	Meaning	Example
$o_T,o_S$	teacher/student hook points	post-block hidden state, pre-softmax QK score, post-softmax relation
$g$	student-to-teacher layer map	student 2 -> teacher 5
$\phi_T,\phi_S$	coordinate transforms	identity, L2 normalization, learned width projection, relation construction
$\mathcal{M}$	token or query-key support	valid tokens, [CLS] only, bidirectional padding mask, causal pair mask
$D$	discrepancy	squared error, cosine distance, forward KL
$R$	reduction	per coordinate, per row, per head, per layer, then global count
$\Theta_A$	trainable alignment state	hidden projection, head mixer, or empty set

Two runs that differ in any field optimize different objectives even when both log them as intermediate_kd_loss. A reproducible run must serialize this tuple or an equivalent manifest, not only the scalar weights.

Five common target families preserve different information:

Target	Preserves	Discards or constrains
raw/projected hidden MSE	teacher coordinates, direction, and magnitude after projection	requires a width map and selected layers
normalized hidden loss	direction in teacher/student feature space	discards radius; needs a zero-norm policy
unnormalized attention-score MSE	token-pair score offsets and scale	needs compatible heads and finite-pair handling
attention-distribution KL	each query’s distribution over visible keys	discards row-wise additive score offsets; still constrains heads
Q/K/V relation KL	token-pair geometry after a dot product	discards raw coordinate basis; preserves mask and relation-head obligations

This is also why an intermediate target cannot repair a broken token contract. If teacher row $s$ predicts a different event or sees a different prefix from student row $s$, matching their hidden states merely forces two semantically different computations into the same coordinate system.

0x04B Hidden Projection Algebra, Reduction, And Exact Gradients

Let student layer $m$ produce

$$H^S_m\in\mathbb{R}^{B\times S\times d_S},$$

and let the immutable map $g$ select

$$H^T_{g(m)}\in\mathbb{R}^{B\times S\times d_T}.$$

The depth map $g$ does not solve width mismatch. For $d_S\ne d_T$, introduce a student-side projection

$$W_m\in\mathbb{R}^{d_S\times d_T}, \qquad \widehat H^S_m=H^S_mW_m.$$

The batch matrix multiplication is conceptually a row-wise map:

H_S:       [B,S,d_S]
flatten:   [B*S,d_S]
W_m:       [d_S,d_T]
project:   [B*S,d_T]
restore:   [B,S,d_T]

Let $M\in\{0,1\}^{B\times S}$ select valid token rows and $N=\sum_{b,s}M_{b,s}$. Define

$$E_m[b,s,:] = H^S_m[b,s,:]W_m-H^T_{g(m)}[b,s,:].$$

A per-coordinate masked objective is

$$L_{hid,m} = \frac{1}{Nd_T} \sum_{b=1}^{B}\sum_{s=1}^{S} M_{b,s}\|E_m[b,s,:]\|_2^2.$$

Why the denominator is $Nd_T$ follows directly from the claimed unit: a mean over valid scalar coordinates. A per-token mean would divide by $N$ and would be $d_T$ times larger for the same error tensor. Neither convention is universally correct, but the coefficient multiplying it cannot be interpreted without the convention.

Gather the $N$ valid rows into matrices $X\in\mathbb{R}^{N\times d_S}$ and $Y\in\mathbb{R}^{N\times d_T}$. Then

$$L(W)=\frac{1}{Nd_T}\|XW-Y\|_F^2.$$

Using $d\|A\|_F^2=2\langle A,dA\rangle$ and $d(XW)=X\,dW+dX\,W$ gives

$$dL = \frac{2}{Nd_T} \left\langle XW-Y, X\,dW+dX\,W\right\rangle.$$

Reordering each Frobenius inner product yields the exact gradients

$$\frac{\partial L}{\partial W} = \frac{2}{Nd_T}X^\top(XW-Y),$$

and

$$\frac{\partial L}{\partial X} = \frac{2}{Nd_T}(XW-Y)W^\top.$$

There is no teacher gradient because $Y$ is a detached target. The projection is therefore not a passive shape cast: it receives gradients, needs optimizer state, participates in clipping and mixed-precision scaling, and must be restored on resume. If it is training-only, it must not leak into the exported student graph.

A concrete trace makes the counts auditable:

B=2, S=4, d_S=3, d_T=4
valid rows per example: [3,2] -> N=5
student layers: [0,1]
teacher map:    [1,3]

each H_S:       [2,4,3]
each W:         [3,4] = 12 trainable scalars
each H_S @ W:   [2,4,4]
valid elements: 5*4 = 20 per matched layer
two projections: 24 trainable scalars before optimizer state

For $L_g$ matched layers, two reduction policies often get conflated:

$$L_{layer\ mean}=\frac{1}{L_g}\sum_{m=1}^{L_g}L_{hid,m},$$

versus

$$L_{global\ element} = \frac{\sum_m\sum_{b,s}M_{b,s}\|E_m[b,s,:]\|_2^2} {\sum_m N_m d_{T,m}}.$$

They coincide only when every mapped layer has the same valid-element count. Across data-parallel ranks, the second requires global numerator and count reductions, exactly like the token objective in 0x03. A mean of rank means changes example weights under uneven valid-row counts.

0x04C Normalized Hidden Geometry And PKD’s Exact Contract

Projected MSE is not the only way to compare hidden states. For a nonzero vector $h$, define

$$u(h)=\frac{h}{\|h\|_2}.$$

The differential is

$$du = \frac{1}{\|h\|_2} \left(I-uu^\top\right)dh.$$

The matrix $I-uu^\top$ projects onto the tangent space orthogonal to $u$. Consequently, a loss on normalized vectors sends no first-order gradient in the radial direction. Positive rescaling $h\mapsto ch$ leaves $u(h)$ unchanged; raw hidden MSE does not. This is the derivation behind the phrase “matches direction rather than magnitude.”

It also exposes a missing contract: $h=0$ has no direction. An implementation must reject zero-norm rows, define an epsilon-clamped normalization, or exclude those rows. Silent division by zero turns one hidden target into NaNs that can contaminate the full student update.

PKD uses this normalized geometry in a much narrower form than generic hidden distillation. In the paper’s BERT setup, $h^S_{i,j}$ and $h^T_{i,I_{pt}(j)}$ are the student and teacher [CLS] vectors for example $i$, not all token rows. Its patient objective is

$$L_{PT} = \sum_i\sum_j \left\| \frac{h^S_{i,j}}{\|h^S_{i,j}\|_2} - \frac{h^T_{i,I_{pt}(j)}}{\|h^T_{i,I_{pt}(j)}\|_2} \right\|_2^2.$$

The paper combines it with supervised and output-distribution losses:

$$L_{PKD} = (1-\alpha)L^S_{CE} +\alpha L_{DS} +\beta L_{PT}.$$

For its 12-layer teacher and 6-layer student, two published layer hypotheses are:

PKD-Skip teacher patient layers: [2,4,6,8,10]
PKD-Last teacher patient layers: [7,8,9,10,11]
student patient layers:          five intermediate student positions
teacher layer 12:                omitted from patient loss; output KD owns it

Source figure: PKD Figure 1, revision 1908.09355v1. It illustrates the paper’s two layer-map contracts; it is not a reproduced accuracy result.

The map is objective state. A framework may return hidden states as [embedding_output, block_1, ..., block_L] while a paper numbers only Transformer blocks. An unrecorded +1 offset can select the wrong teacher layer while every tensor shape remains valid.

def resolve_layer_map(hidden_states, map_spec):
    # The manifest declares whether hidden_states[0] is the embedding output.
    offset = 1 if map_spec.includes_embedding_slot else 0
    resolved = []
    for student_block, teacher_block in map_spec.block_pairs:
        s_idx = student_block + offset
        t_idx = teacher_block + offset
        assert map_spec.student_revision == loaded_student_revision
        assert map_spec.teacher_revision == loaded_teacher_revision
        resolved.append((s_idx, t_idx))
    return tuple(resolved)

Layer indices, indexing convention, selected token rows, normalization, and loss coefficient all belong in the checkpoint manifest. Loading tensors while changing one of those fields is not a faithful resume.

0x04D FitNets: A Regressor Plus A Two-Stage Training Program

FitNets introduced intermediate hints before Transformer-specific distillation. Its teacher hint layer emits $u_h(x;W_{Hint})$; the student guided layer emits $v_g(x;W_{Guided})$. A regressor $r(\cdot;W_r)$ maps the guided output to the hint shape. The paper writes

$$L_{HT}(W_{Guided},W_r) = \frac{1}{2} \left\| u_h(x;W_{Hint}) -r(v_g(x;W_{Guided});W_r) \right\|_2^2.$$

The key mechanism is not only the equation. It is the schedule:

stage 1: freeze teacher
         train student only through the guided layer + regressor
         objective = hint reconstruction

transition:
         retain learned student prefix
         discard the training-only regressor unless explicitly reused
         initialize/activate the rest of the student

stage 2: train the whole student
         objective = output-level knowledge distillation

The paper does not define FitNets as a joint sum of hint and output losses from step zero. It reports stage-wise optimization and discusses deeper guided layers as a possible over-regularization risk: the later the student is forced to reproduce a teacher hint, the fewer downstream layers remain to reorganize that constraint.

The original Figure 1 separates teacher/student architecture, hint training, and output distillation. The following retained panels are source context, not a claim that the CNN regressor can be copied unchanged into a Transformer:

Teacher/student networks	Hint-training stage	Output-KD stage

Source panels: FitNets Figure 1, revision 1412.6550v4. The mechanism is a historical training-program contract; published outcomes remain source-reported.

For a Transformer hidden projection, the analogous stage-one state is:

trainable:
    student blocks 0..guided_block
    W_projection or alignment regressor

frozen:
    teacher
    student blocks after guided_block, if already instantiated

checkpoint:
    student prefix
    alignment parameters
    optimizer/scaler/scheduler state
    exact hint/guided hook names and layer indices

stage transition:
    copy retained prefix into full student
    record whether alignment state is discarded or remains an auxiliary loss

This schedule belongs to Loop 4 because it changes who receives intermediate gradients. Choosing the final student architecture and initialization policy remains Loop 5.

0x04E TinyBERT: A Piecewise Layer Objective, Not One Loss

TinyBERT makes the target surface explicit. Let a teacher have $N$ Transformer layers and a student have $M$ layers. The paper defines a map $g(m)$ that includes boundary objects:

$$0=g(0), \qquad N+1=g(M+1).$$

Index 0 denotes the embedding layer, indices 1..M denote student Transformer layers mapped into teacher layers, and M+1 denotes the prediction layer. The model-level objective is a sum of target-specific layer losses:

$$L_{TinyBERT} = \sum_{m=0}^{M+1}\lambda_m L_{layer}(m).$$

Its piecewise contract is

$$L_{layer}(m)= \begin{cases} L_{embd}, & m=0,\\ L_{hidn}+L_{attn}, & 0<m\le M,\\ L_{pred}, & m=M+1. \end{cases}$$

Embedding And Hidden Targets

When widths differ, TinyBERT uses learned transforms:

$$L_{embd} = \operatorname{MSE}(E^S W_e,E^T),$$

and

$$L_{hidn} = \operatorname{MSE}(H^S W_h,H^T).$$

The equations look like 0x04B, but the implementation must still pin the MSE reduction and valid-token support. The published BERT setting and a padded causal decoder do not imply the same mask semantics.

Unnormalized Attention Scores

For each aligned head $i$, TinyBERT writes

$$L_{attn} = \frac{1}{h} \sum_{i=1}^{h} \operatorname{MSE}(A_i^S,A_i^T),$$

where the paper uses the unnormalized attention matrices and reports that choice as converging faster and performing better in its experiments. This is not MiniLM’s KL between row-softmax distributions.

For a masked Transformer, the safe numerator is over finite valid pairs:

$$N_{pair} = \sum_{b,h,q,k}M^{pair}_{b,q,k},$$ $$L_{score} = \frac{ \sum_{b,h,q,k}M^{pair}_{b,q,k} (Z^S_{b,h,q,k}-Z^T_{b,h,q,k})^2 }{N_{pair}}.$$

Selecting valid pairs before subtraction matters. If both masked scores are -inf, then (-inf)-(-inf) is nan; multiplying the resulting scalar by zero does not repair it.

General And Task-Specific Stages

TinyBERT’s result also depends on a two-stage learning framework:

general distillation:
    teacher = original pretrained BERT
    data = large general-domain text corpus
    targets = embedding + intermediate Transformer targets
    paper footnote excludes prediction-layer loss in this stage

task-specific distillation:
    teacher = task-fine-tuned BERT
    data = task data plus the paper's augmentation procedure
    targets = task-specific intermediate + prediction transfer

Source figure: TinyBERT Figure 1, revision 1909.10351v5. It identifies the two published stages; it does not make the source-reported benchmark portable to another corpus, architecture, or augmentation pipeline.

Source figure: TinyBERT Figure 2, revision 1909.10351v5. It shows the paper’s hidden and attention target paths; the exact mask and reduction still belong to the implementation contract.

Collapsing both stages into one joint fine-tuning run is an alternative method, not a faithful implementation of TinyBERT. Likewise, retaining only hidden MSE while citing the full TinyBERT result removes the attention, prediction, general-corpus, and augmentation contracts that contributed to that result.

0x04F Attention Scores, Attention Distributions, And Mask Algebra

Let one layer expose

$$Q,K,V\in\mathbb{R}^{B\times H\times S\times d_h}.$$

For query $q$ and key $k$, define a Boolean support $M^{pair}_{b,q,k}$. In a bidirectional encoder, it usually excludes padding. In a causal decoder, it also requires $k\le q$. The score and normalized attention row are

$$Z^{QK}_{b,h,q,k} = \frac{\langle Q_{b,h,q,:},K_{b,h,k,:}\rangle}{\sqrt{d_h}},$$ $$A_{b,h,q,k} = \frac{M^{pair}_{b,q,k}\exp Z^{QK}_{b,h,q,k}} {\sum_{j:M^{pair}_{b,q,j}=1}\exp Z^{QK}_{b,h,q,j}}.$$

The denominator derives the softmax axis: a fixed query selects a distribution over visible keys. Normalizing over queries instead produces a valid-shaped tensor that answers a different question.

For teacher distribution $A^T$ and student distribution $A^S$, a masked forward KL is

$$L_{attn} = \frac{1}{HN_q} \sum_{b,h,q}M^q_{b,q} \sum_{k:M^{pair}_{b,q,k}=1} A^T_{b,h,q,k} \log\frac{A^T_{b,h,q,k}}{A^S_{b,h,q,k}},$$

where $M^q$ excludes query rows with no semantic target and $N_q=\sum M^q$. The key mask is used inside normalization; the query mask is used outside it for reduction. One mask cannot substitute for the other.

def masked_row_distribution(scores, pair_mask):
    # scores: [B,H,Q,K], pair_mask: [B,1,Q,K]
    assert pair_mask.dtype == bool
    assert pair_mask.any(dim=-1).all()  # no empty normalized row
    neg_inf = full_like(scores, -inf)
    visible_scores = where(pair_mask, scores, neg_inf)
    row_max = visible_scores.max(dim=-1, keepdim=True).values
    exp_shifted = where(pair_mask, exp(visible_scores - row_max), 0.0)
    return exp_shifted / exp_shifted.sum(dim=-1, keepdim=True)

def masked_row_forward_kl(teacher_p, student_p, query_mask):
    rows = query_mask[:, None, :, None]
    numerator = where(
        rows & (teacher_p > 0),
        teacher_p * (log(teacher_p) - log(student_p)),
        0.0,
    ).sum()
    denominator = query_mask.sum() * teacher_p.shape[1]
    return numerator, denominator

The hook point is part of the target. For rotary-position models, Q/K can be captured before or after the positional rotation. For grouped-query attention, K/V can be stored with $H_{kv}$ heads or expanded to $H_q$. Teacher and student must expose the same semantic stage; equal shapes are insufficient.

Head count is another objective decision:

Choice	What it preserves	What it costs
require $H_T=H_S$	direct head identity	constrains student architecture
group teacher heads	coarse head groups	destroys within-group identity
learned head mixer	adaptable alignment	trainable/checkpointed alignment state
average over heads	head-agnostic token relation	removes head specialization
independent relation heads	width/head flexibility	changes the target to a MiniLMv2-style relation

The Attention Transfer lineage uses the word attention differently: it reduces CNN feature channels into normalized spatial activation maps and aligns selected teacher/student groups. That work motivates relation-like, width-independent supervision, but its activation map is not a Transformer QK attention matrix.

Source figure: Attention Transfer Figure 5, revision 1612.03928v3. It is historical representation-transfer context, not a Transformer attention-score implementation diagram.

0x04G MiniLM: Last-Layer Attention And Value Relations

MiniLM avoids raw hidden-coordinate matching. It selects the last teacher Transformer layer and the last student layer, then transfers two row distributions.

For head $a$, query-key attention is

$$A^T_{L,a} = \operatorname{softmax} \left( \frac{Q^T_{L,a}(K^T_{L,a})^\top}{\sqrt{d_k}} \right),$$

with the analogous student matrix $A^S_{M,a}$. The paper’s attention objective is

$$L_{AT} = \frac{1}{A_h|x|} \sum_{a=1}^{A_h}\sum_{t=1}^{|x|} \operatorname{KL} \left( A^T_{L,a,t}\|A^S_{M,a,t} \right).$$

For values, form

$$VR^T_{L,a} = \operatorname{softmax} \left( \frac{V^T_{L,a}(V^T_{L,a})^\top}{\sqrt{d_k}} \right),$$

and

$$VR^S_{M,a} = \operatorname{softmax} \left( \frac{V^S_{M,a}(V^S_{M,a})^\top}{\sqrt{d'_k}} \right).$$

The value-relation objective is

$$L_{VR} = \frac{1}{A_h|x|} \sum_{a=1}^{A_h}\sum_{t=1}^{|x|} \operatorname{KL} \left( VR^T_{L,a,t}\|VR^S_{M,a,t} \right),$$

and the published transfer objective is

$$L_{MiniLM}=L_{AT}+L_{VR}.$$

Source figure: MiniLM Figure 1, revision 2002.10957v2. It shows the original last-layer relation contract; the paper’s results remain source-reported.

Why The Relation Removes Width Coordinates

For one head, $V\in\mathbb{R}^{S\times d_h}$ but $VV^\top\in\mathbb{R}^{S\times S}$. The inner width contracts, so teacher and student can have different $d_h$ while their token-pair matrices still match. This is a precise form of architectural freedom, not complete invariance.

If $R$ is orthogonal, then

$$(VR)(VR)^\top = VRR^\top V^\top = VV^\top.$$

The relation cannot distinguish an orthogonal change of feature basis. A raw hidden MSE can. But scaling $V\mapsto cV$ multiplies pair logits by $c^2$ and changes row-softmax sharpness, so the target is not scale-invariant. A general non-orthogonal transform also changes the relation.

The original MiniLM contract still assumes the same number of attention heads $A_h$ on teacher and student sides. Width independence should not be misquoted as arbitrary head-count compatibility. MiniLMv2 is the separate method that relaxes that restriction.

0x04H MiniLMv2: Concatenate, Repartition, Then Compare Relations

MiniLMv2 decouples relation heads from model attention heads. For one of $Q,K,V$, start with

teacher tensor: [B,H_T,S,d_hT]
student tensor: [B,H_S,S,d_hS]

At each token, concatenate all model-head coordinates:

teacher: [B,H_T,S,d_hT]
      -> [B,S,H_T*d_hT] = [B,S,d_T]

student: [B,H_S,S,d_hS]
      -> [B,S,H_S*d_hS] = [B,S,d_S]

Choose one common relation-head count $A_r$ such that $A_r\mid d_T$ and $A_r\mid d_S$. Repartition each flat width:

$$d_r^T=\frac{d_T}{A_r}, \qquad d_r^S=\frac{d_S}{A_r},$$

teacher relation vectors: [B,A_r,S,d_rT]
student relation vectors: [B,A_r,S,d_rS]

The chunks need not respect original attention-head boundaries. Therefore $A_r$ need not equal, or be a multiple of, $H_T$ or $H_S$. It must divide the total concatenated widths. This distinction is what makes an 8-head teacher and a 6-head student comparable when both widths admit the selected $A_r$.

For $i,j\in\{Q,K,V\}$, relation head $a$ defines

$$R^T_{ij,l,a} = \operatorname{softmax} \left( \frac{A^T_{i,l,a}(A^T_{j,l,a})^\top}{\sqrt{d_r^T}} \right),$$

and

$$R^S_{ij,m,a} = \operatorname{softmax} \left( \frac{A^S_{i,m,a}(A^S_{j,m,a})^\top}{\sqrt{d_r^S}} \right).$$

The pair-specific loss is

$$L_{ij} = \frac{1}{A_r|x|} \sum_{a=1}^{A_r}\sum_{t=1}^{|x|} \operatorname{KL} \left( R^T_{ij,l,a,t}\|R^S_{ij,m,a,t} \right),$$

and the general objective is

$$L_{MiniLMv2} = \sum_i\sum_j\alpha_{ij}L_{ij}.$$

The paper selects Q-Q, K-K, and V-V relations as a performance/compute trade-off rather than instantiating all nine ordered pairs. When relation heads coincide with attention heads, the Q-K relation recovers ordinary attention weights; Q-Q, K-K, and V-V encode different token-pair geometries.

Source figure: MiniLMv2 Figure 1, revision 2012.15828v2. It establishes the paper’s relation-head mechanism; its layer-selection and benchmark outcomes are source-reported.

def relation_heads(x, relation_head_count):
    # x: [B,H,S,d_h]
    B, H, S, d_h = x.shape
    total_width = H * d_h
    assert total_width % relation_head_count == 0
    d_r = total_width // relation_head_count

    # Preserve token-major coordinate order while joining model heads.
    flat = x.transpose(1, 2).reshape(B, S, total_width)
    rel = flat.reshape(B, S, relation_head_count, d_r)
    return rel.transpose(1, 2)  # [B,A_r,S,d_r]

def relation_distribution(left, right, pair_mask):
    # left/right: [B,A_r,S,d_r]
    assert left.shape == right.shape
    logits = left @ right.transpose(-1, -2) / sqrt(left.shape[-1])
    return masked_row_distribution(logits, pair_mask)

Changing transpose/reshape order permutes coordinates among relation heads and changes the objective without changing the final shape. The flatten order, relation-head count, Q/K/V pair set, teacher layer, and student layer therefore belong in the immutable training manifest.

MiniLMv2’s layer policy is also source-dependent. The paper reports the last teacher layer as effective for its base-teacher settings and upper-middle layers for some large teachers. Those are paper experiments, not a universal rule. A production study must treat teacher-layer choice as a measured hyperparameter while freezing it for each resumable run.

0x04I End-To-End Tensor And Gradient Ownership Trace

The following trace combines projected hidden loss with MiniLMv2-style relations. It is an implementation contract, not a claim that the cited papers jointly train this exact objective.

teacher.eval()
student.train()
alignment.train()

layer_map = ((0, 1), (1, 3))
relation_spec = {
    "teacher_layer": 3,
    "student_layer": 1,
    "relation_heads": 2,
    "pairs": (("q", "q"), ("k", "k"), ("v", "v")),
}

with no_grad():
    teacher_out = teacher(
        input_ids,
        attention_mask=attention_mask,
        output_hidden_states=True,
        capture_qkv_layers={3},
    )

student_out = student(
    input_ids,
    attention_mask=attention_mask,
    output_hidden_states=True,
    capture_qkv_layers={1},
)

valid_token = target_row_mask.to(bool)            # Loop 3 identity
pair_mask = build_query_key_mask(
    padding_mask=attention_mask,
    causal=student.is_causal,
)                                                  # [B,1,S,S]

hidden_num = scalar_zero()
hidden_den = integer_zero()
for student_layer, teacher_layer in layer_map:
    hs = student_out.hidden_states[student_layer]  # [B,S,d_S], has graph
    ht = teacher_out.hidden_states[teacher_layer]  # [B,S,d_T], detached
    aligned = alignment[student_layer](hs)         # [B,S,d_T], has graph

    valid_error = (aligned - ht)[valid_token]      # [N,d_T]
    hidden_num += valid_error.float().square().sum()
    hidden_den += valid_error.numel()

tq, tk, tv = teacher_out.qkv[3]
sq, sk, sv = student_out.qkv[1]

t_rel = {
    "q": relation_heads(tq, relation_spec["relation_heads"]),
    "k": relation_heads(tk, relation_spec["relation_heads"]),
    "v": relation_heads(tv, relation_spec["relation_heads"]),
}
s_rel = {
    "q": relation_heads(sq, relation_spec["relation_heads"]),
    "k": relation_heads(sk, relation_spec["relation_heads"]),
    "v": relation_heads(sv, relation_spec["relation_heads"]),
}

relation_num = scalar_zero()
relation_den = integer_zero()
for left_name, right_name in relation_spec["pairs"]:
    teacher_p = relation_distribution(
        t_rel[left_name], t_rel[right_name], pair_mask
    )
    student_p = relation_distribution(
        s_rel[left_name], s_rel[right_name], pair_mask
    )
    num, den = masked_row_forward_kl(
        teacher_p, student_p, query_mask=valid_token
    )
    relation_num += num
    relation_den += den

# Exact global means require numerator/count reductions, not rank-mean averaging.
hidden_num, hidden_den = all_reduce_sum(hidden_num, hidden_den)
relation_num, relation_den = all_reduce_sum(relation_num, relation_den)

hidden_loss = hidden_num / hidden_den
relation_loss = relation_num / relation_den
loss = token_kd_loss + beta_h * hidden_loss + beta_r * relation_loss
loss.backward()

assert all(p.grad is None for p in teacher.parameters())
assert any(p.grad is not None for p in student.parameters())
assert all(p.grad is not None for p in alignment.parameters())

capture_qkv_layers is a model-specific hook, not a standard API. Its schema must state:

hook location:       pre- or post-positional transform
head representation: native H_q/H_kv or expanded heads
layout:              [B,H,S,d_h] and exact coordinate order
mask topology:       bidirectional, causal, prefix, sliding-window, or custom
dtype:               storage, score accumulation, softmax, and loss reduction
gradient owner:      detached teacher, differentiable student

A hook taken from fused attention may not expose Q/K/V at all. Recomputing them from block inputs is valid only if it repeats the same projection, bias, normalization, positional transform, and head expansion as the actual attention path.

0x04J Liveness, Temporary Traffic, And Checkpoint State

Intermediate targets can dominate training memory even when they add no serving parameters. For $L_g$ hidden targets, a simple payload ledger is

$$M_{hidden,target} = \sum_{m=1}^{L_g}BSd_{T,m}b,$$

before student activations, projected states, gradients, or alignment optimizer state. A dense token-pair relation has

$$M_{relation} = BA_rS^2b.$$

For $B=8$, $A_r=24$, $S=2048$, and $b=2$ bytes, one dense relation tensor is

$$8\times24\times2048^2\times2 = 1.5\ \text{GiB}.$$

Teacher and student Q-Q, K-K, and V-V relations would be six such payloads if all were materialized simultaneously. That is not required by the algebra. A row-chunked implementation can compute teacher and student relation rows, accumulate KL numerator/count, and release each chunk. The chunked schedule is equivalent only if every row is normalized over its full visible key set. Chunking the key dimension and applying a local softmax changes the objective unless an online global-softmax merge is used.

Alignment state also has a persistent ledger. For projections $W_m\in\mathbb{R}^{d_{S,m}\times d_{T,m}}$,

$$P_{align}=\sum_m d_{S,m}d_{T,m}.$$

With parameter bytes $b_p$, gradient bytes $b_g$, and optimizer bytes $b_o$,

$$M_{align,persistent} = P_{align}(b_p+b_g+b_o).$$

This is training state, not student serving size. Loop 5 owns the full teacher/student/alignment memory and capacity ledger; Loop 4 only closes the fact that trainable alignment tensors exist and must survive resume.

Minimum checkpoint fields are:

teacher revision and hook schema
student revision and hook schema
tokenizer/position/mask identity consumed from Loop 3
layer map plus embedding-offset convention
hidden projection tensors and optimizer state
target family and exact reductions
attention or relation head policy
relation flatten/repartition order and Q/K/V pair set
loss weights and schedule stage
global step, scheduler, scaler, RNG, and data cursor

Cached teacher intermediates add artifact fields: source example identity, teacher revision, layer/hook, dtype, token mask, pair mask, positional stage, normalization, and checksum. Loop 8 owns their full storage and invalidation schema.

0x04K Paper-By-Contract Comparison

Method	Target object	Depth policy	Width/head policy	Optimization schedule	What must not be generalized
FitNets	teacher hint reconstructed by student guided layer + regressor	selected hint/guided pair	learned regressor; CNN source design	hint stage, then whole-student KD stage	not a joint Transformer hidden-loss recipe
Attention Transfer	normalized activation-derived spatial maps	selected residual groups	channel-reduced spatial map	auxiliary attention loss in CNN training	not Transformer QK attention
PKD	normalized [CLS] hidden states + output KD	PKD-Skip or PKD-Last	published direct vector match	task-specific joint objective	not all-token hidden MSE
DistilBERT	soft outputs, MLM, cosine hidden direction	reduced-depth BERT; one hidden direction target	same hidden width in published student	pretraining triple loss	initialization and architecture move to Loop 5
TinyBERT	embedding, hidden, unnormalized attention scores, prediction	explicit $g(m)$ including boundaries	learned width transforms; aligned attention heads	general stage, then augmented task-specific stage	full result cannot be attributed to hidden MSE alone
MiniLM	QK attention distributions + VV relations	last teacher and student layers	width may differ; original heads align	task-agnostic pretraining transfer	width flexibility is not arbitrary head compatibility
MiniLMv2	selected Q/K/V relation distributions	one teacher layer to student last layer	concatenate then split into common relation heads	relation distillation	layer choices and outcomes are source-reported settings

These rows are design contracts, not a leaderboard. Paper outcomes are source-reported for their data, architecture, objective, schedule, and runtime. An ablation that removes one contract must be named as an ablation rather than presented under the full paper label.

The alternatives answer different engineering constraints:

Constraint	Plausible target	Trade-off
exact teacher coordinate recovery matters	projected hidden MSE	projection state and coordinate sensitivity
magnitude should not dominate	normalized hidden/cosine	zero-norm policy and radial information loss
student must preserve score calibration	unnormalized score MSE	finite-pair handling and head compatibility
only query-to-key behavior matters	attention KL	score offsets/scales become invisible after softmax
teacher/student widths differ	MiniLM value relations	original head-count restriction remains
widths and attention-head counts differ	MiniLMv2 relation heads	reshape order and relation-head count become contracts
training a very thin/deep student is unstable	FitNets-style staged hinting	extra stage and disposable regressor state
runtime cannot expose internal Q/K/V	hidden or output target	relational attention objective may be unavailable

0x04L Failure Closure And Loop Boundary

Failure	Why ordinary checks miss it	Observable consequence	Required repair
layer map shifts by one because embeddings occupy slot 0	all hidden tensors share [B,S,d]	plausible but wrong supervision depth	pin block numbering and resolved tensor indices
per-token MSE is labeled per-coordinate MSE	scalar remains finite	objective weight changes by $d_T$	serialize denominator convention
projection omitted from optimizer	forward shapes pass	alignment never learns	assert gradient and optimizer ownership
projection omitted from checkpoint	fresh run works	resume changes the objective trajectory	checkpoint alignment and optimizer state
training wrapper exported with projection	model loads	serving graph contains training-only state	export student artifact independently
normalized target receives a zero vector	tensor shape passes	NaN loss/gradient	reject, clamp, or explicitly exclude zero norm
PKD expanded from [CLS] to all tokens without renaming	formula still runs	different target and row weighting	label as a new all-token variant
TinyBERT score loss runs on masked -inf pairs	masks look correct	inf-inf -> nan	gather finite valid pairs before subtraction
attention softmax uses query axis	probabilities still sum to one on an axis	wrong conditional relation	assert key-axis row sums on valid support
BERT mask copied into a causal decoder	matrix dimensions match	future information leaks into target	derive pair support from actual student topology
MiniLM used with unequal heads by broadcasting	framework may broadcast or reshape	silent head mixing	preserve/map heads or use MiniLMv2 contract
relation reshape swaps head/token coordinate order	output is still [B,A_r,S,d_r]	different relation partitions	pin flatten order and reconstruct in oracle
teacher pre-RoPE Q compared with student post-RoPE Q	shapes match	semantically different geometry	capture identical hook stage
row relations normalized inside key chunks	each chunk forms a distribution	objective differs from full-key softmax	use full-key rows or exact online merge
rank-local means averaged under uneven rows	distributed scalar looks stable	rank-dependent sample weights	reduce global numerator and count
paper label retained after stage/target removal	citation remains valid	unsupported benchmark implication	name the ablation and re-evaluate

Loop 4 has now closed the intermediate-transfer object, map, mask, reduction, gradient, schedule, and checkpoint contracts. It deliberately does not own:

Loop 5:
    student depth/width/head/FFN architecture choice
    alternating-layer or other student initialization
    teacher-assistant capacity-gap programs
    full teacher/student/alignment persistent-state and compute ledger

Loop 8:
    cached hidden/relation artifact serialization and invalidation

Loop 9:
    teacher placement, distributed hook implementation, collectives,
    pipeline/context parallel integration, and production scheduling

The immediate handoff is therefore not “add more losses.” It is: choose a student graph whose serving cost is worth the transfer program, then prove that the selected intermediate contracts are available and affordable for that graph.

0x05 Student Architecture, Initialization, And Capacity Gap

Intermediate transfer can recover behavior only inside the student graph that was chosen. That graph determines deployment cost and also determines which teacher targets are mathematically available. Reducing “model size” along one axis does not have the same alignment effect as reducing it along another.

This loop therefore owns a different object from the preceding loss loops:

owns:
    the exact student graph and checkpoint tensor set
    the initialization transition into that graph
    architecture-induced target compatibility
    teacher / student / alignment persistent state
    activation liveness and peak-memory accounting
    direct versus teacher-assistant training paths
    resumable training state versus student-only export

consumes:
    0x02 scalar output-distillation convention
    0x03 causal token rows and vocabulary support
    0x04 intermediate target, map, mask, and gradient contracts
    M4.1 parameter, FLOP, byte, and liveness conventions

defers:
    0x06 generated sequence artifacts and black-box supervision
    0x07 student-generated prefix distributions
    0x08 serialized supervision artifact schemas
    0x09 distributed teacher placement and production scheduling

The causal direction is important:

$$\text{student architecture} \longrightarrow \begin{cases} \text{deployment graph and state},\\ \text{logical forward work},\\ \text{available initialization maps},\\ \text{available white-box transfer targets}. \end{cases}$$

Distillation does not create structural savings by itself. It is a training program that tries to recover behavior after a deployment graph has already been selected. A same-shape student with deleted training-only losses remains a same-shape serving model.

0x05A Five Architecture Axes, Five Alignment Consequences

Student change	Deployment effect	Distillation consequence
fewer layers	fewer sequential blocks and block parameters	requires layer map, last-layer target, or no layer-wise match
smaller hidden width $d$	smaller residual/attention projections	raw hidden coordinates need projection or relational target
smaller FFN width $d_{ff}$	smaller MLP parameters and compute	neuron-wise MLP matching is undefined without another map
fewer/different Q or KV heads	different attention compute and KV layout	head-wise targets require grouping, mapping, or relation heads
different vocabulary/tokenizer	smaller/different embedding and output support	direct token KL may become semantically undefined

Keeping the architecture unchanged avoids most alignment work, but also creates no guaranteed structural saving. Conversely, a radically different student can be an excellent deployment target while making white-box coordinate losses inappropriate. Sequence-level or output-only transfer may be the cleaner contract in that case.

Five axes are enough to introduce the problem, but they are not enough to reconstruct a model. A checkpoint-compatible architecture manifest needs at least

$$\mathcal{A} = \left( L,V,P,T,d_e,d,d_{ff},H_q,H_{kv}, \mathcal{N},\phi,\mathcal{T} \right),$$

where:

Symbol	Meaning	Why KD cares
$L$	block count	determines sequential depth and teacher-layer map
$V$	vocabulary size and token identity	determines embedding/output support
$P,T$	position and token-type tables	determine input-state keys and compatibility
$d_e$	embedding width	may require an embedding-to-residual projection
$d,d_{ff}$	residual and FFN widths	determine block parameters, work, and hidden projection shape
$H_q,H_{kv}$	query and KV head layouts	determine attention tensor interpretation and cache shape
$\mathcal N,\phi$	normalization placement/type and activation	determine graph semantics even when shapes match
$\mathcal T$	weight-tying and alias contract	determines unique parameter storage and checkpoint identity

The manifest must also name positional encoding, bias policy, attention type, FFN family, classifier/pooler heads, and tensor dtypes when those differ from the BERT-like example below. The vector is a teaching shorthand, not a universal Transformer schema.

The four outputs of the diagram are deliberately separate. A reduction in parameter count need not yield proportional latency. A head-layout change can leave dense projection parameter count nearly unchanged while invalidating a head-wise target. A tied embedding saves stored parameters but does not remove the output projection’s matrix multiplication.

The diagram has three separate kinds of arrows. Layer-map arrows select teacher stages. Width-projection arrows transform coordinates only for the loss. Teacher-assistant arrows represent separate training runs and model artifacts. Treating all three as “alignment” hides very different runtime and checkpoint costs.

0x05B Exact BERT-Like Parameter Derivation

To make “student size” auditable, start with one declared graph rather than a parameter total copied from a model card. Consider a BERT-like encoder with:

token, learned position, and token-type embeddings
optional embedding projection d_e -> d
L post-norm blocks
four attention affine maps Q, K, V, O
two affine FFN maps d -> d_ff -> d
two LayerNorms per block
tied token embedding and MLM decoder weight
MLM transform d -> d_e, transform LayerNorm, and decoder bias
optional untied pooler and task head

This contract is intentionally narrower than “Transformer.” Decoder-only models, RMSNorm, gated FFNs, rotary positions, GQA, bias-free linears, and untied language heads require different terms.

The input embedding state is

$$P_{\mathrm{emb}} = Vd_e+Pd_e+Td_e+2d_e +\mathbf{1}[d_e\ne d](d_ed+d).$$

The first three terms are token, position, and token-type tables. The $2d_e$ term is embedding LayerNorm gain and bias. If embedding and residual widths differ, the final term adds a dense projection and its bias.

One encoder block contains:

$$\begin{aligned} P_{\mathrm{attn}} &=4(d^2+d),\\ P_{\mathrm{ffn}} &=(dd_{ff}+d_{ff})+(d_{ff}d+d),\\ P_{\mathrm{norm}} &=2(2d). \end{aligned}$$

Therefore

$$\boxed{ P_{\mathrm{block}} =4d^2+2dd_{ff}+9d+d_{ff} }.$$

The $9d+d_{ff}$ tail is not decorative. It contains four attention biases, the FFN output bias, four LayerNorm vectors, and the FFN intermediate bias. Dropping it can be acceptable for an asymptotic estimate, but not for exact checkpoint reconciliation.

With a tied decoder weight, the MLM head adds

$$P_{\mathrm{MLM}} =dd_e+3d_e+V.$$

The terms are the transform weight, transform bias, transform LayerNorm gain and bias, and decoder bias. The $Vd_e$ decoder matrix is not counted again because it aliases the input token embedding. The complete declared model is

$$P_{\mathrm{total}} =P_{\mathrm{emb}} +L P_{\mathrm{block}} +P_{\mathrm{MLM}} +P_{\mathrm{pooler}} +P_{\mathrm{task}}.$$

For a standard dense pooler, $P_{\mathrm{pooler}}=d^2+d$. A task head depends on label support and must be named rather than hidden in “about 110M.”

Worked checkpoint-scale reconciliation

The following rows use $V=30{,}522$, $P=512$, and the explicit assumptions above. They are article-local reconstructions; paper totals remain source-reported because implementations can differ in heads, aliases, and included task state.

Graph	Declared dimensions	Reconstructed unique parameters	Source-reported scale
BERT-base-like teacher	$L=12,d=d_e=768,d_{ff}=3072,T=2$, tied MLM, pooler	$110{,}104{,}890$	about $109$–$110$M depending on counting surface
DistilBERT-like student	$L=6,d=d_e=768,d_{ff}=3072$, no token-type table, no pooler, tied MLM	$66{,}985{,}530$	about $66$–$67$M
TinyBERT$_4$-like student	$L=4,d=d_e=312,d_{ff}=1200,T=2$, tied MLM, pooler	$14{,}479{,}050$	$14.5$M in the paper

For the BERT-base-like row:

$$\begin{aligned} P_{\mathrm{emb}} &= 23{,}837{,}184,\\ 12P_{\mathrm{block}} &= 12(7{,}087{,}872) =85{,}054{,}464,\\ P_{\mathrm{MLM}} &= 622{,}650,\\ P_{\mathrm{pooler}} &= 590{,}592. \end{aligned}$$

For the TinyBERT-like row, shrinking both $d$ and $d_{ff}$ changes the quadratic and bilinear block terms, while the vocabulary table remains substantial:

$$4P_{\mathrm{block}} =4(1{,}142{,}184) =4{,}568{,}736.$$

This is why “four layers instead of twelve” does not imply exactly one-third of the total model: embeddings, heads, and other non-block state do not scale with $L$.

An executable counter should enumerate unique storage rather than blindly sum named references:

def count_unique_parameters(named_parameters):
    seen_storage = set()
    total = 0
    rows = []

    for name, tensor in named_parameters:
        storage_id = tensor.storage_identity()
        aliased = storage_id in seen_storage
        if not aliased:
            seen_storage.add(storage_id)
            total += tensor.numel()
        rows.append({
            "name": name,
            "shape": tuple(tensor.shape),
            "numel": tensor.numel(),
            "storage_id": storage_id,
            "counted": not aliased,
        })

    return total, rows

A tied embedding and decoder should appear as two semantic names sharing one storage identity. A serialized checkpoint may store aliases differently, so logical unique parameters, serialized bytes, and loaded resident bytes remain three ledgers.

0x05C Architecture-Level Forward Work And Deployment Shape

Use two scalar FLOPs per multiply-add and ignore softmax, normalization, activation, bias, and data movement for the first logical matmul ledger. For a batch $B$ and sequence length $S$:

• Q, K, V, and output projections each perform $2BSd^2$ FLOPs, giving $8BSd^2$.
• $QK^\top$ performs $2BS^2d$ FLOPs across all heads.
• attention-probability times $V$ performs another $2BS^2d$.
• FFN up and down projections each perform $2BSdd_{ff}$.

Thus one dense BERT-like block performs approximately

$$\boxed{ F_{\mathrm{block,fwd}} \approx 8BSd^2+4BS^2d+4BSdd_{ff} }.$$

The derivation explains why architecture axes behave differently:

$$\begin{array}{rcl} L\downarrow &:& \text{scales all block work and sequential block count},\\ d\downarrow &:& \text{shrinks }d^2,\ S^2d,\ \text{and }dd_{ff}\text{ terms},\\ d_{ff}\downarrow &:& \text{shrinks only the FFN bilinear term},\\ H_q\text{ changes} &:& \text{changes partition/layout, not dense }d^2 \text{ projection count at fixed }d. \end{array}$$

For training, backward and recomputation add convention-dependent multiples; M4.1 owns the full training-compute convention. For serving, the formula is still not a latency prediction. Fewer layers shorten the sequential critical path. Narrow widths can produce less favorable kernel tiles. Different head counts affect launch geometry and KV organization. MobileBERT’s paper, for example, separately changes architecture and operations because equal FLOP counts do not imply equal device latency.

The student release must therefore carry both logical and observed evidence:

logical graph:
    exact unique parameters
    exact tensor shapes and aliases
    prefill/decode FLOP convention
    expected persistent bytes

observed deployment:
    target runtime and hardware
    sequence and batch distribution
    latency quantiles and throughput
    peak resident memory
    numerical and quality gates

Neither ledger can substitute for the other.

0x05D Initialization Is A State Transition, Not A Distillation Target

When student and teacher blocks have compatible tensor shapes, selected teacher parameters can initialize the student. DistilBERT keeps BERT’s hidden width, removes the token-type embeddings and pooler, halves the number of layers, and initializes the student by taking one teacher layer out of two. Its training then uses a triple objective: masked-language modeling, output distillation, and cosine hidden-state alignment.

Initialization answers where optimization starts. A distillation loss answers which behavior remains constrained during training. Copying teacher layers and then training only on hard labels is not the same procedure as continuing to compare teacher outputs. Conversely, a randomly initialized student can still be distilled.

Write initialization as an explicit state transition:

$$\Theta_S^{(0)} = \mathcal I \left( \mathcal A_S, \Theta_T, \Theta_{\mathrm{compact}}, \mathcal D_{\mathrm{init}}, \omega_{\mathrm{init}} \right).$$

$\mathcal I$ is not necessarily a tensor copy. It may train a compact model, run a general-distillation stage, or sample a random initializer. The initialization manifest $\omega_{\mathrm{init}}$ must identify every source revision, mapping, transform, dtype conversion, random seed, and excluded tensor. The subsequent update still follows a declared objective:

$$\Theta_S^{(u+1)} = \Theta_S^{(u)} -\eta_u \nabla_{\Theta_S} \mathcal L_{\mathrm{transfer}} \left(\Theta_S^{(u)};\Theta_T,\mathcal D_u\right).$$

Common choices are:

Initialization	When it is available	Benefit	Risk
alternating or mapped teacher layers	matching student/teacher block shapes	starts from learned features	constrains width/head architecture; layer semantics may not compress cleanly
copy embeddings/output head	shared tokenizer and equal widths/support	preserves lexical interface	impossible or partial under vocabulary/width changes
initialize from an independently pretrained small model	suitable checkpoint exists	student already fits its capacity	teacher/student representations may use different coordinate systems
random initialization	always	maximum architecture freedom	more optimization and data burden
staged general then task-specific student	broad corpus and task teacher available	separates general/domain transfer	extra stages and artifact lineage

Shape-compatible layer copying: DistilBERT

The DistilBERT paper couples three choices that should not be credited to one another. The student removes the token-type embeddings and pooler and halves depth while retaining the teacher’s hidden dimensions. Shape compatibility permits initialization by taking one teacher layer out of two. Training then uses masked-language-model, soft-output, and cosine hidden-direction losses. The paper’s random- initialization ablation is evidence for this specific training contract; it does not prove that every KD student should copy teacher layers.

For a copy map $g_{\mathrm{init}}$, the complete compatibility predicate is stronger than equal element count:

$$\operatorname{copyable}(S_m,T_{g(m)}) = \bigwedge_{p\in\mathcal P_m} \left[ \operatorname{name}_S(p)=\operatorname{name}_T(p) \land \operatorname{shape}_S(p)=\operatorname{shape}_T(p) \land \operatorname{semantic}_S(p)=\operatorname{semantic}_T(p) \right].$$

The semantic term covers Q/K/V ordering, norm placement, activation family, position mechanism, and alias behavior. A transposed or sliced tensor can have the desired shape and still implement a different function.

0x05E Compact Pretraining, General Distillation, And Task Transfer

Pretraining a compact graph and distilling a compact graph are different operations even when they use the same unlabeled text.

Source figure: Turc et al., Well-Read Students Learn Better: On the Importance of Pre-training Compact Models, recipe figure from the revision-pinned source archive. It establishes the paper’s stage order—compact-model pretraining, distillation on an unlabeled transfer set, and optional fine-tuning. It does not prove the paper’s source-reported gains for another architecture or dataset.

The paper studies a grid of compact Transformer students and argues that pretraining the student in its own architecture can materially improve later distillation. The state transition is

$$\Theta_S^{\mathrm{pre}} = \operatorname{Pretrain} \left(\mathcal A_S,\mathcal D_{\mathrm{LM}}\right), \qquad \Theta_S^* = \operatorname{Distill} \left(\Theta_S^{\mathrm{pre}},\Theta_T,\mathcal D_{\mathrm{transfer}}\right).$$

The first stage has no teacher in its loss. It establishes a compact-model prior. The second stage uses teacher behavior. Calling both stages “pretraining” erases the teacher boundary; calling both “distillation” erases the independently learned compact checkpoint.

TinyBERT exemplifies the last choice. It first performs general distillation from an unfine-tuned BERT on general-domain data, then task-specific distillation from a fine-tuned teacher on augmented task data. Its results cannot be attributed to hidden MSE alone; target types, data augmentation, and the two-stage schedule are part of the method.

TinyBERT’s paper contract is:

$$\Theta_S^{\mathrm{general}} = \operatorname{KD} \left( \Theta_{T,\mathrm{pre}}, \mathcal D_{\mathrm{general}}, \mathcal L_{\mathrm{emb}}+ \mathcal L_{\mathrm{hid}}+ \mathcal L_{\mathrm{attn}} \right),$$

followed by

$$\Theta_S^{\mathrm{task}} = \operatorname{KD} \left( \Theta_S^{\mathrm{general}}, \Theta_{T,\mathrm{task}}, \mathcal D_{\mathrm{task,aug}}, \mathcal L_{\mathrm{intermediate}}+ \mathcal L_{\mathrm{prediction}} \right).$$

The general stage in the paper omits prediction-layer distillation. The task-specific stage uses a fine-tuned teacher and augmented task data. For TinyBERT$_4$, the source reports $L_S=4,d_S=312,d_{ff,S}=1200,H_S=12$ and mapping $g(m)=3m$ from a 12-layer, 768-wide BERT-base teacher. The paper also instantiates a six-layer, 768-wide student for controlled comparison. These are two different architecture contracts under one method name.

Initialization path	Teacher used to create initial state?	Architecture must match teacher?	Extra artifact
DistilBERT mapped copy	yes, by tensor copy	copied tensors must match exactly	copy/provenance manifest
compact LM pretraining	no	no	compact pretrained checkpoint
TinyBERT general distillation	yes, by a full training stage	target-specific compatibility only	general-distilled checkpoint
random initialization	no	no	RNG/initializer record

Initialization loader and provenance trace

A safe initialization path checks keys and shapes instead of relying on layer names alone:

for s_layer, t_layer in initialization_map.items():
    s_block = student.blocks[s_layer]
    t_block = teacher.blocks[t_layer]

    for name, s_parameter in s_block.named_parameters():
        t_parameter = t_block.get_parameter(name)
        assert s_parameter.shape == t_parameter.shape
        copy_(s_parameter, t_parameter)

record_provenance({
    "teacher_revision": teacher_revision,
    "initialization_map": initialization_map,
    "copied_parameter_names": copied_names,
})

If hidden width, FFN width, attention-head layout, positional encoding, or normalization placement differs, a partial copy needs an explicit transform. Slicing a larger matrix to fit is an initialization algorithm with its own quality assumptions, not a neutral loading convenience.

0x05F MobileBERT Co-Designs Teacher And Student Interfaces

DistilBERT exploits shape compatibility that already exists. MobileBERT takes the opposite route: it modifies both teacher and student so that layer-wise interfaces agree while their internal capacities differ.

Source figure: Sun et al., MobileBERT, Figure 1 from the revision-pinned source archive. The figure shows BERT, IB-BERT, and MobileBERT block topology, including feature-map and attention transfer. It is paper-local architecture evidence, not a claim that these blocks are optimal for modern decoder-only runtimes.

The paper’s teacher, IB-BERT, preserves a large intra-block representation while exposing a 512-wide inter-block feature map. The MobileBERT student also exposes a 512-wide inter-block feature map but compresses its intra-block path to 128. That common interface allows direct layer-wise feature comparison:

$$H^{T,\ell},H^{S,\ell}\in\mathbb R^{B\times S\times512},$$

even though the internal block coordinates and parameter counts differ. This is architecture co-design for transfer, not a learned hidden projection added after the student has been chosen.

The revision-pinned paper and official configuration describe the principal MobileBERT student as:

blocks L                         24
inter-block residual width u    512
intra-block bottleneck width b  128
attention heads                 4
FFN intermediate width f        512
stacked FFNs per block m        4
token embedding width d_e       128
trigram embedding input         enabled
normalization                   NoNorm in the released optimized config
activation                      ReLU
source-reported total           25.3M parameters

This graph cannot be reconstructed from the tuple $(L,d,d_{ff},H)$ used for ordinary BERT. It has two residual coordinate widths and multiple FFNs per block.

Source-to-code weight trace

In the pinned official modeling.py, the released student configuration follows this path:

prev_output [B,S,u]
    -> input bottleneck [B,S,b]                    weight u*b
    -> shared Q/K bottleneck from prev_output      weight u*b
    -> Q and K projections b -> b                  weight 2*b^2
    -> V projection u -> b                         weight u*b
    -> attention output projection b -> b          weight b^2
    -> m stacked FFNs b -> f -> b                  weight 2*m*b*f
    -> output bottleneck b -> u                    weight b*u
    -> residual with prev_output [B,S,u]

Under that exact path, and excluding biases, normalization parameters, embeddings, heads, and task state, one block has

$$\boxed{ P_{\mathrm{MobileBERT,core}} =4ub+3b^2+2mbf }.$$

The $4ub$ terms are input bottleneck, shared Q/K bottleneck, value projection, and output bottleneck. The $3b^2$ terms are Q, K, and attention output projections. The final term contains all $m$ FFN up/down pairs.

For $u=512,b=128,f=512,m=4$:

$$\begin{aligned} P_{\mathrm{core}} &=4(512)(128)+3(128)^2+2(4)(128)(512)\\ &=262{,}144+49{,}152+524{,}288\\ &=835{,}584 \end{aligned}$$

weight elements per block, or

$$24P_{\mathrm{core}}=20{,}054{,}016$$

before the excluded state. This is a code-contract cross-check, not a reconstruction of the paper’s full source-reported $25.3$M total. Exact reconciliation must add trigram embedding projection, embeddings, biases, NoNorm parameters, pooler/head state, and aliases from the same pinned implementation.

The official code also rejects $b\bmod H_q\ne0$ before attention construction. That assertion belongs in the architecture manifest:

$$b\equiv0\pmod{H_q}, \qquad d_h=b/H_q.$$

Using $u/H_q$ would describe a different head geometry.

0x05G MobileBERT AKT, JKT, And PKT Are Different Schedules

The MobileBERT paper holds the broad teacher/student architecture fixed and compares three ways to schedule its layer-wise feature and attention targets.

Source figure: Sun et al., MobileBERT, Figure 2 from the revision-pinned source archive. Dashed arrows denote knowledge transfer, dotted arrows denote copied state, and lighter blocks denote frozen state in that stage. Reported accuracy and speed results remain source-reported.

The paper defines feature-map and attention targets at layer $\ell$:

$$\mathcal L_{\mathrm{FMT}}^\ell = \frac{1}{SN} \sum_{t=1}^{S}\sum_{n=1}^{N} \left(H_{t,\ell,n}^{T}-H_{t,\ell,n}^{S}\right)^2,$$ $$\mathcal L_{\mathrm{AT}}^\ell = \frac{1}{SH} \sum_{t=1}^{S}\sum_{h=1}^{H} D_{\mathrm{KL}} \left( a_{t,\ell,h}^{T} \middle\| a_{t,\ell,h}^{S} \right),$$

and pretraining distillation

$$\mathcal L_{\mathrm{PD}} = \alpha\mathcal L_{\mathrm{MLM}} +(1-\alpha)\mathcal L_{\mathrm{KD}} +\mathcal L_{\mathrm{NSP}}.$$

Loop 4 owns the target-object and reduction caveats. Here the new object is the schedule:

Strategy	Layer-wise transfer	Pretraining distillation	Frozen/copied state
AKT	all layer losses act as auxiliary terms	optimized in the same combined objective	no separate layer-transfer stage
JKT	all layer losses optimized jointly first	a later PD stage	embedding and classifier copied for the staged paper path
PKT	one layer-transfer stage per depth	a later PD stage	lower layers frozen, or softened to a smaller learning rate, while the current layer is trained

These are not algebraic rearrangements of one sum. They produce different optimizer trajectories and checkpoint boundaries. In particular:

$$\operatorname{Optimize} \left(\sum_\ell\mathcal L_\ell+\mathcal L_{\mathrm{PD}}\right) \ne \operatorname{Optimize}(\mathcal L_{\mathrm{PD}}) \circ \operatorname{Optimize}\left(\sum_\ell\mathcal L_\ell\right)$$

for a non-convex network and finite optimizer schedule.

The pinned official code computes $\text{steps\_per\_phase}=\lfloor U/L\rfloor$. Its helper constructs piecewise values $[0,1,0.1]$ around each layer’s phase. The loss call uses the helper in binary mode, thresholding those values at $0.5$, so the layer-wise loss gate is effectively active only in its selected phase:

$$g_\ell(u) = \mathbf 1 \left[ \ell\left\lfloor U/L\right\rfloor \le u < (\ell+1)\left\lfloor U/L\right\rfloor \right].$$

Do not infer the complete parameter-learning-rate schedule from that loss gate. The training script separately invokes an optimizer with layer-wise warmup enabled, while the paper states that lower layers may be frozen or updated with a smaller learning rate. A faithful implementation must pin both the loss gate and optimizer multiplier for every parameter group.

The paper source-reports a 500k-step teacher/pretraining schedule and an additional 240k progressive-transfer steps over 24 layers. Those numbers are paper evidence, not a recommended modern default.

0x05H Alignment State Can Be Larger Than It Looks

Return to the running decoder example with teacher width $d_T=4096$, student width $d_S=3072$, and four mapped hidden layers. One independent dense projection per mapped layer contains

$$P_{align} =4\times3072\times4096 =50{,}331{,}648$$

parameters. At FP16, projection weights alone occupy $96$ MiB. A common mixed-precision Adam layout with FP16 weights/gradients and FP32 master, first-moment, and second-moment state can approach 16 bytes per trainable parameter, or $768$ MiB for these adapters. Exact optimizer layouts differ, so this is a planning ledger rather than a universal framework constant.

The state must be classified correctly:

State	Trainable?	Checkpointed for resume?	Exported with student?
student weights	yes	yes	yes
student optimizer state	optimizer	yes during training	no
frozen teacher weights	no	by reference/revision	no
hidden projections $W_m$	yes	yes	normally no
learned head/layer map	if used	yes	normally no
fixed layer/token masks	no	configuration/provenance	no

Sharing one projection across mapped layers lowers parameter state but assumes all student layers use a common coordinate transform into all matched teacher layers. Low-rank projections lower training state but constrain the alignment map. Relation transfer removes raw width projections but can create quadratic-in-sequence temporary tensors. The correct choice follows the memory and geometry budget, not the desire to avoid one extra module.

Let

$$P_S=\text{trainable student parameters}, \qquad P_A=\text{trainable alignment parameters}.$$

For one declared mixed-precision optimizer layout, define:

$$\begin{array}{ll} b_w &: \text{bytes per forward weight},\\ b_g &: \text{bytes per stored gradient},\\ b_m &: \text{bytes per master weight, zero if absent},\\ b_1,b_2 &: \text{bytes per first and second optimizer moment}. \end{array}$$

Then the persistent trainable state is

$$M_{\mathrm{trainable}} =(P_S+P_A)(b_w+b_g+b_m+b_1+b_2).$$

This formula is preferable to saying “Adam costs 16 bytes per parameter” because it exposes the actual assumptions. FP16 forward weights and gradients, an FP32 master copy, and two FP32 moments give

$$b_w+b_g+b_m+b_1+b_2 =2+2+4+4+4 =16$$

bytes per trainable parameter. BF16 without a master copy, fused state, 8-bit moments, sharding, offload, and gradient accumulation change this ledger. A frozen teacher adds

$$M_T=P_Tb_T+M_{T,\mathrm{buffers}},$$

but no teacher gradient or optimizer terms when its graph is correctly detached.

The visual separates three questions:

ownership:
    who creates, updates, and checkpoints the state?

liveness:
    between which execution events must the state remain resident?

export:
    which state is needed to execute the student after training?

Conflating them produces common errors: counting frozen teacher optimizer state, omitting teacher target activations, or exporting training-only alignment modules.

0x05I Activation Liveness, Streaming, And Peak

With $B=8$, $S=2048$, four selected teacher layers, $d_T=4096$, and two-byte elements, the selected hidden-state payload is

$$4\times8\times2048\times4096\times2 =512\ \text{MiB}.$$

The four student source tensors at $d_S=3072$ add $384$ MiB. Materializing all four projected student tensors at $d_T=4096$ adds another $512$ MiB. The matched hidden path can therefore expose about $1.375$ GiB of payload before the rest of either model, backward activations, allocator overhead, or loss temporaries are counted.

This does not mean every implementation pays the full sum simultaneously. The ledger shows what must be scheduled:

retain all teacher layers
    simple code, high activation lifetime

stream one teacher target and loss at a time
    lower lifetime, more model/hook coordination

recompute selected teacher layers
    lower storage, higher teacher compute

cache hidden targets offline
    lower repeated teacher compute, large versioned artifacts

replace raw states with final-layer relations
    fewer layers, but [S,S] temporary state

Teacher no_grad() removes its autograd graph, not the need to hold target activations. Student hidden states remain connected to backward. Projection outputs also remain live unless the loss is formed and released immediately. Peak memory must be measured around the actual objective schedule rather than estimated from teacher parameters alone.

For each live object $o$, record byte size $m_o$, production event $\ell_o$, and final-consumer event $r_o$. Peak memory is

$$\boxed{ M_{\mathrm{peak}} = \max_t \left[ M_{\mathrm{persistent}} + \sum_o \mathbf 1[\ell_o\le t<r_o]m_o \right] }.$$

The $1.375$ GiB calculation is therefore an exposed-payload upper ledger for one eager materialization schedule, not an automatic peak. A streamed schedule can shorten teacher and projected-student lifetimes:

teacher_handles = install_selected_layer_hooks(teacher, layer_map.values())
student_handles = install_selected_layer_hooks(student, layer_map.keys())

with no_grad():
    teacher.forward(batch)

student_loss = student.forward_hard_objective(batch)
for s_layer in ordered_student_layers:
    t_layer = layer_map[s_layer]

    teacher_target = teacher_handles.pop(t_layer)       # detached [B,S,dT]
    student_state = student_handles.pop(s_layer)        # autograd [B,S,dS]
    projected = align[s_layer](student_state)           # autograd [B,S,dT]

    student_loss = student_loss + hidden_loss(projected, teacher_target)
    release(teacher_target)
    release(projected_when_graph_allows())

student_loss.backward()

This pseudocode is still a contract sketch. Some frameworks retain the projected tensor for backward; releasing the Python reference does not free storage that autograd saved. Hooks may also keep teacher outputs alive until the whole forward returns. A real implementation must inspect allocator snapshots and saved-tensor state around each event.

Checkpointing and caching trade different resources:

Policy	Teacher forward compute	Device target lifetime	External artifact	Invalidation surface
retain all selected layers	once per batch	long	none	runtime code
stream loss by selected layer	once per batch	shorter if hooks permit	none	schedule/hook code
recompute selected teacher regions	more than once	short	none	teacher graph and RNG
cache teacher targets offline	once per artifact build	loaded slice only	potentially very large	teacher, tokenizer, data, target schema
use last-layer relations	once per batch	fewer layers, possible $S^2$ state	none or relation cache	relation contract

Loop 8 owns cached-artifact schemas. Loop 9 owns distributed placement and whether streaming actually overlaps teacher and student execution.

The deployment ledger remains separate. From 0x00, the toy teacher’s 32 blocks at $(d,d_{ff})=(4096,11008)$ contain about $6.48$ billion simplified gated projection parameters, while 16 student blocks at $(3072,8192)$ contain about $1.81$ billion. Alignment projections can add hundreds of MiB to training yet still add zero bytes to a student-only export. Conversely, if the student architecture remains 32 layers at the teacher widths, deleting the alignment state after KD does not make the deployed graph smaller.

0x05J Capacity Gap Is Not One Scalar

It is tempting to assume that the largest, most accurate teacher is always the best source for a fixed student. Teacher Assistant Knowledge Distillation provides evidence against that universal rule: in the paper’s CNN/ResNet experiments, student performance can degrade as the capacity gap grows even while teacher accuracy improves. The proposed repair is a staged path:

large teacher T
    -> distill intermediate-capacity assistant A
        -> distill deployment student S

The assistant is a complete trained model, not a hidden-width projection. This adds at least one training stage, checkpoint, evaluation gate, and set of teacher targets. A chain of assistants adds more. The paper’s evidence is source-reported for its evaluated image-classification architectures; it supports capacity gap as a design risk, not a theorem that every language model needs an assistant.

Represent the gap as a vector rather than a parameter ratio:

$$\Delta(T,S) = \left( \Delta_L, \Delta_d, \Delta_{ff}, \Delta_H, \Delta_V, \Delta_{\mathrm{objective}}, \Delta_{\mathrm{data}}, \Delta_{\mathrm{calibration}} \right).$$

Two teacher/student pairs with the same parameter ratio can differ in every component. A width gap threatens coordinate targets. A tokenizer gap threatens token KL. An objective/data gap can make a same-shape teacher provide poor supervision. “Capacity gap” is therefore a diagnosis category, not a sufficient intervention rule.

Several mechanisms can contribute to a gap:

student cannot represent the teacher function closely
teacher outputs are too sharp to expose useful alternatives
raw hidden coordinates are poorly aligned
target set does not cover the student's failure contexts
objective weights demand mutually incompatible matches

A teacher assistant directly changes the function being imitated at the final stage. Temperature changes output smoothness. Projections change coordinate compatibility. On-policy data changes visited prefixes. These are alternative interventions on different contracts; they should be diagnosed separately.

MiniLM reports that a teacher assistant helped its smaller Transformer students. That does not collapse MiniLM and teacher-assistant KD into one method: the relation objective defines what is transferred, while the assistant defines which model supplies that target at a stage.

0x05K Teacher-Assistant Paths Add Models, Runs, And Artifacts

Source figure: Mirzadeh et al., Improved Knowledge Distillation via Teacher Assistant, Figure 1 from the revision-pinned source archive. The experiments are primarily CNN/ResNet classification experiments. The figure motivates a staged teacher-assistant path; it does not establish a universal language-model ordering theorem.

Write a $K$-stage path as

$$\mathcal M_0=T \longrightarrow \mathcal M_1 \longrightarrow\cdots\longrightarrow \mathcal M_K=S.$$

Its training cost is not the cost of the final stage:

$$C_{\mathrm{chain}} = \sum_{i=1}^{K} C_{\mathrm{train}} \left( \mathcal M_{i-1}\rightarrow\mathcal M_i \right).$$

It also creates a lineage:

$$\mathcal R_{\mathrm{chain}} = \left\{ \text{stage config}_i, \text{teacher revision}_{i-1}, \text{student checkpoint}_i, \text{evaluation gate}_i, \text{supervision artifact}_i \right\}_{i=1}^{K}.$$

With $n$ candidate assistants ordered between teacher and student, choosing any ordered subset yields $2^n$ possible paths, including the direct path. That is a search-space count, not evidence that every subset should be trained. The paper’s appendix proposes principles and a dynamic-programming view under empirical assumptions; the authors explicitly distinguish those principles from theorems. This article therefore does not promote its path heuristic to a general law.

An equal-budget comparison must charge the chain for everything it adds:

Ledger	Direct $T\to S$	$T\to A\to S$
training runs	1	2
trained deployment candidates before final choice	1	assistant + student
teacher forward programs	$T$ during one stage	$T$ during stage 1, $A$ during stage 2
resumable checkpoints	one stage lineage	two stage lineages
evaluation/promotion gates	one	at least two
final serving graph	$S$	still $S$

The assistant earns its place only if final-student quality or optimization reliability improves enough to justify those added resources. A smaller teacher, a smoother temperature, relation targets, better data, or on-policy prefixes may address a different diagnosed component at lower cost.

0x05L Architecture-To-Training Implementation Contract

Before model construction, freeze a machine-readable manifest:

student_spec = {
    "architecture_revision": "...",
    "layers": L,
    "vocab_revision": "...",
    "vocab_size": V,
    "position_contract": {"type": "learned", "size": P},
    "token_type_size": T,
    "embedding_width": d_e,
    "residual_width": d,
    "ffn_width": d_ff,
    "query_heads": H_q,
    "kv_heads": H_kv,
    "norm": {"type": "layer_norm", "placement": "post"},
    "activation": "gelu",
    "bias_policy": "bert_dense_bias",
    "weight_aliases": [["emb.token.weight", "mlm.decoder.weight"]],
    "pooler": True,
}

Then resolve the initialization plan against actual tensors:

def resolve_initialization(student, sources, plan):
    resolved = []
    for rule in plan:
        dst = student.tensor(rule.dst_name)
        src = sources[rule.source_id].tensor(rule.src_name)

        assert tuple(src.shape) == tuple(rule.expected_source_shape)
        assert tuple(dst.shape) == tuple(rule.expected_destination_shape)
        assert semantic_role(src) == rule.source_role
        assert semantic_role(dst) == rule.destination_role

        value = apply_declared_transform(src, rule.transform)
        assert tuple(value.shape) == tuple(dst.shape)
        copy_(dst, cast(value, rule.destination_dtype))

        resolved.append({
            "dst": rule.dst_name,
            "src": [rule.source_id, rule.src_name],
            "transform": rule.transform,
            "source_sha256": tensor_sha256(src),
            "destination_sha256": tensor_sha256(dst),
        })
    return resolved

A rule with transform="slice_first_d" is not equivalent to transform="identity". It must be evaluated and versioned as an initialization algorithm. Unmatched student tensors require an explicit initializer; unmatched teacher tensors are expected when depth is reduced.

At runtime, build four disjoint parameter sets:

$$\Theta_{\mathrm{frozen\ teacher}}, \quad \Theta_{\mathrm{student}}, \quad \Theta_{\mathrm{alignment}}, \quad \Theta_{\mathrm{nontrainable\ metadata}}.$$

Only student and declared alignment tensors enter the optimizer. An assertion should reject both missing trainable state and accidental teacher gradients:

optimizer_ids = ids(optimizer.parameters)
assert optimizer_ids == ids(student.trainable) | ids(alignment.trainable)
assert not (optimizer_ids & ids(teacher.parameters))
assert all(not p.requires_grad for p in teacher.parameters())

0x05M Export And Resume Invariants

Training should fail early when architecture and target contracts disagree:

assert set(student_to_teacher) == set(requested_student_layers)
assert max(student_to_teacher) < student.num_layers
assert max(student_to_teacher.values()) < teacher.num_layers

for s_layer, t_layer in student_to_teacher.items():
    assert alignment[s_layer].in_features == student.hidden_size
    assert alignment[s_layer].out_features == teacher.hidden_size

assert relation_head_contract(teacher, student, head_map)
assert tokenizer_contract(teacher_tokenizer, student_tokenizer, token_loss)

A resumable checkpoint needs:

student parameters
student optimizer and scheduler
alignment projection/head-map parameters
alignment optimizer state
teacher revision and tokenizer revision
layer map and loss weights
normalization, mask, and reduction contracts
data/cache position and RNG state

A deployment export should prove the opposite boundary:

artifact = export_student_only(student)
loaded = load_student_without_teacher_or_alignment_modules(artifact)

assert loaded.parameter_count == expected_student_parameters
assert not contains_prefix(artifact.state_dict, "teacher.")
assert not contains_prefix(artifact.state_dict, "alignment.")
assert student_only_quality_gate(loaded)
assert student_only_runtime_gate(loaded)

The first list preserves the training objective after interruption. The second proves that training-only state has not become an undeclared serving dependency.

Resume equivalence requires more than loading tensors. Let the training state at update $u$ be

$$\mathcal C_u = \left( \Theta_S, \Theta_A, \Omega, \Sigma, u, \rho, \delta, \chi \right),$$

where $\Omega$ is optimizer state, $\Sigma$ is scheduler state, $\rho$ is RNG state, $\delta$ is data/artifact cursor, and $\chi$ contains the teacher, tokenizer, architecture, initialization, target, mask, and reduction manifests. A resume test should compare the next update from uninterrupted and restored state:

$$\operatorname{Step}(\mathcal C_u,\mathcal B_{u+1}) \stackrel{?}{=} \operatorname{Step} \left( \operatorname{Load}(\operatorname{Save}(\mathcal C_u)), \mathcal B_{u+1} \right)$$

within the declared deterministic or numerical tolerance. Merely loading the student without error does not test resume.

The deployment artifact has the inverse inclusion rule:

$$\mathcal E_S = \left( \mathcal A_S, \Theta_S^*, \text{tokenizer}, \text{runtime ABI}, \text{release evidence} \right),$$

and excludes $\Theta_T,\Theta_A,\Omega,\Sigma$, hooks, and training data cursors. If an alignment projection is needed during serving, it was not training-only and must be incorporated into $\mathcal A_S$, recounted, and benchmarked.

0x05N Paper-By-Contract Comparison

The papers below do not answer one interchangeable question:

Method	Student architecture	Starting state	Main transfer interface	Stage structure	Serving consequence
DistilBERT	half-depth, same principal hidden dimensions; token-type embeddings and pooler removed	every-other teacher-layer copy	MLM + output KD + cosine hidden alignment	pretraining distillation	student only
Well-Read Students	grid of compact Transformer depths/widths	independently pretrained compact model	black-box teacher output transfer in the paper setup	compact pretrain, distill, optional fine-tune	student only
TinyBERT	four-layer narrow and six-layer same-width variants	general-distilled compact checkpoint	embedding, hidden, attention-score, and prediction targets	general KD then task KD with augmentation	student only
MobileBERT	deep/thin bottleneck student with co-designed IB-BERT teacher	copied interfaces plus staged transfer under the paper recipe	common inter-block features, attention maps, MLM KD	AKT, JKT, or PKT plus PD	student only; altered operations affect runtime
MiniLM	smaller student, paper variants include teacher-assistant use	task-specific paper recipe	last-layer QK and VV relations	relation KD, optionally from an assistant	student only
Teacher Assistant KD	ordered capacity chain in CNN/ResNet evidence	each stage starts a complete student model	stage-local output KD	$T\to A_1\to\cdots\to S$	final student only; extra training artifacts

This table prevents several invalid comparisons:

DistilBERT versus TinyBERT:
    not only one hidden loss versus another;
    architecture, initialization, data, targets, and stages differ.

TinyBERT versus MobileBERT:
    not only shallow versus deep;
    MobileBERT co-designs teacher interfaces and block topology.

direct KD versus teacher-assistant KD:
    not one loss versus another;
    the latter adds complete models and training runs.

same parameter count:
    not an equal deployment contract;
    depth, width, operations, and target runtime still differ.

Any source-reported benchmark belongs beside its exact architecture, data, teacher, schedule, hardware, and measurement surface. Loop 5 does not reproduce those training runs.

0x05O Failure Closure

Failure	Observable symptom	Broken contract	Required repair
approximate total copied from paper but graph differs	checkpoint count mismatch	architecture/accounting	enumerate named unique tensors and aliases
tied embedding counted twice	logical parameters exceed unique storage	alias contract	group semantic names by storage identity
layer copy checks shape only	model loads but behavior degrades immediately	initialization semantics	validate role, ordering, norm, position, and transform
silent slice or transpose during load	non-reproducible starting state	initialization provenance	name and hash the transform
student graph unchanged	no structural serving saving	deployment graph	choose and benchmark an actual smaller/different graph
frozen teacher enters optimizer	unexpected gradients/state growth	ownership	assert optimizer parameter-set equality
teacher uses no-grad but targets are all retained	out-of-memory despite frozen teacher	liveness	stream, recompute, or cache under a measured schedule
all created activations are summed as peak	pessimistic and unauditable capacity estimate	liveness interval	compute maximum overlap by event
projected targets exported accidentally	teacher-free artifact fails to load or grows	export boundary	strip training-only modules and reload in a clean process
correction made by temperature only	hidden/token/data mismatch persists	capacity diagnosis	identify which gap component failed
assistant credited without charging first stage	apparent quality gain under unequal budget	path accounting	include all runs, artifacts, and gates
MobileBERT loss gate treated as optimizer schedule	wrong parameters update in each phase	schedule contract	pin loss gate and parameter LR separately
paper parameter/latency number generalized	runtime prediction fails	source-report scope	reproduce on declared graph, runtime, and hardware

The deterministic Loop 5 oracle checks architecture arithmetic, initialization maps, aliasing, state ownership, liveness intervals, teacher-assistant path cost, and student-only export. It does not train any cited model, execute a framework autograd graph, reproduce a benchmark, or validate a production runtime.

0x05P Loop Boundary

Intermediate transfer and student shape now have explicit contracts:

depth        -> layer selection g(m)
width        -> projection, compatible coordinates, or relation target
heads        -> equal layout, aggregation, map, or relation-head design
tokens       -> shared positions, masks, and output support
initialization -> starting state, separate from continuing supervision
capacity gap -> direct teacher or staged teacher-assistant path
runtime      -> teacher activations + student graph + training-only alignment
export       -> student alone

Loop 5 has established the complete graph-and-state contract:

$$\boxed{ \mathcal A_S +\mathcal I +\mathcal C_{\mathrm{train}} +\mathcal L_{\mathrm{live}} +\mathcal R_{\mathrm{path}} \longrightarrow \mathcal E_S }$$

where the architecture, initialization, resumable training state, liveness schedule, and optional multi-stage path are all versioned separately. This is the precondition for comparing supervision artifacts fairly.

The next loop will change a different axis: the supervision artifact and the prefix distribution. Sequence-level KD replaces a full local teacher distribution with teacher-generated trajectories, while on-policy methods ask the teacher to score contexts generated by the student. Neither can be derived by simply adding another hidden-state loss.

0x06 Sequence-Level And Black-Box Distillation

Owner question. What exactly can be transferred when the teacher returns generated text rather than weights, hidden states, or a complete vocabulary distribution, and what state must exist before that text becomes a valid student-training row?

Loop 6 changes the supervision object. It consumes the student graph and state contracts closed in 0x05, but it does not change the student architecture. It owns complete-sequence objectives, black-box access levels, generated labels, rationales, explanations, prompts, proxy distributions, and the semantic identity of one generated artifact. It deliberately defers four adjacent problems:

0x07  student-generated token prefixes, MiniLLM, GKD, and divergence direction
0x08  physical corpus layout, sharding, compression, filtering jobs, and revisions
0x09  distributed teacher placement, online hooks, collectives, and scheduling
0x11  final quality gates, deployment promotion, release, and rollback

The sources in this loop establish different contracts rather than one linear history:

Source	Contract used here	Status and ceiling
Sequence-Level KD	exact sequence objective, mode, K-best, Monte Carlo, and reference-near interpolation	Externally verified paper contract; NMT results remain source-reported
Distilling Step-by-Step	label and rationale as distinct task-conditioned targets	Externally verified paper and commit-pinned code contract
Orca	system-conditioned explanation traces and progressive teachers	Externally verified paper contract; data sizes and outcomes are source-reported
Self-Instruct	instruction, input, and output generation plus similarity filtering	Externally verified paper and commit-pinned code; adjacent synthetic-data method, not automatically KD
Lion	imitation, student-aware discrimination, and hard-instruction generation	Externally verified paper and commit-pinned code; paper-specific referee policy
Proxy-KD	black-box hard sequences plus an aligned white-box proxy distribution	Externally verified paper contract; proxy logits are never promoted to teacher logits

0x06A Black-Box Is An Access Contract, Not A Model Size

A model is black-box with respect to a particular caller. A hosted API may return only text, text plus a sequence score, chosen-token log probabilities, a top-$k$ list, or a full logit tensor. Those interfaces expose different random variables and support different losses.

Returned object	Smallest faithful artifact	Objectives it can directly support	It cannot establish
one text response	raw response bytes plus generation identity	hard sequence NLL, label/rationale/explanation generation	alternatives, teacher entropy, full KL, hidden transfer
$K$ responses plus calibrated sequence log probabilities	candidate rows and their exact score semantics	finite-support sequence expectation	probability mass outside the retained set
chosen-token or top-$k$ log probabilities	sparse row events plus residual-mass policy	path scoring or sparse-logit approximation	unseen token identities and their individual probabilities
full ordered-vocabulary logits	[valid_rows, V_T] tensor plus support map	word-level cross-entropy, KL, entropy, temperature	hidden-state or relation targets
hidden/attention tensors	layer-named tensors plus projections and maps	0x04 intermediate transfer	black-box portability

The access levels are ordered by information, but the arrow is not an inverse. Given one returned string $\hat y$, there are infinitely many teacher distributions for which $\hat y$ is the selected output. Neither the tail mass nor its allocation over alternative sequences is identifiable from that string. Calling hard-label instruction tuning “black-box KD” describes the lineage of the target; it does not make the hard target algebraically equivalent to Hinton-style full-distribution KD.

This distinction also prevents a common Proxy-KD overclaim. Proxy-KD obtains text from a black-box teacher and logits from a separately trained white-box proxy. The proxy logits may be useful supervisory state. They are still samples from $\pi_P$, not recovered calls to an unavailable $\pi_T$ logit interface.

0x06B The Exact Sequence Objective Is An Expectation Over Stopping Paths

For prompt or source sequence $x$ and output $y=(y_1,\ldots,y_L)$, include EOS in $y_L$. The autoregressive sequence probability is

$$P(y\mid x) =\prod_{t=1}^{L}p(y_t\mid x,y_{<t}).$$

Variable-length generation means that $\mathcal Y(x)$ contains both token choices and stopping paths. Omitting EOS changes the sample space and removes the teacher’s length decision from the objective.

If $P_T$ and $P_S$ denote teacher and student sequence distributions, their sequence cross-entropy is

$$\mathcal L_{seq}^{full}(x) =-\sum_{y\in\mathcal Y(x)}P_T(y\mid x)\log P_S(y\mid x) =\mathbb E_{y\sim P_T(\cdot\mid x)}[-\log P_S(y\mid x)].$$

Substitute the student factorization:

$$-\log P_S(y\mid x) =-\sum_{t=1}^{|y|}\log p_S(y_t\mid x,y_{<t}),$$

so

$$\mathcal L_{seq}^{full}(x) =\mathbb E_{y\sim P_T} \left[-\sum_{t=1}^{|y|}\log p_S(y_t\mid x,y_{<t})\right].$$

Regroup the expectation by visited prefix $c_t=(x,y_{<t})$. Let $d_T(c)$ be the teacher-induced occupancy measure, including the probability that the trajectory survives to that row. Then

$$\mathcal L_{seq}^{full}(x) =\sum_c d_T(c) H\!\left(p_T(\cdot\mid c),p_S(\cdot\mid c)\right).$$

This identity separates two independent choices:

prefix occupancy:     which contexts receive weight?       d_T(c)
local target:         what distribution is matched there?  p_T(.|c)

Word-level KD on human references uses reference-prefix occupancy. Sequence distillation uses teacher-generated occupancy. Loop 7 will use current-student occupancy. Forward versus reverse KL changes the local divergence; it does not, by itself, choose the prefix distribution.

The exact sum is exponential. Kim and Rush therefore do not compute it. They define it, then study estimators and support-reducing approximations.

0x06C Sampling, Mode Decoding, And K-Best Lists Approximate Different Objects

Monte Carlo Preserves The Expectation, Not Every Batch

If $y^{(1)},\ldots,y^{(K)}$ are independent samples from the declared teacher distribution, then

$$\widehat{\mathcal L}_{MC} =-\frac{1}{K}\sum_{k=1}^{K}\log P_S(y^{(k)}\mid x)$$

is unbiased:

$$\mathbb E[\widehat{\mathcal L}_{MC}] =\mathcal L_{seq}^{full}, \qquad \operatorname{Var}(\widehat{\mathcal L}_{MC}) =\frac{\operatorname{Var}_{y\sim P_T}[-\log P_S(y\mid x)]}{K}.$$

The condition matters. Samples from a top-$p$, top-$k$, constrained, filtered, or temperature-modified decoder come from $Q_\gamma$, not necessarily $P_T$. Without importance weights and support coverage, they estimate $\mathbb E_{y\sim Q_\gamma}[-\log P_S(y\mid x)]$. In practical black-box distillation that policy-defined distribution is often the intended teacher, but it must be named.

One Decoded Sequence Is A Delta Approximation

Let

$$\hat y_T=\operatorname{Decode}_T(x;\gamma),$$

where $\gamma$ contains the complete search, sampling, stopping, and reranking contract. Replace the sequence distribution by a point mass:

$$\widehat P_T(y\mid x)=\mathbf 1[y=\hat y_T].$$

Then

$$\begin{aligned} \widehat{\mathcal L}_{\delta} &=-\sum_y\mathbf 1[y=\hat y_T]\log P_S(y\mid x)\\ &=-\log P_S(\hat y_T\mid x)\\ &=-\sum_{t=1}^{|\hat y_T|}\log p_S(\hat y_{T,t}\mid x,\hat y_{T,<t}). \end{aligned}$$

The last line is ordinary teacher-forced maximum likelihood on a synthetic target. The unchosen token probabilities have disappeared. Beam search does not repair that information loss; it selects the point at which the delta mass is placed.

Kim and Rush use beam search as an approximate mode finder. Their paper itself reports that a greedy candidate accounted on average for only $1.3\%$ and $2.3\%$ of sequence mass in its two translation directions, and a beam-five list for $2.8\%$ and $3.8\%$. Those paper-local numbers are evidence that the retained path can work empirically in that setup, not evidence that one mode contains the full teacher distribution.

A Renormalized K-Best Loss Has A Missing-Mass Term

For retained set $\mathcal Y_K(x)$, define

$$M_K=\sum_{y\in\mathcal Y_K(x)}P_T(y\mid x)$$

and, only when the stored scores are calibrated log probabilities under one common sequence policy,

$$\tilde w_k =\frac{P_T(y^{(k)}\mid x)}{M_K}, \qquad \widehat{\mathcal L}_K =-\sum_{k=1}^{K}\tilde w_k\log P_S(y^{(k)}\mid x).$$

Split the exact objective into retained and tail support:

$$\mathcal L_{seq}^{full} =M_K\widehat{\mathcal L}_K +(1-M_K)\mathbb E_{y\sim P_T(\cdot\mid y\notin\mathcal Y_K)} [-\log P_S(y\mid x)].$$

This derivation shows what renormalization does: it conditions the teacher on the retained list, drops the tail expectation, and usually drops the factor $M_K$. If the API returns only a beam ranking score, a length-normalized score, or a reranker score, $M_K$ is not even identifiable. Keep these fields separate:

raw token log-probability sum
length-normalized model score
search score after constraints or penalties
reranker score
candidate rank

0x06D Sequence-KD And Sequence Interpolation Select Different Training Rows

The original paper defines multiple programs, not one generic SeqKD switch:

Method	Occupied prefix during student training	Local target	Required teacher return	Artifact meaning
Word-KD	observed $y_{<t}$	full teacher token distribution	logits	local alternatives on reference prefixes
Seq-KD	decoded $\hat y_{T,<t}$	hard decoded token	one selected sequence	approximate teacher-mode trajectory
Seq-Inter	selected beam candidate $\tilde y_{<t}$	hard selected token	beam list plus reference similarity	teacher-supported, reference-near trajectory
Seq-KD + Word-KD	decoded teacher prefix	hard token plus local distribution	sequence plus logits	global selected path and local alternatives

For sequence interpolation, let $\mathcal Y_K(x)$ be the teacher beam and $y^*$ the observed target. The paper’s implementable selection is

$$\tilde y =\arg\max_{y\in\mathcal Y_K(x)}\operatorname{sim}(y,y^*),$$

using smoothed sentence-level BLEU in its experiment. It does not interpolate token IDs. It chooses one complete teacher-supported candidate whose text is closest to the reference. The broader idealized criterion in the paper also includes teacher probability:

$$\tilde y =\arg\max_{y\in\mathcal Y(x)} \operatorname{sim}(y,y^*)P_T(y\mid x).$$

Changing similarity, beam width, tie-breaking, or reference normalization can change the selected target while leaving the student code untouched. Those are artifact-generation parameters.

The article’s primary-source Figure 1 near the historical introduction shows the paper’s Word-KD, Seq-KD, and Seq-Inter paths. The native diagram above adds the later occupancy boundary: a cached teacher path belongs here; a student-generated token prefix rescored during training belongs to 0x07.

0x06E Generation Configuration And Semantic Identity Are Target State

The same source prompt can produce different supervision after any of these changes:

source row or source schema revision
teacher provider, endpoint, returned model revision, or tokenizer if visible
rendered prompt, system instruction, demonstrations, and chat template
decode mode, temperature, top-k, top-p, beam width, and candidate count
length normalization, repetition/frequency penalties, constraints, and reranker
maximum tokens, stop strings, EOS policy, timeout, and seed when meaningful
normalizer, parser, safety policy, quality filter, and deduplication revision
student tokenizer, vocabulary, special-token, packing, and truncation revision

Define canonical semantic identity $I(a)$ for artifact $a$ as a typed map of those fields. Serialize it with sorted keys, stable number encoding, UTF-8, and no volatile timestamps. The artifact key is

$$k(a)=\operatorname{SHA256}(\operatorname{CanonicalJSON}(I(a))).$$

A cache row is reusable only when the expected and stored semantic identities are equal:

$$\operatorname{Reusable}(a,I_{expected}) =\mathbf 1[k(a)=\operatorname{SHA256}(\operatorname{CanonicalJSON}(I_{expected}))].$$

This is narrower than the physical corpus lifecycle owned by 0x08. Loop 6 defines what makes one supervision object the same object. Loop 8 will own how millions of such objects are serialized, partitioned, compacted, migrated, filtered, and versioned as a dataset.

Store provider-returned model identity separately from the requested alias. A moving alias such as latest cannot by itself prove that two calls used the same teacher state. If the provider exposes no immutable revision, record that limitation and treat regeneration as a new empirical corpus rather than exact reproduction.

0x06F Raw Teacher Text And Student Causal Tensors Are Different Artifacts

A text-only teacher and a student do not need the same tokenizer. The interchange object is raw text. Student token IDs are a derived cache tied to a particular student tokenizer and packing policy.

Let

$$p=\operatorname{Tok}_S(\operatorname{Render}_S(x)), \qquad r=\operatorname{Tok}_S(\operatorname{Normalize}(y_{raw})).$$

After applying one explicit truncation policy, append EOS exactly once:

$$z=p\,\Vert\,r\,\Vert\,[eos].$$

The causal tensors are

$$\operatorname{input\_ids}=z_{0:n-1}, \qquad \operatorname{labels}=z_{1:n}.$$

For response-only training, label row $j$ is valid when its shifted token belongs to the response or is its terminal EOS:

$$m_j =\mathbf 1[z_{j+1}\in\{\text{response tokens},eos\}].$$

The first valid row is therefore the final prompt input row, because that row predicts the first response token. Masking according to the input segment instead of the shifted label segment silently drops that first target.

One deterministic toy row makes the shift auditable:

prompt IDs P:      [101, 102]
response IDs R:    [201, 202]
EOS:               [2]

packed z:          [101, 102, 201, 202, 2]
input_ids:         [101, 102, 201, 202]
labels:            [102, 201, 202, 2]
response mask:     [  0,   1,   1, 1]

Truncation is part of the target definition. A safe policy reserves terminal space before cutting content:

def pack_sequence_row(prompt_ids, response_ids, *, max_length, eos_id):
    prompt_ids = remove_terminal_eos(prompt_ids, eos_id)
    response_ids = remove_terminal_eos(response_ids, eos_id)

    # Policy choice: retain the full prompt if possible, then response prefix,
    # and always reserve one terminal EOS position.
    room_for_response = max_length - len(prompt_ids) - 1
    if room_for_response < 1:
        raise ArtifactError("prompt_leaves_no_supervised_response_row")
    response_ids = response_ids[:room_for_response]

    z = prompt_ids + response_ids + [eos_id]
    input_ids = z[:-1]
    labels = z[1:]
    first_response_label = len(prompt_ids) - 1
    loss_mask = [int(j >= first_response_label) for j in range(len(labels))]

    assert labels[-1] == eos_id
    assert sum(token == eos_id for token in labels) == 1
    assert labels[first_response_label] == response_ids[0]
    return input_ids, labels, loss_mask

Alternative policies may left-truncate a conversation, retain the answer tail, or reject overlong rows. They are not interchangeable. The artifact must record which semantic content was removed, not just the final tensor length.

0x06G Rationales Can Be Targets Without Becoming Inference Inputs

Text-only access can return more than an answer. Distilling Step-by-Step elicits a teacher label $\hat y_i$ and rationale $\hat r_i$, then trains a smaller task-specific model with two task-conditioned outputs.

The label objective is

$$\mathcal L_{label} =\frac{1}{N}\sum_{i=1}^{N} \ell\!\left(f([\texttt{predict}:x_i]),\hat y_i\right),$$

and the rationale objective is

$$\mathcal L_{rationale} =\frac{1}{N}\sum_{i=1}^{N} \ell\!\left(f([\texttt{explain}:x_i]),\hat r_i\right).$$

The paper writes the combined objective as

$$\mathcal L_{paper} =\mathcal L_{label}+\lambda\mathcal L_{rationale}.$$

The pinned official implementation at commit ef944263c9ddfd40a9888557f3e7d17513fd1a7e constructs separate prediction and explanation batches in model_utils.py:26-39, executes two forwards in model_utils.py:49-53, and uses

$$\mathcal L_{code} =\alpha\mathcal L_{label}+(1-\alpha)\mathcal L_{rationale}.$$

For $\alpha>0$,

$$\mathcal L_{code} =\alpha\left( \mathcal L_{label} +\frac{1-\alpha}{\alpha}\mathcal L_{rationale} \right),$$

so the relative coefficient corresponds to $\lambda=(1-\alpha)/\alpha$, but the whole loss is also scaled by $\alpha$. That outer scale preserves an unconstrained minimizer but changes gradient magnitude under a fixed learning-rate, clipping, weight-decay, or scheduler contract. Paper notation and executable weighting must both be recorded.

Source figure: Hsieh et al., Distilling Step-by-Step, arXiv:2305.02301v2, framework figure from the revision-pinned source archive. It establishes label-plus-rationale multi-task supervision. The source-reported data efficiency and model results are not reproduced here.

This design differs from supplying a rationale as an inference-time input:

input-rationale design:
    teacher generates rationale at deployment
    student predicts label from [x, rationale]
    teacher dependency remains

multi-task target design:
    rationale is auxiliary training output
    deployment can call [predict: x] without teacher or rationale
    student release remains teacher-independent

A generated rationale is evidence about teacher behavior, not automatically a faithful causal explanation. Parsing success, label-rationale consistency, factuality, and downstream usefulness require separate evaluation. Do not let a fluent rationale overwrite a correct label without an explicit conflict policy.

0x06H Explanation Tuning Changes Prompt State And Teacher Schedule

Orca uses richer explanation traces for general instruction following rather than the task-specific two-output interface above. One training row includes a system instruction $s$, user instruction $u$, optional task input $x$, and teacher response $y$:

$$a=(s,u,x,y,teacher\_id,template\_id).$$

The system instruction is not decorative metadata. It changes the desired response style and often the amount or form of explanation. Removing it while keeping the response creates a different conditional task.

Source figure: Mukherjee et al., Orca, arXiv:2306.02707v1, explanation-tuning example from the revision-pinned source archive. It demonstrates that system instructions are part of the input contract and that explanation form varies with that contract. It is not evidence that generated reasoning is faithful or portable to another teacher.

Orca also uses a progressive teacher schedule. Abstractly, with a cheaper assistant corpus $D_A$ and a stronger-teacher corpus $D_T$:

$$\theta_1 =\operatorname{Train}(\theta_0,D_A), \qquad \theta_2 =\operatorname{Train}(\theta_1,D_T).$$

This order is training state. The paper reports five million ChatGPT-augmented rows followed by one million GPT-4-augmented rows. Those counts and its benchmark outcomes are source-reported; they are not a universal ratio. A fair cost ledger charges both generation phases and both student phases. Swapping the stages, mixing them from step zero, or using one teacher are alternative methods.

Orca’s assistant is also distinct from 0x05K’s classical teacher assistant. Here the intermediate teacher chiefly changes generated data, cost, and curriculum. It does not necessarily provide a differentiable bridge between a white-box teacher distribution and student distribution.

0x06I Generating Prompts Changes The Training Distribution, Not Only Labels

Response-only distillation fixes prompts $x\sim D_X$ and asks a teacher for $y$. Synthetic instruction pipelines can also generate $x$. Their artifact is therefore a joint object:

$$a=(instruction,input,response,provenance),$$

and their training distribution is a generated $\widehat D_{X,Y}$, not merely teacher labels on the original $D_X$.

Self-Instruct Bootstraps A Task Pool

Self-Instruct starts from a seed task pool $\mathcal T_0$. At round $r$, the generator samples demonstrations from the current pool, proposes instructions, classifies task type, creates instances, filters them, and adds accepted tasks:

$$\mathcal T_{r+1} =\mathcal T_r\cup F_{\phi_r}\!\left( G_{\gamma_r}(\operatorname{Sample}(\mathcal T_r)) \right).$$

$G_{\gamma_r}$ owns the prompt and decode policy. $F_{\phi_r}$ owns parser, length, modality, consistency, and similarity rules. The paper’s diversity gate accepts a new instruction $u$ only if

$$\max_{v\in\mathcal T_r} \operatorname{ROUGE\mbox{-}L}(u,v)<0.7.$$

The pinned implementation at commit 0b26ccaa415992100fa32df62d41b994cf928e23 loads prior generated rows and resumes request_idx in bootstrap_instructions.py:140-149, samples two machine and remaining human demonstrations at lines 163-171, declares the generation policy at lines 173-187, and enforces the $0.7$ threshold at lines 195-213. These are implementation-local choices, not mathematical constants of synthetic data generation.

Source figure: Wang et al., Self-Instruct, arXiv:2212.10560v2, Figure 2 rendered from the revision-pinned source archive. Self-Instruct is included as a generated instruction-data contract. It is self-training in its original form and becomes black-box distillation only when a distinct stronger teacher supplies the supervision being transferred.

The generated task pool needs two non-collapsible identities:

lineage identity:
    which seed and previously accepted tasks influenced this generation?

training identity:
    which final instruction/input/output row trained the student?

Deduplicating only final responses misses duplicated instructions. Deduplicating only instructions misses conflicting outputs for the same input. Loop 8 will generalize those filters across the full corpus.

Lion Uses Student Failure To Generate New Instructions

Lion gives a proprietary model three paper-specific roles: teacher $T$, referee $R$, and generator $G$. For cache prompt $x_i$, let $y_i^T$ and $y_i^S$ be teacher and student responses. The official implementation evaluates both presentation orders and averages the aligned scores. Define

$$\bar R_T(i) =\frac{R(T,S)_T+R(S,T)_T}{2}, \qquad \bar R_S(i) =\frac{R(T,S)_S+R(S,T)_S}{2},$$

then

$$\Delta_i=\bar R_T(i)-\bar R_S(i), \qquad x_i\in\mathcal H\iff\Delta_i\ge\tau.$$

The pinned code at commit 0d94c4c7e2e5e545dc68253f6e242712b9e4edc0 computes the two-order averages in src/discrimination.py:19-30 and uses the paper implementation’s $\tau=1$ split at lines 47-51. The repository’s documented iteration is:

1. imitation
   query teacher on Train Pool
   train student on teacher responses

2. discrimination
   query teacher and student on Cache Pool
   referee both response orders
   split hard/easy prompts by score gap

3. generation
   generate new prompts from hard and easy seeds
   replace/add them to the next pool

Source figure: Jiang et al., Lion, arXiv:2305.12870v2, method overview from the revision-pinned source archive. The teacher, referee, and generator are logical API roles even when one provider model serves all three. The threshold, judge reliability, data mix, and reported outcomes remain paper-specific.

Lion is student-aware, but its placement here is precise: student responses alter the future prompt pool. It does not, in the paper contract used here, ask the teacher for a full token distribution on every student-generated token prefix. The latter is the on-policy occupancy problem owned by 0x07.

0x06J Proxy-KD Adds A Surrogate Distribution And Another Training Program

When a black-box teacher returns only text, one alternative is to train a white-box proxy $\pi_P$ that imitates the teacher and then expose the proxy’s logits to the student. Proxy-KD makes that two-stage path explicit.

Source figure: Chen et al., Knowledge Distillation of Black-Box Large Language Models, arXiv:2401.07013v2, Figure 2 rendered from the revision-pinned source archive. It establishes the paper’s proxy-alignment and student-distillation stages. It does not establish that the proxy recovers the unavailable black-box teacher distribution.

Stage 1: Align The Proxy

For teacher-generated sequence $(x,y^T)$, the proxy hard-label objective is

$$\mathcal L_{P,NLL} =\mathbb E_{(x,y^T)\sim D_P}[-\log\pi_P(y^T\mid x)].$$

The paper also forms a preference pair from teacher response $y^T$ and proxy response $y^P$. Under a standard DPO minimization convention with reference $\pi_{ref}$:

$$\mathcal L_{P,pref} =-\mathbb E\log\sigma\!\left( \beta\left[ \log\frac{\pi_P(y^T\mid x)}{\pi_{ref}(y^T\mid x)} -\log\frac{\pi_P(y^P\mid x)}{\pi_{ref}(y^P\mid x)} \right] \right).$$

Thus

$$\mathcal L_P =\mathcal L_{P,NLL}+\lambda_{pref}\mathcal L_{P,pref}.$$

Teacher-over-proxy preference is a paper assumption and data-construction rule, not a human preference label. If the teacher fails on a prompt, the pair can align the proxy toward that failure.

Stage 2: Train The Student From Two Different Sources

The hard sequence still comes from the black-box teacher:

$$\mathcal L_{S,NLL} =\mathbb E_{(x,y^T)\sim D_S}[-\log\pi_S(y^T\mid x)].$$

The soft row distribution comes from the proxy:

$$\mathcal L_{S,KL} =\mathbb E_{c\sim d_{D_S}} D_{KL}\!\left(\pi_P(\cdot\mid c)\,\|\,\pi_S(\cdot\mid c)\right).$$

The paper downweights examples on which the proxy assigns low sequence likelihood to the teacher response. Let

$$s(x,y^T)=\log\pi_P(y^T\mid x), \qquad \mu=\mathbb E[s], \qquad \gamma^2=\operatorname{Var}(s).$$

Then

$$w(x,y^T) =\sigma\!\left(\frac{s(x,y^T)-\mu}{\gamma}\right)$$

and

$$\mathcal L_S =\mathcal L_{S,NLL} +\alpha\mathbb E\left[ w(x,y^T)D_{KL}(\pi_P\|\pi_S) \right].$$

An executable implementation must define four details that the compact formula can hide:

1. whether $s$ is a sum or length-normalized mean over response tokens;
2. the valid-row mask and tokenizer/support alignment for proxy and student;
3. the numerical fallback when $\gamma=0$ or the calibration shard is tiny;
4. whether $\mu$ and $\gamma$ are frozen corpus statistics or moving estimates.

The paper’s 10/45/45 warm-up/proxy/student split is source-reported design state, not a universal prescription. Proxy-KD also incurs proxy generation, proxy alignment, proxy forward, and proxy storage costs. It is not “free soft labels” for a text-only API.

0x06K Every API Call Belongs To An Idempotent Transaction And Cost Ledger

A generation worker can time out after the provider completed a request but before the client committed the row. Blind retry can create two outputs with the same semantic identity. Exactly-once effects therefore require an idempotency key and atomic commit, even when the provider offers only at-least-once request execution.

One semantic record can retain the states needed by later loops without fixing their physical file format:

{
  "artifact_key": "sha256:...",
  "sample_id": "prompt-004219",
  "source_revision": "prompts@sha256:...",
  "teacher": {
    "provider": "provider-a",
    "requested_model": "teacher-alias",
    "returned_model": "teacher-revision-if-exposed"
  },
  "prompt": {
    "template_revision": "chat-v3",
    "system_instruction_revision": "helpful-v2",
    "rendered_sha256": "..."
  },
  "decode": {
    "kind": "sample",
    "temperature": 0.7,
    "top_p": 0.95,
    "max_new_tokens": 256,
    "stop_revision": "stop-v2"
  },
  "attempts": [
    {"attempt_id": 1, "status": "retryable_timeout", "input_tokens": 91},
    {"attempt_id": 2, "status": "complete", "input_tokens": 91, "output_tokens": 37}
  ],
  "raw_teacher_text": "Paris.",
  "normalizer_revision": "unicode-nfc-v1",
  "student_tokenizer_revision": "student-tokenizer@sha256:...",
  "student_target_ids": [8712, 13, 2],
  "response_mask_revision": "response-eos-v1",
  "filter": {"revision": "quality-v4", "decision": "accept"}
}

The monetary generation cost is a sum over attempts, not accepted rows:

$$C_{API} =\sum_{a\in\mathcal A} \left(c_{in,a}n_{in,a}+c_{out,a}n_{out,a}+c_{request,a}\right).$$

The complete artifact-creation ledger is

$$C_{artifact} =C_{API}+C_{student\_decode}+C_{proxy}+C_{referee} +C_{filter}+C_{storage}+C_{egress}.$$

With $N_{accepted}$ accepted rows and $N_{attempt}$ attempted calls,

$$\eta_{yield}=\frac{N_{accepted}}{N_{attempt}}, \qquad C_{effective/row}=\frac{C_{artifact}}{N_{accepted}}.$$

Retries, rejected safety outputs, parse failures, duplicate proposals, and referee calls all cost resources even though they produce no student row.

def generate_once(source_row, semantic_identity, provider):
    key = sha256(canonical_json(semantic_identity))
    prior = artifact_store.get(key)
    if prior and prior.status == "complete":
        return prior

    attempt_id = artifact_store.begin_attempt(key)
    try:
        response = provider.generate(
            rendered_prompt=semantic_identity["rendered_prompt"],
            decode=semantic_identity["decode"],
            idempotency_key=key,
        )
        row = parse_normalize_tokenize(response, semantic_identity)
        validate_semantic_row(row, semantic_identity)
        return artifact_store.commit_once(key, attempt_id, row)
    except RetryableProviderError as error:
        artifact_store.record_retryable(key, attempt_id, error)
        raise
    except ArtifactPolicyError as error:
        artifact_store.reject(key, attempt_id, typed_reason=error.code)
        raise

commit_once must reject a second accepted row for the same key. If sampling intentionally requests multiple outputs, candidate index or deterministic sample identity belongs in $I(a)$; otherwise deduplication will incorrectly collapse intended candidates.

0x06L Artifact Creation And Student Training Are Separate Executables

An offline path should permit student training with no teacher credentials, teacher weights, proxy weights, or provider client installed:

@no_grad()
def artifact_producer(source_rows, teacher_client, contract):
    for source in source_rows:
        identity = build_semantic_identity(source, contract)
        generate_once(source, identity, teacher_client)


def artifact_validator(row, expected_contract):
    assert row.artifact_key == hash_identity(expected_contract, row.sample_id)
    assert row.raw_teacher_text_sha256 == sha256(row.raw_teacher_text)
    assert row.student_tokenizer_revision == expected_contract.tokenizer_revision
    assert row.labels[-1] == expected_contract.eos_id
    assert count_supervised_eos(row) == 1
    assert first_supervised_label_is_first_response_token(row)
    assert row.filter.decision == "accept"


def student_train_step(student, rows):
    batch = collate_prevalidated_rows(rows)
    logits = student(input_ids=batch.input_ids).logits
    token_loss = causal_cross_entropy(
        logits,
        batch.labels,
        mask=batch.response_mask,
        reduction="global_valid_token_mean",
    )
    token_loss.backward()

There is exactly one causal shift in this pseudocode: it occurred when the artifact row was packed. A framework that expects unshifted labels may move the shift into its loss helper, but the contract must choose one owner. Applying both shifts trains token $t$ against target $t+2$ while shapes still match.

The offline split has concrete benefits and limitations:

Property	Offline generated artifacts	Online teacher call
teacher loaded during student training	no	yes
target reuse across student runs	high when identity matches	low unless responses are cached
student sees fresh teacher feedback after drift	no	possible
provider failure interrupts optimizer step	no	yes
artifact storage and privacy burden	explicit	lower storage, higher runtime coupling
reproducibility	strong only with immutable identity and retained raw outputs	bounded by provider and request replayability

0x06M Paper-By-Contract Comparison

Method	Prompt source	Teacher return	Student target	Extra model or role	Iteration state	Deployment dependency
Seq-KD	fixed observed source inputs	approximate-mode sequence	response-only sequence NLL	beam search	one offline generation pass	student only
Seq-Inter	fixed source plus observed reference	K-best sequence list	one reference-near candidate	similarity metric	one offline beam/rerank pass	student only
Distilling Step-by-Step	fixed labeled or unlabeled task inputs	label and rationale	two task-prefixed outputs	rationale parser	generate, then multi-task train	student only when rationale is target
Orca	sampled instruction mixtures plus system messages	explanation trace	full response text	progressive teacher assistant	assistant corpus then stronger-teacher corpus	student only
Self-Instruct	evolving generated task pool	instruction, input, output	instruction-tuning row	classifier and filters	iterative pool bootstrap	tuned model only
Lion	train/cache pools plus generated prompts	teacher response and referee scores	response NLL on evolving prompt pool	teacher/referee/generator roles	imitation, discrimination, generation	student only
Proxy-KD	fixed teacher-response corpus	black-box sequence	teacher NLL plus proxy KL	aligned white-box proxy	proxy warm-up/alignment then student KD	student only after training

These methods choose along at least six axes:

access:          text / candidates / sparse logprobs / full logits / internals
prompt source:   observed / generated / student-failure-generated
target object:   answer / rationale / explanation / preference / proxy distribution
prefix source:   observed / teacher-decoded / current-student (deferred to 0x07)
execution:       offline / iterative corpus creation / online
release:         student only / student plus required external teacher

Equal model size does not make two methods equal-budget. Compare teacher calls, returned tokens, referee/generator calls, proxy training, student updates, artifact bytes, filter yield, and wall time under the same quality gates.

0x06N Primary-Source And Code Trace

Source	Exact revision or commit	Observed mechanism span	Claim limit
Sequence-Level KD	1606.07947v4	source main.tex:346-478: exact sequence objective, mode approximation, K-best/Monte Carlo alternatives, Seq-Inter	paper definition and source-reported NMT evidence only
Distilling Step-by-Step	2305.02301v2; code ef944263...	paper sections/03_method.tex:1-71; code model_utils.py:26-55, run.py:138-150	paper/code weighting and two-task data path; no training reproduction
Orca	2306.02707v1	explanation-tuning.tex:1-280: system messages, sampling, progressive ChatGPT/GPT-4 data	paper data-generation contract; provider behavior is historical and source-reported
Self-Instruct	2212.10560v2; code 0b26ccaa...	paper main.tex:172-255; code bootstrap_instructions.py:140-213	task-pool and filter mechanics; not universal synthetic-data quality
Lion	2305.12870v2; code 0d94c4c7...	paper 3_method.tex:1-164; code README.md:109-252, discrimination.py:19-51	paper-specific iterative roles and threshold
Proxy-KD	2401.07013v2	acl_latex.tex:248-461: proxy alignment, preference path, weighted proxy KL	proxy is a surrogate; no official implementation was identified in the pinned source

The code repositories were inspected statically at the commits above. No provider API, framework training run, distributed backend, paper checkpoint, or benchmark was executed by this loop.

0x06O Sequence Artifact Failure Closure

Failure	Why shape or type checks pass	Observable consequence	Repair
returned text called a teacher distribution	string is valid	alternatives and entropy are overclaimed	declare delta/finite-support artifact
decode policy unversioned	response parses	regenerated corpus drifts without attributable cause	hash complete generation identity
moving model alias treated as immutable revision	provider accepts alias	exact replay cannot be claimed	retain returned model identity or mark non-reproducible
beam ranking score used as probability	each candidate has a scalar	K-best weights are length/reranker biased	preserve probability and search/reranker scores separately
top-$k$ logprobs treated as full support	sparse rows normalize locally	unseen tail is silently deleted	store residual-mass contract or keep claim sparse
prompt tokens included in response loss	tensors align	objective partly reconstructs the prompt	build mask on shifted label ownership
first response token masked	lengths align	answer starts are never supervised	test the final-prompt-row boundary
EOS omitted or duplicated	visible answer tokens train	stopping behavior is absent or overweighted	reserve and assert one supervised EOS
tokenizer revision changes while IDs are reused	IDs remain integers	cached targets mean different events	preserve raw text and invalidate derived IDs
truncation cuts after EOS insertion	maximum length is valid	EOS or response semantics disappear	reserve EOS before content truncation
rationale concatenated into inference input	model trains normally	teacher/rationale dependency remains at serving	use separate task-conditioned target when teacher-independent release is required
rationale and label disagree	both strings parse	auxiliary loss can train contradictory behavior	retain both, detect conflict, apply versioned policy
Self-Instruct filter changes silently	accepted rows remain valid	prompt distribution changes	version generator, pool lineage, and filter
referee position bias ignored	two scalar scores exist	hard/easy split follows answer order	evaluate both orders or use an audited judge contract
proxy logits called teacher logits	[R,V] shapes match	surrogate error disappears from claims	name $\pi_P$ and measure agreement/calibration
proxy likelihood uses unnormalized sequence sum	one score per row exists	long responses receive systematically different weights	declare sum/mean/length policy
retry commits two accepted rows	both calls succeeded	duplicate supervision and cost undercount	key by semantic identity and commit once
rejected rows disappear from ledger	train corpus is clean	effective cost and policy cannot be audited	retain typed rejection and attempt metadata
credentials or raw teacher cache enter release	student checkpoint loads	license, privacy, and dependency boundary is violated	student-only export allowlist plus lineage references

0x06P Loop Boundary

Loop 6 closes the generated-supervision object:

$$\mathcal S_{source} +\mathcal I_{teacher} +\mathcal G_{decode} +\mathcal O_{raw} +\mathcal T_{student} +\mathcal M_{response} +\mathcal F_{policy} +\mathcal Q_{attempt} \longrightarrow \mathcal A_{sequence}.$$

Here $\mathcal A_{sequence}$ can be an answer, label-rationale pair, explanation trace, generated instruction row, or teacher sequence augmented by a proxy distribution. Its access level, semantic identity, causal tensors, cost, rejection state, and claim ceiling are explicit.

The next loop changes a different axis. It replaces the fixed reference- or teacher-generated prefix distribution with prefixes visited by the changing student, and it must then separate occupancy, divergence direction, sampling, and gradient ownership. None of the offline artifact contracts above implies that on-policy problem is solved.

0x07 Autoregressive Mismatch And On-Policy Distillation

Sequence-KD changes the training corpus from observed targets to teacher targets, but the resulting corpus is still fixed while the student changes. At deployment, the student conditions on its own earlier decisions. A mistake at position $t$ creates a prefix that may never have appeared in either the reference or teacher-generated corpus, yet every later prediction is conditioned on it.

On-policy distillation changes the source of prefixes. It lets the current student generate a trajectory, then asks the teacher how the next-token distribution should look on those student-visited states.

0x07A Three Coordinates Define An On-Policy Distillation Method

“On-policy distillation” is incomplete unless three independent coordinates are named:

occupancy:
    which prefixes receive training weight?

local discrepancy:
    which probability distributions are compared on each selected prefix?

sampling estimator:
    does changing the probability of visiting a prefix contribute a gradient,
    or are sampled token IDs stopped and treated as training data?

The first coordinate is about states. The second is about distributions on one already selected state. The third is about credit assignment through the probability of selecting that state. Conflating them turns materially different algorithms into the same slogan.

Prefix Occupancy Is Independent Of Divergence Direction

Let $d_{data}$, $d_T$, and $d_S^\theta$ denote prefix distributions induced by reference sequences, teacher generation, and the current student. For any token-level divergence $D$, three objectives are possible:

$$\mathcal{L}_{fixed} =\mathbb{E}_{c\sim d_{data}} \left[D\left(p_T(\cdot\mid c),p_S^\theta(\cdot\mid c)\right)\right],$$ $$\mathcal{L}_{teacher-prefix} =\mathbb{E}_{c\sim d_T} \left[D\left(p_T(\cdot\mid c),p_S^\theta(\cdot\mid c)\right)\right],$$

and

$$\mathcal{L}_{on-policy} =\mathbb{E}_{c\sim d_S^\theta} \left[D\left(p_T(\cdot\mid c),p_S^\theta(\cdot\mid c)\right)\right].$$

Changing $D$ does not change which prefixes were selected. Changing the prefix source does not force a particular divergence. GKD is useful partly because it makes these axes independently configurable.

Consider one small trace:

prompt:       The capital of France is
reference:    Paris . <eos>
student draw: Lyon is located ...

fixed-prefix KD rows:
    [prompt]
    [prompt, Paris]
    [prompt, Paris, .]

on-policy KD rows:
    [prompt]
    [prompt, Lyon]
    [prompt, Lyon, is]

Both methods can use forward KL at every row. Only the student-prefix path asks the teacher how to recover after Lyon. The selected student token remains part of the next prefix even when the teacher assigned it low probability. Replacing it with the teacher argmax would move training back toward a teacher trajectory and would no longer expose later student states.

This vocabulary descends from imitation learning. In DAgger, the learner executes a policy, the expert labels learner-visited states, and the next learner is trained on the aggregate. ImitKD adapts that idea to autoregressive models but replaces examples inside each mini-batch instead of growing an unbounded aggregate. GKD later exposes the same output-source choice as a student-data fraction. These methods share a state-distribution concern; they do not share one objective or estimator.

0x07B Sequence KL Factorization Determines Its Own Occupancy Measure

The occupancy distinction is not only an intuition about exposure bias. It falls out of the chain rule for sequence distributions.

For a prompt $x$, let a complete stopping path include its EOS decision:

$$P(y\mid x)=\prod_{t=1}^{|y|}p(y_t\mid c_t), \qquad Q_\theta(y\mid x)=\prod_{t=1}^{|y|}q_\theta(y_t\mid c_t),$$

where $c_t=(x,y_{<t})$. Define the unnormalized prefix occupancy of policy $\pi$ as

$$d_\pi(c\mid x) =\sum_{t\ge1}\Pr_{y\sim\pi(\cdot\mid x)}[c_t=c].$$

Its total mass is expected generated length, not necessarily one. That choice is useful because one occupied prefix corresponds to one next-token loss row. If a paper averages by response length, it is using a normalized variant and must state where that normalization occurs.

Start with sequence-level forward KL:

$$\begin{aligned} D_{KL}(P\parallel Q_\theta) &=\sum_yP(y\mid x)\log\frac{P(y\mid x)}{Q_\theta(y\mid x)}\\ &=\sum_yP(y\mid x)\sum_t \log\frac{p(y_t\mid c_t)}{q_\theta(y_t\mid c_t)}. \end{aligned}$$

Group complete paths by the prefix and next token they visit:

$$D_{KL}(P\parallel Q_\theta) =\sum_c d_P(c\mid x) D_{KL}\!\left(p(\cdot\mid c)\parallel q_\theta(\cdot\mid c)\right).$$

Forward sequence KL therefore carries teacher occupancy. Reversing the arguments reverses both the local support weighting and the trajectory measure:

$$D_{KL}(Q_\theta\parallel P) =\sum_c d_{Q_\theta}(c\mid x) D_{KL}\!\left(q_\theta(\cdot\mid c)\parallel p(\cdot\mid c)\right).$$

This equality is exact when paths and EOS are included consistently. It does not justify evaluating reverse token KL on arbitrary fixed prefixes and calling the result sequence reverse KL: fixed prefixes replace $d_{Q_\theta}$ with a different measure.

The Loop 7 oracle makes the chain rule finite. A two-step tree uses

teacher root:    [A=.70, B=.30]
student root:    [A=.40, B=.60]

teacher after A: [A=.80, B=.20]
student after A: [A=.60, B=.40]

teacher after B: [A=.25, B=.75]
student after B: [A=.30, B=.70]

The four complete paths are AA, AB, BA, and BB. Direct path-space sums and occupancy-weighted local sums reconcile to

$$D_{KL}(P\parallel Q_\theta)=0.2496975320,$$ $$D_{KL}(Q_\theta\parallel P)=0.2377427189.$$

The forward decomposition weights the after A row by $0.70$; the reverse decomposition weights it by $0.40$. The local probability tables did not change. Only the trajectory distribution supplying the expectation changed.

0x07C Forward KL And Reverse KL Weight Different Support

At one already selected prefix $c$, define $p_T(v)=p_T(v\mid c)$ and $p_S(v)=p_S^\theta(v\mid c)$. Forward KL is

$$D_{FKL}(p_T\parallel p_S) =\sum_{v\in\mathcal{V}}p_T(v) \log\frac{p_T(v)}{p_S(v)}.$$

The teacher weights the sum. If $p_T(v)>0$ while $p_S(v)$ approaches zero, the penalty grows without bound. This pressure asks the student to represent every teacher-supported alternative that carries meaningful mass.

Reverse KL changes the weighting distribution:

$$D_{RKL}(p_S\parallel p_T) =\sum_{v\in\mathcal{V}}p_S(v) \log\frac{p_S(v)}{p_T(v)}.$$

Now student-supported tokens dominate the expectation. Probability placed where the teacher has almost none is expensive, while a teacher mode that the student does not visit contributes little directly. For a capacity-limited family, this can favor concentrating on a subset of teacher modes.

For example, let a teacher assign $(0.49,0.49,0.02)$ to three alternatives and a concentrated student assign $(0.98,0.01,0.01)$. The second teacher mode has a large forward-KL contribution

$$0.49\log\frac{0.49}{0.01}\approx1.91,$$

but its reverse-KL contribution is weighted by the student’s $0.01$:

$$0.01\log\frac{0.01}{0.49}\approx-0.039.$$

Individual KL summands can be negative even though the complete divergence is non-negative. The example explains support weighting; it does not prove that a reverse-KL-trained language model is always more truthful or that forward KL always produces bad text. Model capacity, parameterization, prefix source, sampling temperature, optimization estimator, and task all matter. The GKD paper reports that its preferred divergence was task-dependent.

The difference is also visible before any optimizer is chosen. Let the student probabilities be $q=\operatorname{softmax}(z)$. The softmax Jacobian is

$$\frac{\partial q_i}{\partial z_j}=q_i(\mathbf 1[i=j]-q_j).$$

For forward KL, only the cross-entropy term depends on $z$:

$$\begin{aligned} \frac{\partial}{\partial z_j}D_{KL}(p\parallel q) &=-\sum_i p_i\frac{1}{q_i} \frac{\partial q_i}{\partial z_j}\\ &=-p_j+q_j\sum_i p_i\\ &=q_j-p_j. \end{aligned}$$

Reverse KL differentiates both the student weight and its logarithm:

$$\begin{aligned} \frac{\partial}{\partial z_j}D_{KL}(q\parallel p) &=\sum_i q_i(\mathbf 1[i=j]-q_j) \left(\log\frac{q_i}{p_i}+1\right)\\ &=q_j\left( \log\frac{q_j}{p_j}-D_{KL}(q\parallel p) \right). \end{aligned}$$

The constant $1$ cancels because the softmax rows sum to one. These are local logit gradients on a fixed prefix. They explain different support pressure; they do not yet include how changing the student changes which future prefixes are visited.

Source-reported GKD comparison. The figure motivates exposing divergence as a method coordinate; it does not establish one universally optimal direction.

0x07D The Reverse-KL Gradient Contains An Occupancy Derivative

Sequence reverse KL depends on the student twice. The student appears inside the log ratio, and it defines the probability of visiting the complete path:

$$\mathcal L(\theta) =\mathbb E_{y\sim Q_\theta} \left[\log\frac{Q_\theta(y\mid x)}{P(y\mid x)}\right].$$

Write $f_\theta(y)=\log Q_\theta(y\mid x)-\log P(y\mid x)$. Applying the score-function identity before simplifying gives

$$\begin{aligned} \nabla_\theta\mathcal L &=\sum_y\nabla_\theta Q_\theta(y)f_\theta(y) +\sum_yQ_\theta(y)\nabla_\theta f_\theta(y)\\ &=\mathbb E_{Q_\theta} \left[(f_\theta(y)+1)\nabla_\theta\log Q_\theta(y)\right]. \end{aligned}$$

The first term is the occupancy derivative: changing the student changes which paths are sampled. The second is the ordinary derivative of the log student probability on the selected path. A stopped-token local KL update retains the second kind of gradient on each rescored prefix but omits the score-function contribution from selecting that prefix.

Factor the path log probability and define teacher-relative reward

$$r_t=\log\frac{p(y_t\mid c_t)}{q_\theta(y_t\mid c_t)}, \qquad R_t=\sum_{j=t}^{T}r_j.$$

Terms before action $y_t$ are a conditional baseline for $\nabla\log q_\theta(y_t\mid c_t)$, so the exact gradient can be written

$$\nabla_\theta\mathcal L =-\mathbb E_{Q_\theta} \left[ \sum_{t=1}^{T}(R_t-1) \nabla_\theta\log q_\theta(y_t\mid c_t) \right].$$

The $-1$ is not an arbitrary reward bonus. It comes from differentiating $\log Q_\theta$ inside the expectation. The Loop 7 two-step oracle evaluates this expression exactly: its analytic reverse-KL gradient and reward-to-go gradient differ by only $5.55\times10^{-17}$, while central finite differences differ by $9.41\times10^{-11}$. No derivative is taken through a discrete token ID; trajectory probability is handled through the score function.

0x07E MiniLLM Optimizes A Sequence Reverse-KL Objective

MiniLLM owns this full sequence reverse-KL contract. Its target is not merely token reverse KL evaluated on a convenient batch:

$$\mathcal L_{MiniLLM} =D_{KL}(Q_\theta\parallel P) =\mathbb E_{y\sim Q_\theta} \left[\log\frac{Q_\theta(y\mid x)}{P(y\mid x)}\right].$$

Source-reported MiniLLM method figure. It establishes the paper’s intended trajectory source and teacher interaction, not the correctness of every later runtime implementation.

The method begins from a student already fine-tuned for conditional generation. It samples student trajectories, obtains teacher probabilities on those exact trajectories, and optimizes the estimator above. That base estimator exposes three practical problems identified by the paper: high-variance long-horizon credit, exploitation of poorly shaped likelihood-ratio rewards, and a bias toward short outputs. The published recipe therefore includes estimator and sampling machinery, not only a divergence name.

0x07F MiniLLM Splits Single-Step And Long-Term Credit

The MiniLLM decomposition follows by writing $R_t=r_t+R_{t+1}$:

$$\begin{aligned} \nabla\mathcal L =&-\mathbb E_Q\sum_t(r_t-1) \nabla\log q_\theta(y_t\mid c_t)\\ &-\mathbb E_Q\sum_tR_{t+1} \nabla\log q_\theta(y_t\mid c_t). \end{aligned}$$

Condition on $c_t$ in the first line and sum exactly over the current token:

$$-\mathbb E_{c_t\sim d_Q} \nabla_\theta \mathbb E_{a\sim q_\theta(\cdot\mid c_t)} \left[ \log\frac{p(a\mid c_t)}{q_\theta(a\mid c_t)} \right].$$

This is the gradient of local reverse KL on student-occupied prefixes. It can be evaluated with a vocabulary sum and does not require sampling the current action. The second line is the long-term term: action $y_t$ receives credit for how it changes all later teacher/student likelihood ratios. It is the part a stopped-prefix local divergence does not recover.

exact reverse-sequence-KL gradient
    = single-step local reverse-KL gradient
    + future-return occupancy gradient

The equality requires on-policy expectation under the same $Q_\theta$. Turning either term into a clipped, replayed, or behavior-policy estimate changes the estimator and must be recorded. The finite oracle checks the decomposition before any Monte Carlo variance or neural parameterization is introduced.

0x07G Teacher-Mixed Sampling Is A Behavior-Policy Change

MiniLLM also samples from a token-level teacher/student mixture

$$\mu_t(a\mid c_t) =\alpha p(a\mid c_t)+(1-\alpha)q_\theta(a\mid c_t).$$

Once trajectories come from $\mu$ rather than $Q_\theta$, an exact change-of-measure correction through time is a prefix product:

$$w_t =\prod_{j=1}^{t} \frac{q_\theta(y_j\mid c_j)}{\mu_j(y_j\mid c_j)}.$$

For the finite Loop 7 tree, weighting the behavior-policy expectation with this product reconstructs the exact reverse-KL target $0.23774271886135426$. A practical token-local approximation uses

$$\widehat w_t =\frac{q_\theta(y_t\mid c_t)}{\mu_t(y_t\mid c_t)}.$$

That approximation can reduce variance and cost, but it is not the exact prefix importance ratio. The paper and official code must therefore be read at two levels: the source derivation defines the exact correction; the training implementation uses a lower-variance practical estimator.

0x07H Stabilizers Are Part Of The MiniLLM Method

The pinned MiniLLM paper and the official LMOps implementation add the following training state:

Mechanism	Problem it addresses	Contract that must be pinned
remaining-length normalization	long returns and short-output bias	EOS inclusion and denominator at every token
PPO-style ratio clipping	large policy updates from stale rollout probabilities	rollout revision, old log probability, clipping interval
teacher-mixed sampling	low-quality or exploitative student trajectories	$\alpha$, exact or approximate importance ratio
local reverse-KL term	lower-variance immediate correction	vocabulary reduction and coefficient
auxiliary pretraining LM loss	retention of general language behavior	corpus, mixing coefficient, masking

The official code samples with gradients disabled, stores rollout-time student and teacher selected-token log probabilities, and later performs a fresh student forward. Its reward implementation computes remaining-length normalization from the valid response mask. This resolves a notation edge in the paper more concretely than copying an index expression without checking the code.

def minillm_update(rollout, student, teacher_mix_alpha, clip_eps):
    # rollout owns ids, masks, old student log-probs, teacher log-probs,
    # and the student revision that generated them.
    ids = rollout.input_ids
    mask = rollout.response_mask

    new_logits = student(ids, use_cache=False).logits[:, :-1]
    new_log_q = selected_log_probs(new_logits, ids[:, 1:])

    token_reward = rollout.teacher_log_p - rollout.old_log_q
    return_to_go = reverse_cumsum(token_reward, mask)
    normalized_return = divide_by_remaining_valid_length(return_to_go, mask)

    ratio = exp(new_log_q - rollout.old_log_q)
    clipped = clamp(ratio, 1 - clip_eps, 1 + clip_eps)
    policy_loss = -masked_mean(
        minimum(ratio * normalized_return, clipped * normalized_return),
        mask[:, 1:],
    )
    return policy_loss

This is PPO-style clipping, not full PPO. MiniLLM does not introduce a learned value network or inherit PPO’s usual reference-policy KL penalty merely because it uses a clipped ratio. Naming those absent states prevents a familiar algorithm label from silently expanding the implementation.

0x07I GKD Stops Sampling Gradients And Rescores Exact Student Prefixes

Generalized Knowledge Distillation (GKD) chooses a different estimator. It lets a fraction $\lambda$ of outputs come from the current student, stops the sampling path, and minimizes a configurable local divergence on the selected prefixes:

$$\begin{aligned} \mathcal L_{GKD} =&(1-\lambda) \mathbb E_{(x,y)\sim\mathcal D_{fixed}} [\mathcal D(y,x)]\\ &+\lambda \mathbb E_{x\sim\mathcal D_x, y\sim\operatorname{stopgrad}(Q_\theta)} [\mathcal D(y,x)], \end{aligned}$$

where

$$\mathcal D(y,x) =\frac{1}{|y|}\sum_{t=1}^{|y|} D\left(p(\cdot\mid x,y_{<t}),q_\theta(\cdot\mid x,y_{<t})\right).$$

At $\lambda=0$, all output rows are fixed. At $\lambda=1$, all are generated by the current student. The divergence $D$ is independent of that source choice. GKD can therefore use forward KL on student prefixes or reverse KL on fixed prefixes; neither phrase alone identifies the complete method.

Source-reported GKD method comparison. The student-generated IDs are the prefix-selection artifact; gradients are owned by the later student scoring pass, not by the discrete generation operation.

The GKD paper assumes a student already capable of supervised fine-tuning and adequate generation. That is an initialization contract. With an untrained student, fully on-policy rows can concentrate teacher queries on degenerate states; a fixed/on-policy mixture or prior SFT is not merely an optimization convenience.

0x07J Generalized JSD Needs A Pinned Beta Convention

For $0<\beta<1$, define

$$m_\beta=\beta p+(1-\beta)q,$$ $$JSD_\beta(p,q) =\beta D_{KL}(p\parallel m_\beta) +(1-\beta)D_{KL}(q\parallel m_\beta).$$

The mixture should be computed in log space:

$$\log m_\beta =\operatorname{logsumexp} \left(\log\beta+\log p, \log(1-\beta)+\log q\right).$$

There is an endpoint trap. The unscaled interior expression above tends to zero as $\beta\to0$ or $\beta\to1$. The pinned TRL GKD implementation instead defines beta == 0 as forward KL and beta == 1 as reverse KL. Those endpoint values are an explicit implementation extension, consistent with the paper’s intended interpolation after scaling, not the literal endpoint of the unscaled interior formula. A reproduction must pin which convention it uses.

PyTorch adds another source of silent reversal: torch.nn.functional.kl_div expects log-probabilities as input and the comparison distribution as target. A wrapper must name teacher, student, and direction explicitly; a shape-correct call can otherwise optimize the opposite divergence.

0x07K The Runtime Is Generate, Teacher-Score, Student-Rescore

Stopped-sampling GKD has three distinct compute phases:

1. student generate
       sequential decode, no gradient

2. teacher score the exact packed student IDs
       frozen forward, no gradient

3. student rescore the same packed IDs
       differentiable forward and backward

For one fixture:

prompt ids:          [101, 102]
student response:    [12, 13, 2]       # 2 is EOS
packed ids:          [101, 102, 12, 13, 2]
labels:              [-100, -100, 12, 13, 2]
shifted labels:      [-100, 12, 13, 2]
valid shifted mask:  [0, 1, 1, 1]

teacher logits:      [B, S-1, V], requires_grad=False
student logits:      [B, S-1, V], requires_grad=True

The first response token is scored from the final prompt position. Prompt rows are masked after the causal shift. The first EOS remains valid, while padding after EOS does not. Teacher and student must see identical IDs, positions, attention masks, tokenizer mapping, and vocabulary axis.

def gkd_on_policy_step(prompts, student, teacher, divergence, gen_cfg):
    with no_grad():
        response_ids = student.generate(prompts, **gen_cfg)

    packed, labels, attention_mask = pack_causal(prompts, response_ids)
    shifted_labels = labels[:, 1:]
    valid = shifted_labels.ne(-100)

    with no_grad():
        teacher_logits = teacher(packed, attention_mask).logits[:, :-1, :]
    student_logits = student(packed, attention_mask).logits[:, :-1, :]

    assert teacher_logits.requires_grad is False
    assert student_logits.requires_grad is True
    assert teacher_logits.shape == student_logits.shape

    t_rows = teacher_logits[valid]                    # [C,V]
    s_rows = student_logits[valid]                    # [C,V]
    token_loss = divergence(t_rows, s_rows, vocab_axis=-1)  # [C]
    return token_loss.sum() / valid.sum()

A first-order phase ledger is

$$C_{onpolicy} =C_{student\ generate} +C_{teacher\ score} +C_{student\ rescore/backward}.$$

Generation may be latency-bound while both scoring passes are prefill-shaped. Reusing detached generation logits eliminates the training gradient; retaining the entire generation graph implements a different, much larger liveness contract. Peak memory and wall time therefore cannot be inferred from the student-only release artifact.

0x07L ImitKD And f-Distill Change Different Contracts

The on-policy family did not begin with MiniLLM or GKD, and related papers do not make the same approximation:

• DAgger establishes the imitation-learning pattern of executing the learner, asking an expert to label learner-visited states, and aggregating those states, often while mixing expert and learner policies during collection.
• ImitKD transfers that idea to autoregressive sequence models. It schedules a fixed/student-generated sequence mixture and replaces rows inside the current mini-batch rather than growing DAgger’s unbounded aggregate.
• f-Distill derives sequence $f$-divergence decompositions. Its exact chain results for forward KL, reverse KL, and sequence Jensen-Shannon divergence are valuable, but its practical token objectives stop the sampling gradient. The paper’s own notes clarify that a local mixture $\tfrac12p_t+\tfrac12q_t$ is not generally the conditional of the global sequence mixture, and direct backpropagation omits the trajectory-distribution derivative.

That last distinction matters. A mathematically exact sequence decomposition, a local divergence evaluated on sampled prefixes, and an unbiased gradient estimator of the sequence objective are three separate claims. The article does not upgrade one into another without a proof.

0x07M On-Policy State Is Revisioned And Ephemeral

An on-policy row is not identified only by prompt and response IDs. At minimum its runtime record needs:

prompt/example id
generated response ids and first-EOS position
generating student checkpoint or revision
generation temperature, top-p/top-k, maximum length, and RNG seed lineage
teacher revision and tokenizer/vocabulary identity
old student selected-token log probabilities, if an importance ratio is used
teacher selected-token log probabilities or full distributions required by loss
response mask, truncation state, and rejection reason
collection timestamp/step and replay age

The student distribution changes after every update. A rollout from revision $r$ used at revision $r+k$ is replay or off-policy data, even if the prompt is unchanged. Replay can be deliberate, but policy lag and any correction must be observable. Loop 8 owns the complete artifact schema, retention, filtering, and revision transaction; this loop establishes why those fields are necessary.

0x07N Paper-By-Contract Comparison

Method	State/prefix source	Local or sequence target	Sampling-gradient contract	Behavior correction	Training/runtime state
DAgger	learner/expert policy mixture	expert action labels	supervised after collection	collection-policy schedule	aggregated state dataset
ImitKD	scheduled fixed/student mini-batch rows	teacher token distribution	stopped sampled IDs	no sequence importance product	batch replacement schedule
MiniLLM	student or teacher-mixed trajectories	sequence reverse KL	score-function reward-to-go	exact prefix ratio in theory; token-local approximation in practice	old log-probs, returns, clipping, length normalization, LM mix
GKD	fixed/student rows controlled by $\lambda$	configurable local FKL/RKL/JSD	stopped sampled IDs, fresh rescore	none for full sequence objective because it optimizes a stopped-prefix surrogate	generation config, row source, divergence, masks
f-Distill	divergence-specific sampled prefixes	sequence $f$-divergence decomposition plus practical local objectives	practical direct backprop stops sampling gradient	method-dependent	prefix sampler and divergence family

The table is about contracts, not a leaderboard. Source-reported results remain tied to each paper’s teacher, student, data, decoding, and evaluation setup.

0x07O Primary-Source And Code Trace

Evidence	Pinned revision	What was checked	Claim ceiling
MiniLLM paper	2306.08543v6	sequence reverse KL, exact policy gradient, single/long-term split, teacher mixture, importance ratios, clipping, length normalization	paper method and source-reported experiments
MiniLLM official LMOps code	commit 4f2a9deb5f08e459fd44c2e4792344d78ca89fc3	sampler, reward, storage, trainer, practical per-token correction, remaining-length normalization	pinned implementation behavior, not all forks
GKD paper	2306.13649v3	student-data fraction, stopped sampling, divergence alternatives, student initialization	paper method and source-reported experiments
TRL GKD	commit 24db2a825029f7e0e60f6d3c160485565a784503	same-vocabulary guard, beta endpoints, log-space mixture, causal shift, EOS mask, generate/teacher-score/student-rescore	pinned trainer behavior, not a timeless API promise
DAgger	1011.0686v3	learner occupancy, expert labels, aggregation, policy mixing	imitation-learning contract
ImitKD	2009.07253v2	sequence source schedule and mini-batch replacement	paper algorithm
f-Distill	2307.15190v1	sequence decompositions and documented practical-gradient caveat	paper derivation and stated approximation

The executable Loop 7 oracle is an independent finite check. It proves the identities on a fully enumerable tree; it does not reproduce any paper’s neural training run or benchmark.

0x07P On-Policy Failure Closure

Failure	Observable symptom	Violated contract	Repair
local reverse KL is called sequence reverse KL	loss decreases but future-state credit is absent	occupancy derivative	name the stopped-prefix surrogate or use a valid sequence estimator
token-local teacher-mixture ratio is called exact	apparent variance reduction with biased expectation	behavior-policy correction	record approximation; use prefix products for exact finite target
detached generation logits are reused	no student gradient	fresh student rescore	run a differentiable scoring forward
full generation graph is retained unintentionally	memory grows with decode length	stopped sampling contract	generate under no_grad() and rescore
teacher scores its own continuation	tensors may be shape-compatible but rows differ	shared-prefix identity	score exact student-generated IDs
student mistake is overwritten by reference token	recovery rows disappear	learner occupancy	preserve sampled IDs in every scorer
prompt/padding rows enter reduction	deceptively low loss	causal shift and response mask	shift first, mask prompt and post-EOS padding
kl_div arguments are reversed	stable optimization of wrong direction	divergence identity	test against a tiny explicit vocabulary sum
unscaled JSD endpoint is equated with KL	beta endpoint disagrees with implementation	pinned endpoint convention	handle beta==0/1 explicitly
stale rollout is called on-policy	current student differs from collecting policy	revision identity	regenerate or report lag/correction
weak student uses only generated rows	teacher labels mostly degenerate states	initialization/source mixture	SFT first or lower $\lambda$
source figure is treated as benchmark proof	visual similarity becomes outcome claim	evidence ceiling	label figures and results source-reported

0x07Q Loop Boundary

Loop 7 now owns and closes:

prefix occupancy versus local divergence versus sampling estimator
sequence FKL/RKL chain rules and local logit gradients
the reverse-KL occupancy derivative and reward-to-go gradient
MiniLLM's exact objective, decomposition, behavior policy, and stabilizers
GKD's stopped-sampling generate/score/rescore runtime
generalized-JSD endpoint and implementation conventions
DAgger, ImitKD, and f-Distill boundaries
on-policy revision state and failure closure

Loop 8 owns a different problem: the persistent schema, storage format, filtering, revisioning, retention, and invalidation rules for full logits, sparse logits, hidden targets, hard sequences, and rollout records. This loop requires those fields but does not pre-empt their data-system design.

0x08 Supervision Artifact Schema, Storage, Filtering, And Revisioning

The phrase distillation dataset hides at least four different objects. An example distribution says which prompts, documents, labels, conversations, or student states may be queried. A logical target says what the teacher returned. A physical representation says how those targets are indexed and stored. A publication manifest says which immutable bytes a training run is allowed to read. Changing any one of them can change the experiment even when the other three remain byte-identical.

This distinction matters because the previous loops deliberately produced different supervision objects. Loop 3 produced valid causal logit rows. Loop 4 produced hidden and relation tensors. Loop 6 produced generated sequences. Loop 7 produced policy-revision-bound rollouts. Loop 8 does not invent another distillation objective. It defines how those objects survive outside the teacher forward pass without losing their semantics.

Owner question. What must be recorded, stored, validated, versioned, and filtered so that a later reader can prove it trained on the intended teacher target rather than on merely shape-compatible bytes?

0x08A Four Layers Define One Supervision Artifact

A usable artifact has four layers with different failure modes:

Layer	Question it answers	Example fields	Failure if omitted
semantic identity	which mathematical target is this?	source bytes, render, teacher, tokenizer, target estimator, filter graph	stale or incompatible targets appear reusable
logical payload	what values and coordinates were materialized?	logits, IDs, counts, hidden states, masks, offsets, dtype	a reader reconstructs the wrong loss
physical layout	how are logical rows encoded and located?	record batch, row group, tensor sidecar, tar member, index	random access or scans become pathological
publication state	which complete immutable objects are visible?	shard checksums, row ranges, validation state, manifest root	partial output is mistaken for a cache hit

The layers are related but not interchangeable. Apache Arrow specifies typed arrays, buffers, record batches, and IPC framing. Apache Parquet specifies row groups, column chunks, pages, and indexes. Neither format can tell whether a stored sparse row means normalized top-$k$, an unnormalized partial target, or an importance-sampling draw. That meaning belongs to semantic identity and the logical payload schema.

The strict dependency is

$$\text{training run} \longrightarrow \text{manifest root} \longrightarrow \text{immutable shards} \longrightarrow \text{typed rows} \longrightarrow \text{semantic target identity}.$$

Reading those arrows backwards is unsafe. Finding a tensor file does not prove that a manifest admitted it, and finding a manifest does not prove that its rows implement the intended objective unless the semantic contract also validates.

0x08B Semantic Identity Is Canonical And Content-Derived

A folder named teacher-v2/topk64 is a human hint, not an identity. The artifact key must be derived from every field that can change row meaning:

$$I_{\text{artifact}} =H\!\left( I_{\text{source}}, I_{\text{family/split}}, I_{\text{render/pack}}, I_T, I_{\text{tokenizer/vocab}}, I_{\text{target}}, I_{\text{producer}}, I_{\text{filter}} \right).$$

The exact serialization used by $H$ is itself part of the schema. A canonical JSON implementation, for example, must fix key order, string encoding, number representation, and excluded self-hash field. A cache implementation that hashes an unordered language map or the display name of a mutable model alias does not have a reproducible key.

semantic identity
  source:
    corpus + immutable revision + sample ID + exact content hash
  lineage and split:
    parent family ID + immutable split-manifest hash + split name
  rendered input:
    prompt-template hash + packing hash + exact rendered-input hash
  teacher:
    checkpoint-content hash + architecture revision + precision
    eval/train mode + stochastic-layer policy
  tokenizer:
    tokenizer revision + ordered vocabulary hash
  target:
    schema version + value semantics + temperature/layer map/decode rule
    reduction + sparse-selection or sampling estimator
  producer:
    code revision + deterministic/nondeterministic execution contract
  filter:
    filter-graph revision, not only final threshold values

This is stricter than a generic dataset-cache fingerprint. Hugging Face Datasets, for example, provides transform fingerprints and cache behavior in a specific implementation. That is useful lineage evidence, but a generic transform fingerprint does not automatically include teacher mode, ordered vocabulary support, target estimator, hidden-state hook location, or rollout policy age. Loop 8 therefore treats it as an implementation example rather than a complete distillation identity.

Identity invalidation is field-sensitive:

Change	Raw full logits	Top-$k$/sampled target	Hidden target	Teacher sequence	Student rollout
teacher weights	invalidate	invalidate	invalidate	invalidate	rescore or regenerate
vocabulary order	invalidate	invalidate	normally invalidate	retokenization may alter prefixes	regenerate
prompt or packing bytes	invalidate affected rows	invalidate affected rows	invalidate affected rows	regenerate	regenerate
temperature	reusable only if raw logits were stored	invalidate derived values, selection, or proposal	usually unchanged	invalidate if used during decoding	invalidate generation/scoring contract
mapped layer or hook	unchanged	unchanged	invalidate	unchanged	loss-dependent
filter graph	raw rows may remain reusable under a new view	raw rows may remain reusable under a new view	raw rows may remain reusable under a new view	selected subset changes	selected subset changes
student revision	unchanged	unchanged	unchanged	unchanged for teacher sequences	age and policy relation change

“Raw rows may remain reusable” means that a new manifest may refer to the same immutable shards. It never means mutating the old manifest or silently mixing two target identities.

0x08C One Envelope Carries A Typed Payload Union

The common envelope owns provenance and coordinates; the payload is a tagged union. A single untagged field called teacher_values is not sufficient:

Payload tag	Required method-specific fields	Shape sketch	What remains reconstructible
full_logits	raw/log-prob/probability semantics, dtype, ordered vocabulary hash	[N_valid,V]	full local teacher distribution under declared temperature
topk_logits	IDs, values, $k$, tie rule, normalized/unnormalized objective, global normalizer, tail rule	[N_valid,k]	only the declared truncation estimator
sampled_sparse_target	proposal $q$, sample count, replacement, temperature, RNG algorithm/seed/counter, counts or weights	ragged or [N_valid,K]	the declared finite-sample estimator
hidden_state	teacher layers, module path, pre/post-norm hook, width, dtype, loss mask	[N_valid,m,d_T]	mapped feature values at exact hooks
sequence_target	token IDs, offsets, prompt length, decode rule, EOS/truncation, scores	ragged [N,S_i]	selected trajectories under one decode contract
rollout_record	generated IDs, masks, student revision, generation config, RNG state, age policy	ragged [B,S_i]	states visited by one collecting policy

The union prevents a reader from applying temperature twice to cached probabilities, treating raw logits as normalized values, or interpreting sample counts as top-$k$ probabilities. It also makes migrations explicit: a new payload tag or changed field semantics requires a new schema version rather than a permissive reader guess.

The common envelope is still not enough by itself. Two hidden-state payloads with the same [N,m,d] shape are different targets if one records residual stream values before LayerNorm and the other records normalized values after it. Two sparse rows with the same token IDs and values are different if one stores logits and the other stores teacher probabilities. Shape agreement is necessary, not semantic proof.

0x08D Ragged Rows Need Explicit Coordinate Maps

One source example can expand into multiple packed tokens, only some causal rows own targets, and each valid row can own a fixed or ragged payload. These are four coordinate spaces:

$$i_{\text{source}} \xrightarrow{\;o^{\text{source}}\;} i_{\text{packed}} \xrightarrow{\;m^{\text{loss}}\;} i_{\text{valid}} \xrightarrow{\;o^{\text{payload}}\;} i_{\text{value}}.$$

For source offsets $o^{\text{source}}$, payload offsets $o^{\text{payload}}$, and $N_{\text{valid}}$ target rows, the reader must verify

$$o^{\text{source}}_0=0, \qquad o^{\text{source}}_{N_{\text{sample}}}=N_{\text{packed}}, \qquad o^{\text{payload}}_0=0, \qquad \lvert o^{\text{payload}}\rvert=N_{\text{valid}}+1.$$

All offsets must be monotone. The explicit valid-row map must equal the indices selected by the loss mask:

$$r_j =\operatorname{nonzero}(m^{\text{loss}})_j, \qquad j=0,\ldots,N_{\text{valid}}-1.$$

The Loop 8 fixture packs two source samples into nine token positions, selects five target rows, and maps those rows to twenty dense-logit scalars:

source sample offsets:       [0, 5, 9]
packed token positions:      [0, 1, 2, 3, 4, 5, 6, 7, 8]
loss mask:                   [0, 1, 1, 0, 0, 0, 1, 1, 1]
valid -> packed row:          [1, 2, 6, 7, 8]
payload offsets, width V=4:  [0, 4, 8, 12, 16, 20]

The map is more reliable than reconstructing row ownership from sequence lengths at read time. Packing algorithms, prompt masks, EOS ownership, dropped rows, and later schema migrations can all change that reconstruction.

0x08E Dense Logits Define The Reference Storage And Gradient

Let $z\in\mathbb R^V$ be student logits, $p=\operatorname{softmax}(z)$, and let a nonnegative target vector $t$ have total mass

$$m=\sum_{i=1}^{V}t_i.$$

The generalized target cross-entropy is

$$L(t,z)=-\sum_{i=1}^{V}t_i\log p_i.$$

Substitute $\log p_i=z_i-\log\sum_k e^{z_k}$:

$$L(t,z) =-\sum_i t_i z_i +m\log\sum_k e^{z_k}.$$

Differentiating with respect to one student logit gives the identity that all sparse variants must reconcile:

$$\boxed{ \frac{\partial L}{\partial z_j}=m p_j-t_j }.$$

For full KD, $t=p_T$ and $m=1$, so

$$\frac{\partial L_{\text{full}}}{\partial z_j}=p_j-p_{T,j}.$$

This is the reference objective, not necessarily the preferred storage plan. For $N$ valid rows, vocabulary width $V$, and $b$ bytes per value, dense raw storage is

$$M_{\text{full}}=NVb.$$

With $N=10^8$, $V=128{,}000$, and $b=2$:

M_full = 100,000,000 * 128,000 * 2
       = 25,600,000,000,000 bytes
       = 25.6 TB decimal
       ~= 23.28 TiB binary

At 1,024 balanced shards, the raw logit payload is about 25 GB per shard before offsets, masks, metadata, framing, checksums, replicas, and temporary producer files. Offline caching moves teacher compute out of the student step; it does not make teacher output small.

Hidden-state caches can also dominate storage. Four two-byte layers of width 4,096 over the same $10^8$ rows require

$$M_{\text{hidden}} =N m d_T b =10^8\cdot4\cdot4096\cdot2 =3.2768\ \text{TB decimal}.$$

Sequence IDs are much smaller: $10^8$ four-byte IDs are roughly 400 MB before offsets, prompts, scores, and provenance. The smaller object is not equivalent supervision; it is a harder information bottleneck.

0x08F Deterministic Top-k Is Compact But Changes The Target

For a deterministic selected set $K$ of size $k$, define retained teacher mass

$$m_K=\sum_{i\in K}p_{T,i}.$$

There are at least two common top-$k$ objectives, and they are not the same. The unnormalized partial target keeps the original selected probabilities:

$$t_i^{K}=\mathbf 1[i\in K]p_{T,i}, \qquad \sum_i t_i^K=m_K.$$

The generalized gradient derived above becomes

$$\frac{\partial L_K^{\text{unnorm}}}{\partial z_j} =m_Kp_j-\mathbf 1[j\in K]p_{T,j}.$$

The omitted student logits still receive a positive $m_Kp_j$ term, but every student-probability term is scaled by $m_K$. The renormalized target instead defines

$$\widetilde t_i^K =\begin{cases} p_{T,i}/m_K,&i\in K,\\ 0,&i\notin K, \end{cases}$$

and therefore

$$\frac{\partial L_K^{\text{norm}}}{\partial z_j} =p_j-\widetilde t_j^K.$$

This forces every omitted teacher target to zero and increases each retained target by $1/m_K$. Unless $m_K=1$, neither gradient equals full KD.

A ghost or residual bucket stores $1-m_K$ as one extra event. It preserves aggregate tail mass, but it cannot identify how that mass was distributed over omitted tokens. For a two-token tail with mass $r$, both $(r,0)$ and $(0,r)$ produce the same bucket. The student gradient over those individual tokens is therefore underdetermined unless an additional tail model is declared.

The raw payload byte ledger is

$$M_{\text{top-}k} \approx Nk(b_{\text{value}}+b_{\text{index}}) +M_{\text{normalizer/tail}} +M_{\text{row metadata}}.$$

For $N=10^8$, $k=64$, two-byte values, and four-byte IDs:

M_topk_payload = 100,000,000 * 64 * (2 + 4)
               = 38,400,000,000 bytes
               = 38.4 GB decimal
               ~= 35.76 GiB binary

dense / sparse raw-payload ratio ~= 667x

That ratio is a byte result, not a quality result. The cache must also record the value domain, global log-normalizer, tie rule, selection stage, $k$, temperature, retained mass, tail treatment, and exact objective.

0x08G Sampled Sparse Targets Trade Bias For Variance

Sparse Logit Sampling starts from the same incomplete-target gradient and replaces deterministic truncation with a random estimator. Let $q_i>0$ wherever $p_{T,i}>0$, draw $N_s$ token IDs with replacement from $q$, and let $c_i$ be token $i$‘s count. An importance-weighted target estimator is

$$\widehat t_i =\frac{c_i}{N_s}\frac{p_{T,i}}{q_i}.$$

Since $\mathbb E[c_i]=N_sq_i$,

$$\mathbb E[\widehat t_i] =\frac{\mathbb E[c_i]}{N_s}\frac{p_{T,i}}{q_i} =p_{T,i}.$$

When $q=p_T$, the weights cancel and

$$\widehat t_i=\frac{c_i}{N_s}.$$

The sampled gradient is

$$\widehat g_j =p_j-\widehat t_j,$$

so its expectation recovers full KD:

$$\mathbb E[\widehat g_j] =p_j-p_{T,j}.$$

Unbiased does not mean noiseless. Under teacher-proposal multinomial sampling,

$$\operatorname{Var}\!\left(\frac{c_i}{N_s}\right) =\frac{p_{T,i}(1-p_{T,i})}{N_s},$$

and count covariances between different tokens are negative. Larger $N_s$ reduces variance while increasing bytes and reader work. A proposal $q_i\propto p_{T,i}^{\tau_s}$ changes variance and requires the importance weight $p_{T,i}/q_i$; omitting it changes the objective.

The Loop 8 oracle enumerates every multinomial count vector for $p_T=(0.5,0.3,0.15,0.05)$ and $N_s=3$. It recovers $p_T$ and the full gradient exactly in expectation, while retaining variances

$$\left( \frac{0.5(0.5)}{3}, \frac{0.3(0.7)}{3}, \frac{0.15(0.85)}{3}, \frac{0.05(0.95)}{3} \right).$$

The persistent row must therefore record proposal family, proposal temperature, sample count, replacement policy, RNG algorithm, seed and counter, count/weight semantics, and any quantization. Re-running “the same” sampler without that state does not reproduce the same finite artifact.

The paper links an official repository. Static inspection at commit 9d0f3816de14fe96175d0b33931ab46d8f14feb5 finds a README and poster but no executable implementation. The paper derivation is therefore pinned as primary evidence, but Loop 8 does not claim a source-to-running-code reproduction for SLS.

0x08H Physical Layout Follows Access Pattern, Not Method Name

Once target semantics are frozen, physical layout should follow the reader’s access pattern:

Layout family	Natural strength	Relevant physical unit	Main tradeoff
Arrow IPC	typed columns, variable-length arrays, record-batch interchange, memory mapping	array buffers and record batches	very large dense tensors often benefit from separate tensor payloads
Parquet	column pruning, compression, predicate filtering, analytical scans	row group, column chunk, page	token-row random access requires another index/translation layer
fixed/ragged tensor sidecar	direct dense or indexed tensor reads	contiguous tensor extent plus offset table	metadata and tensor files must be checksum-bound together
tar/WebDataset-style stream	sequential streaming and colocated row members	archive member and stream shard	predicate pushdown and arbitrary random access are weaker

No row format is universally best. An offline top-$k$ trainer may scan Parquet metadata while fetching token IDs and values from aligned tensor sidecars. A large sequential sequence-KD job may prefer stream shards. A diagnostic tool may need Arrow record batches for zero-copy metadata inspection. The manifest must bind all files, schemas, indexes, offsets, and checksums that together implement one logical shard.

A useful cost model separates bytes selected from bytes read:

$$M_{\text{logical selected}} =\sum_{r\in\mathcal R}M_r,$$ $$M_{\text{physical read}} =M_{\text{pages/chunks touched}} +M_{\text{indexes}} +M_{\text{decompression buffers}}.$$

Predicate pruning can reduce the first or second term depending on format and layout, but it does not alter target semantics. Conversely, changing a sparse estimator can alter target semantics even if physical bytes read remain the same.

0x08I Shards Become Visible Through Manifest-Gated Atomic Publication

Artifact production is a state machine:

planned -> staged -> validated -> immutable shard committed
                                      |
                                      v
                     manifest committed after every required shard

For each staged shard, validation must close at least shape, mask, finiteness, row range, schema, byte count, and content digest. Only then may it receive an immutable object name. The manifest root is a digest over ordered shard records:

$$R_{\text{manifest}} =H\!\left( \left[ (I_s,H_s,B_s,[r_s^{\min},r_s^{\max}),H_{\text{schema}},\text{state}_s) \right]_{s=1}^{S} \right).$$

The producer protocol is:

for shard_spec in immutable_plan.shards:
    staged = write_staging_object(
        materialize_targets(shard_spec, frozen_semantic_contract)
    )
    stats = validate_shape_mask_finiteness_and_ranges(staged)
    payload_hash = sha256_exact_bytes(staged)
    committed = publish_immutable_object(staged, payload_hash, stats)
    append_candidate_manifest_record(committed)

assert every_planned_shard_is_committed()
manifest_root = hash_ordered_shard_records()
publish_manifest_atomically(manifest_root)

A readable staging file is not a cache hit. A valid shard absent from the manifest is not part of the training input. A manifest referring to a staged or failed shard is invalid. This ordering makes interrupted production resumable without allowing a consumer to train on whichever files happened to finish first.

Atomicity here is a logical protocol. The Loop 8 oracle proves its state and hash rules on tiny synthetic records; it does not validate the transaction semantics of any cloud object store or distributed metadata service.

0x08J Revisions Form A DAG, Not An Overwritten Folder

Artifact evolution has dependencies:

raw source r1
    -> rendered/packed input r2
        -> teacher target r3
            -> filter view r5
                -> training manifest r6

Those nodes should be immutable and edges should form a directed acyclic graph. A changed filter can create filter-r6 that reuses target-r3; it should not rewrite filter-r5. A changed prompt template creates a new rendered node and invalidates every descendant target, filter view, and manifest while leaving the raw source node intact.

For revision graph $G=(V,E)$, validity requires

$$\lvert\operatorname{TopoSort}(G)\rvert=\lvert V\rvert.$$

The DAG answers two questions a flat cache directory cannot:

1. Why is this row stale? Trace which ancestor identity changed.
2. What can be reused? Reuse immutable ancestors whose semantics did not change, then materialize only new descendants.

Garbage collection is also graph-aware. A shard may be deleted only when no pinned manifest or retained descendant references it. This lifecycle belongs to artifact production; deployment package retention remains outside this article, and M8.9/M8.10 own it when the student is composed with a low-bit representation.

0x08K Filtering Is A Selection Operator

Quality, safety, rights, length, language, duplication, and contamination filters may be necessary. They are not distribution-neutral cleanup. Let the source distribution be $q(x)$ and let $a(x)\in[0,1]$ be acceptance probability. The retained distribution is

$$q_{\text{kept}}(x) =\frac{q(x)a(x)}{Z}, \qquad Z=\mathbb E_{x\sim q}[a(x)].$$

The derivation is conditional probability:

$$q_{\text{kept}}(x) =q(x\mid A=1) =\frac{P(A=1\mid x)q(x)}{P(A=1)} =\frac{a(x)q(x)}{Z}.$$

The fixture starts with slice masses $(0.5,0.2,0.2,0.1)$ and acceptance probabilities $(0.9,0.5,0.25,0)$. Therefore $Z=0.6$ and the kept distribution is

$$q_{\text{kept}} =\left(0.75,\frac16,\frac1{12},0\right).$$

An overall 60% acceptance rate hides that the final slice lost all support. No inverse-propensity weighting can recover a slice for which $a(x)=0$ because no accepted example contains information about it. Preserving rejected-row IDs, reason codes, and before/after slice counts is therefore part of auditability, not optional logging.

Dolma supplies a concrete source context: filtering, tagging, deduplication, mixing, and decontamination are interacting data-pipeline stages, and threshold choices change the selected corpus. Loop 8 uses that evidence to justify explicit filter-graph identity and slice reports; it does not transfer Dolma’s source-reported ablation numbers to distillation artifacts.

0x08L Deduplication And Decontamination Need Lineage Groups

Exact-string deduplication is not enough when one source record produces paraphrases, prompt templates, rationale variants, teacher responses, or student rollouts. Let $f(x)$ be the parent/near-duplicate family ID. A split function must operate on the family:

$$\operatorname{split}(x) =g(f(x)),$$

not independently on each derived row. The invariant is

$$x_i\sim x_j \Longrightarrow \operatorname{split}(x_i)=\operatorname{split}(x_j),$$

where $\sim$ denotes shared parent or declared near-duplicate lineage.

Deduplication should keep both a representative ID and alias lineage. Dropping all aliases without a map destroys the ability to explain source weights, filter outcomes, and derived-example ancestry. Decontamination should compare evaluation families against raw prompts and parents before teacher generation where possible; checking only final generated text can miss transformed siblings.

The exact paper and its official code also distinguish a conceptual policy from an executable deduplication pipeline. The code pin is evidence for one implementation lineage, not proof that its thresholds or cluster definition are universally correct for prompts, responses, or multilingual examples.

0x08M Rollout Records Have Policy Age And Expiry

A sequence generated by the teacher is stable with respect to future student updates. A student rollout is not. Let $r_c$ be the collecting student revision and $r_n$ the current revision. Define policy age

$$\Delta r=r_n-r_c.$$

Under a bounded-age policy with maximum $A_{\max}$, admission is

$$\operatorname{admit}(x) =\mathbf 1[0\le\Delta r\le A_{\max}].$$

The fixture records $r_c=5$, $r_n=7$, $\Delta r=2$, and $A_{\max}=3$, so the rollout remains admissible. At revision 9 it would expire. A stricter policy may require $\Delta r=0$; an importance-corrected replay policy requires behavior probabilities or another valid correction contract. Merely storing the token IDs is not sufficient.

The rollout row must bind:

collecting student checkpoint/content hash
generation configuration hash
RNG algorithm, seed, and counter
prompt and packed token identity
collection time or monotone revision
current age and maximum admitted age
replay policy and any behavior probabilities
teacher scoring revision, if scores were cached separately

Calling a bounded-age or corrected replay buffer on-policy without qualification erases an important approximation. Loop 7 owns the exact occupancy mathematics; Loop 8 makes the collecting policy and staleness persistent.

0x08N Reader Validation Reconstructs Semantics Before Loading Payload

A reader should fail closed before allocating large target tensors:

def open_supervision_artifact(run_contract, manifest):
    assert manifest.root_hash == run_contract.manifest_root_hash
    assert manifest.semantic_identity == run_contract.semantic_identity
    assert acyclic(manifest.revision_dag)

    for shard in manifest.ordered_shards:
        assert shard.state == "committed"
        assert shard.schema_hash == manifest.schema_hash
        assert checksum(shard.object) == shard.payload_sha256
        assert shard.row_range.continues_previous_range()

        metadata = read_metadata_before_payload(shard)
        assert typed_payload_union_is_supported(metadata)
        assert offsets_are_monotone_and_close(metadata)
        assert loss_mask_rows_equal_target_rows(metadata)
        assert sparse_estimator_is_fully_specified(metadata)
        assert lineage_and_split_are_compatible(metadata)
        assert rollout_age_is_admissible(metadata)

        yield load_validated_payload(shard, metadata)

Validation order matters. Reading terabytes and only then discovering a tokenizer mismatch wastes resources; worse, permissively coercing an unknown schema can train a valid-shaped but mathematically different objective. The reader should reject unknown payload tags, unknown estimator revisions, nonmonotone offsets, split-family leakage, expired rollouts, and manifest/shard digest disagreement.

0x08O Paper, Format, And Code Trace

Evidence	Exact pin	What Loop 8 takes from it	What remains outside the claim
Sparse Logit Sampling	2503.16870v2	incomplete-target gradient, deterministic top-$k$ bias, sampled estimator, variance/tail alternatives	no paper training or benchmark reproduction; pinned repo has no executable implementation
Datasheets for Datasets	1803.09010v8	composition, collection, preprocessing, distribution, maintenance, archival versions	not a distillation-specific row schema
Deduplicating Training Data	2107.06499v2	near-duplicate families, train/validation overlap, contamination risk	source-reported C4 results are not universal
Dolma	2402.00159v2	filter/tag/dedupe/mix pipeline, overlap, decontamination, release maintenance	source thresholds and ablations are not copied as prescriptions
Apache Arrow	commit a7d0bfa5...	arrays, record batches, buffers, IPC/file framing	does not choose a KD target estimator
Apache Parquet format	commit 2076361b...	row groups, column chunks, pages, page indexes	not automatically best for token-row access
Hugging Face Datasets	commit 48b7ee7b...	one concrete transform-fingerprint/cache lineage	not a complete teacher-target identity
Google deduplication code	commit 4e9888ac...	one executable near-duplicate pipeline lineage	no universal threshold or multilingual guarantee
Dolma code	commit 669f5348...	concrete mixer and deduper implementation surfaces	not executed by this article

Every paper PDF, source archive, relevant TeX file, official repository commit, and inspected code/document file is digest-pinned in the Loop 8 evidence bundle. The executable oracle is independent: it proves finite arithmetic, hashes, offsets, graph, filtering, split, and expiry contracts, not the papers’ training outcomes or a production data platform.

0x08P Artifact Failure Closure

Failure	Observable symptom	Violated contract	Repair
cache directory named only by model alias	results drift after quiet model update	semantic identity	pin checkpoint content and every target-changing field
top-$k$ values have no objective tag	stable tensors train different gradients	sparse estimator identity	declare normalized/unnormalized/tail/sampling contract
global log-normalizer omitted	retained logits cannot recover teacher probabilities	value semantics	store normalizer or explicitly store probabilities
ghost bucket treated as full tail	omitted-token gradients are invented	tail allocation	name aggregate-only semantics or add a tail model
sampled target omits proposal/RNG	finite rows cannot be replayed or reweighted	sampling identity	persist $q$, temperature, count, replacement, seed, counter
hidden rows omit hook location	matching is shape-correct but wrong	payload union	persist module path and pre/post-norm capture point
offsets are inferred from current packer	valid rows shift after code update	coordinate identity	persist source, packed, valid-row, and payload maps
Arrow/Parquet file name defines method	correct bytes receive wrong objective	layer separation	bind storage files to semantic schema in manifest
readable staging shard is consumed	training row count depends on interruption	atomic publication	expose only committed shards through final manifest
filter revision overwrites old subset	previous run cannot be reconstructed	revision DAG	publish immutable filter views and manifest roots
accepted rows only are retained	support loss cannot be audited	filter observability	retain rejection IDs/reasons and before/after slices
derived siblings cross splits	evaluation information reaches training	family lineage	split by parent/near-duplicate family
old rollout called current on-policy	prefix distribution lags student	policy age	persist collector revision and enforce replay policy
source figure treated as local result	external evidence becomes reproduction claim	evidence ceiling	label source-reported and keep independent oracle scope

0x08Q Loop Boundary

Loop 8 now owns and closes:

four-layer artifact architecture
canonical semantic identity and typed payload union
source -> packed -> valid-row -> payload coordinates
full-KD reference gradient and dense storage ledger
unnormalized top-k, normalized top-k, and ghost-tail distinctions
importance-sampled sparse targets, expectation, and finite variance
Arrow, Parquet, tensor-sidecar, and stream-layout alternatives
immutable shard validation and manifest-gated publication
append-only revision DAG and reuse/invalidation rules
filter selection distribution and zero-support failure
family-safe deduplication/decontamination lineage
rollout policy age, expiry, and reader fail-closed behavior

Loop 9 consumes those immutable row and shard contracts and owns the runtime problem: online versus offline versus hybrid execution, teacher/student placement, transfer and prefetch, vocabulary-parallel loss, checkpoint/resume, and failure recovery. Loop 8 deliberately does not claim that its logical state machine validates a production object store or distributed trainer.

0x09 Training Runtime And Distributed Execution

The same mathematical loss can create very different training systems. An online run pays for the teacher every batch. An offline run pays once and then turns storage bandwidth into the target source. A hybrid run carries both failure modes unless it identifies which path produced every row.

Owner question. Given the objective, row schema, and student architecture from 0x01-0x08, which processes execute each step, where do their tensors live, which collective preserves the objective, and what state makes an interrupted run resume as the same method?

Loop 9 owns the execution graph rather than another loss. It owns online, offline, hybrid, and on-policy process topology; teacher/student placement; logical target transfer; bounded queues and prefetch; exact vocabulary- and data-parallel reduction; checkpoint publication; and topology-changing resume. It consumes Loop 8’s immutable target manifest without redefining top-$k$ or sampling semantics. Loop 10 owns paper comparison, Loop 11 owns release, and M8.9/M8.10 own deployment packaging when the student is composed with a low-bit representation.

This section uses three evidence levels deliberately:

• primary papers establish algorithms and source-reported systems results;
• pinned framework repositories expose concrete, version-scoped code paths;
• the local Python oracle proves only small accounting and state-transition examples. It does not execute a GPU collective, framework trainer, object store, or production checkpoint backend.

0x09A Online, Offline, Hybrid, And On-Policy Are Cost Plans

The mode name is shorthand for a process graph. A complete graph names the target producer, prefix owner, transport, lifetime, reuse count, and failure boundary:

Mode	Target producer	Prefix owner	Transport and lifetime	Reuse/freshness state
online	live teacher each update	current training batch	device-local or cross-mesh; step/chunk lifetime	no amortization unless explicitly reused
offline	producer job	immutable artifact producer	committed shard -> reader -> device	manifest revision and counted reuse
hybrid	cache or live fallback	artifact or current batch	both paths converge on one schema	hit/miss source plus identical teacher/target identity
on-policy	current or age-bounded student rollout, then live teacher score	declared collector revision	rollout artifact plus teacher/student score passes	policy revision, RNG, age, and replay rule

An online token-KD step has approximate work

C_online ~= C_teacher_forward
          + C_student_forward
          + C_student_backward
          + C_alignment_losses
          + optional C_student_generation

The teacher has no optimizer state and normally no backward graph, but its weights, forward workspace, logits, communication, and any intermediate targets remain real. A useful peak-memory ledger is

M_peak ~= M_student_training_state
        + M_teacher_resident_weights
        + M_teacher_targets_still_live
        + M_alignment_state
        + M_runtime_and_collective_buffers

This is a scheduling ledger, not a promise that all terms peak at exactly the same instruction. Their overlap depends on forward order, pipeline schedule, chunked losses, activation checkpointing, and where the teacher is placed.

Execution mode	When teacher work occurs	Persistent artifact	Freshness	Dominant systems risk
online logits/states	every student batch	little or none	current teacher and input	teacher residency and throughput
cached full targets	producer job once per manifest	extreme logit/state shards	fixed	capacity, I/O, invalidation
cached sparse targets	producer job once per manifest	compact estimator payload	fixed	biased or under-specified tail
teacher sequences	generation job before training	token IDs plus provenance	fixed	decode policy collapses distribution
student on-policy	student generation, teacher scoring, student rescoring	usually ephemeral or short replay	high	three-phase scheduling and policy lag
hybrid	cache hits plus selected live queries	both	mixed	silent target-version drift

Offline execution amortizes teacher work only when artifacts are reused enough to repay production and storage. If $E$ student epochs replay the same target rows, the teacher forward may be amortized over $E$ uses. If augmentation, packing, or on-policy prefixes create new rows each epoch, the assumed reuse may not exist.

The amortization follows from comparing like-for-like student updates. Let $C_T$ be one teacher production pass, $C_I$ one offline read-plus-transfer, $C_S$ the student forward/loss/backward work, and $R$ the number of valid uses of the produced target:

$$C_{\mathrm{online}}=C_T+C_S, \qquad C_{\mathrm{offline/use}}=\frac{C_T}{R}+C_I+C_S.$$

Offline target delivery is cheaper on this narrow ledger only if

$$\frac{C_T}{R}+C_I<C_T \quad\Longrightarrow\quad R>\frac{C_T}{C_T-C_I},$$

which requires $C_I<C_T$. The fixture uses $C_T=4.0$ s, $C_I=0.5$ s, and $C_S=6.5$ s. Online is $10.5$ s per update. At $R=4$, offline is $8.0$ s per use, while the strict break-even is $R>1.143$. This does not include producer queueing, target invalidation, storage replication, or a changed example distribution; those can remove the apparent saving.

The recent offline top-$k$ and fused chunked-KL paper makes the same systems separation concrete: target production, persistent sparse payload, output projection, and loss materialization are independent memory decisions. Its speed, throughput, and H200 memory values remain source-reported for that implementation.

Source figure: Efficient Knowledge Distillation for LLMs, Figure 1, revision 2608.03796v1. The displayed memory values belong to the paper’s model, sequence, precision, and hardware setup; the local oracle does not reproduce them.

Throughput Is A Queue-Stability Contract

An offline or separate-teacher design has at least a producer, transport, and consumer. If teacher production serves $\mu_T$ rows/s and transport serves $\mu_I$ rows/s, the sustainable upstream service rate is

$$\mu=\min(\mu_T,\mu_I).$$

For a student consuming $\lambda_S$ rows/s, a steady bounded queue requires $\mu\ge\lambda_S$ after accounting for bursts and finite buffers. A discrete backlog evolves as

$$B_{n+1}=\max(0,B_n+A_n-C_n),$$

where $A_n$ is demanded work and $C_n$ is completed upstream service during interval $n$. In the fixture, teacher, transport, and student rates are $3000$, $2500$, and $2000$ rows/s: the steady service rate is $2500$ rows/s. When transport falls to $1800$ rows/s, the backlog grows by $200$ rows/s even though the teacher itself is fast enough. More prefetch delays the stall; it does not make an unstable queue stable.

0x09B Forward Order Determines Tensor Lifetime

A minimal co-resident online step looks simple:

teacher.eval()
student.train()

with no_grad():
    teacher_targets = teacher.forward(batch, return_targets=target_spec)

student_outputs = student.forward(batch, return_targets=target_spec)
loss = hard_loss(student_outputs, batch)
loss += distillation_loss(student_outputs, teacher_targets, masks)

del teacher_targets
loss.backward()                 # gradients belong to student/alignment only
optimizer.step()

The apparent no_grad() saving is limited. It avoids teacher gradient and optimizer state, but output logits still have shape [B,S,V], and mapped teacher activations must survive until their corresponding student loss is formed. Reversing the forward order retains the student’s backward activations while the teacher runs. Either order can produce a large overlap.

There are several implementation choices:

Choice	Target lifetime	Memory consequence	Compute/communication consequence
materialize all teacher logits	until student logits exist	full [B,S,V] target tensor	simplest loss implementation
chunk/fuse vocabulary loss	one vocabulary chunk at a time	avoids one full target copy	more kernel and collective coordination
compute mapped layers together	all mapped teacher states may remain live	can dominate logits	simple multi-loss backward
stream one mapped target	target released after local comparison	lower peak	recomputation or synchronized execution
cache targets on host/NVMe	device target is staged per batch	lower accelerator residency	PCIe/storage bandwidth and prefetching
separate teacher ranks/service	teacher weights leave student ranks	student memory improves	target transport and backpressure appear

The correct peak is an interval calculation, not the sum of every allocation named in the program. For tensor or state $i$ with live interval $[a_i,b_i)$ and size $M_i$,

$$M_{\mathrm{peak}} = \max_t\sum_i \mathbf 1[a_i\le t<b_i]M_i.$$

One deliberately small schedule demonstrates the distinction:

Interval	Live state beyond 28 GiB base weights/training state	Total
$[0,4)$ s	4 GiB teacher activations	32.0 GiB
$[4,6)$ s	1 GiB streamed target chunk + 6 GiB student activations	35.0 GiB
$[6,6.5)$ s	previous state + 0.5 GiB loss scratch	35.5 GiB
$[6.5,10.5)$ s	6 GiB student activations through backward	34.0 GiB

Materializing all teacher logits would replace the 1 GiB chunk with the full target allocation. Running the student first would retain its 6 GiB backward state while the teacher executes. Streaming hidden targets can shorten their lifetime, but it may force teacher/student pipeline synchronization or recomputation. The schedule is therefore part of the memory proof.

A tensor-level implementation should make release points reviewable:

with torch.no_grad():
    teacher_hidden = teacher(batch).last_hidden_state       # [B,S,d_T]
    target_chunk = project_or_slice(teacher_hidden, spec)   # [B,S_c,d_A]
del teacher_hidden                                         # no longer live

student_hidden = student(batch).last_hidden_state          # saved for backward
numerator = tensor(0.0, device=student_hidden.device)
denominator = tensor(0, device=student_hidden.device)
for rows in causal_target_chunks(batch.loss_mask):
    local_target = stage(target_chunk, rows, non_blocking=True)
    local_student = map_student(student_hidden, rows)
    numerator += loss_sum(local_student, local_target)
    denominator += rows.valid_count
    del local_target

loss = globally_normalized(numerator, denominator)
loss.backward()

This code is a lifetime trace, not a recommendation to retain one giant student_hidden tensor. Pipeline schedules, selective activation checkpointing, and fused output losses can change its implementation while preserving the same ownership contract.

On-policy execution adds another phase boundary:

with no_grad():
    rollout = student.generate(prompts, generation_config, rng_state)

packed = pack_exact_student_prefixes(prompts, rollout.ids)

with no_grad():
    teacher_scores = teacher.score(packed)

student_scores = student.score(packed)  # fresh differentiable forward
loss = masked_divergence(student_scores, teacher_scores, rollout.mask)
loss.backward()

Generation K/V, rollout IDs, teacher scoring buffers, and the fresh student backward graph do not all need to coexist, but an implementation that keeps them accidentally can exceed the memory of ordinary online KD. The rollout and scoring schedules also need the exact same packed token IDs, positions, EOS policy, and response mask established in 0x07.

0x09C Exact Vocabulary-Parallel KL Does Not Require A Logit All-Gather

Suppose teacher and student share one vocabulary and rank $r$ owns vocabulary shard $\mathcal V_r$. For one [B,S] grid of valid rows, each rank first finds its local teacher maximum. A MAX all-reduce gives the stable global maximum:

$$m_T=\max_r\max_{v\in\mathcal V_r}z^T_v.$$

Each rank then computes a shifted local exponential sum. A SUM all-reduce gives the full shifted denominator:

$$\widetilde Z_T=\sum_r\sum_{v\in\mathcal V_r}\exp(z^T_v-m_T), \qquad \log p_T(v)=z^T_v-m_T-\log \widetilde Z_T.$$

Repeat the same two reductions for the student. Each rank can then accumulate its local forward-KL contribution

$$k_r=\sum_{v\in\mathcal V_r} p_T(v)\left[\log p_T(v)-\log p_S(v)\right],$$

and a final SUM all-reduce produces the exact per-row KL. In pseudocode:

def sharded_log_probs(local_logits, vocab_group):
    local_max = local_logits.max(dim=-1).values        # [B,S]
    global_max = all_reduce_max(local_max, vocab_group)
    local_sum = exp(local_logits - global_max[..., None]).sum(dim=-1)
    global_sum = all_reduce_sum(local_sum, vocab_group)
    return local_logits - global_max[..., None] - log(global_sum[..., None])

logp_t = sharded_log_probs(local_teacher_logits, teacher_vocab_group)
logp_s = sharded_log_probs(local_student_logits, student_vocab_group)
p_t = exp(logp_t)
local_kl = (p_t * (logp_t - logp_s)).sum(dim=-1)       # [B,S]
row_kl = all_reduce_sum(local_kl, aligned_vocab_group)
loss = masked_mean(row_kl, response_mask)

This is communication pseudocode, not a drop-in autograd implementation. The student path needs an autograd-aware collective or fused vocabulary-parallel loss whose backward propagates through the global sum. The teacher path can be evaluated under no_grad().

The Forward And Backward Must Close On The Same Global Distribution

For student logits $z_v$ split over vocabulary shards, define

$$m_r=\max_{v\in\mathcal V_r}z_v, \qquad m=\max_r m_r,$$ $$Z=e^m\sum_r\sum_{v\in\mathcal V_r}e^{z_v-m}, \qquad \log q_v=z_v-\log Z.$$

The maximum can be detached because adding or subtracting a common constant does not change softmax. The global exponential sum cannot be replaced by a non-differentiable local value on the student path: every shard’s logits affect $Z$ and therefore every $q_v$.

For normalized teacher probabilities $p$, forward KL is

$$L=\sum_v p_v(\log p_v-z_v+\log Z).$$

Differentiating one logit $z_u$ gives

$$\frac{\partial L}{\partial z_u} =-p_u+\left(\sum_v p_v\right)\frac{e^{z_u}}{Z} =q_u-p_u.$$

That last equality uses $\sum_vp_v=1$. Loop 8’s unnormalized sparse target has retained mass $m_p<1$, so its corresponding gradient is $m_pq_u-p_u$ on retained coordinates and $m_pq_u$ elsewhere. A distributed runtime must not quietly turn one estimator into the other while optimizing communication.

The fixture uses $p=[0.4,0.3,0.2,0.1]$ and $z=[1.0,0.2,-0.1,-0.4]$, split as $[0,2)$ and $[2,4)$. It obtains

global max m                 = 1.000000
local shifted sums           = [1.449329, 0.579468]
global shifted sum           = 2.028797
log Z                        = 1.707443
q                            = [0.492903, 0.221476, 0.164073, 0.121548]
shard KL terms               = [0.007502, 0.020087]
dense KL = sharded KL        = 0.0275887865
concatenated shard gradient  = q - p
                             = [0.092903,-0.078524,-0.035927,0.021548]

The equality checks both the scalar objective and every logit gradient. A forward-only scalar test would miss a broken autograd collective.

The pinned Megatron cached-logit path at commit c3d44dcf... provides one current implementation trace: cached_logits_loss.py performs a TP MAX reduction, an autograd-aware SUM reduction, local lookup of the teacher’s global top-$k$ indices, and optional residual-mass handling. The pinned Full-Chunked-KL-Loss implementation recomputes output-projection chunks in backward and forms local sparse-target gradients. These code paths verify implementation surfaces; they do not prove that every Megatron-derived trainer or fused kernel has identical semantics.

Data Parallelism Reduces Numerators And Denominators, Not Rank Means

After vocabulary shards produce one loss per selected row, data-parallel ranks still have unequal valid-token counts because of padding, truncation, packed examples, or response masks. Let $N_{r,m}$ be the summed loss and $D_{r,m}$ the valid-token count for rank $r$, microbatch $m$. The gradient-accumulation correct objective is

$$L=\frac{\sum_r\sum_m N_{r,m}}{\sum_r\sum_m D_{r,m}}.$$

It is not the mean of rank-local means. With rank sums $[3.0,1.5]$ and counts $[3,1]$, the global mean is $4.5/4=1.125$; averaging rank means gives $(1.0+1.5)/2=1.25$. The error changes with padding and world size. The loss implementation therefore needs a globally reduced denominator or an equivalent scaling contract coordinated with DDP/FSDP gradient averaging.

The current pinned TRL GKD implementation explicitly accepts a global num_items_in_batch for gradient-accumulation-correct normalization. That is a useful code trace, not evidence that every trainer supplies that value correctly under every distributed sampler.

The derivation assumes teacher and student shards represent the same token IDs on ranks that can pair their local contributions. If shard boundaries differ, one side must be repartitioned. If teacher and student occupy separate rank pools, teacher target shards must be transported or recomputed where the loss runs. If tokenizers differ, matching same-index logits is not repaired by an all-reduce; 0x03D’s semantic alignment problem remains.

For scale, with B=4, S=2048, V=128000, and two-byte logits, gathering a full local batch vocabulary would expose about 2.10 GB of logit payload:

4 * 2,048 * 128,000 * 2 bytes ~= 2.10 GB

Each max or sum reduction above operates on an [B,S] array instead. At FP32, that logical array is 32 KiB. Collective algorithms still move more physical traffic than this logical payload, and separate teacher/student rank pools add target transfer, but the vocabulary dimension need not be globally gathered.

Megatron Core’s vocabulary-parallel cross-entropy API documents the same systems principle for hard-label CE: logits remain split across tensor-parallel ranks and the tensor-parallel group participates in the reduction. The KL construction above is a derivation for distillation, not a claim that every framework exposes this exact fused primitive.

0x09D Teacher Placement Changes The Bottleneck

Teacher placement is part of the method specification:

Placement	Teacher weights	Target path	Useful when	New constraint
co-resident on student ranks	overlap student model state	device-local or same-rank shards	both models fit and topology aligns	memory peak and kernel scheduling
separate accelerator ranks	isolated teacher pool	streamed logits/features	student needs its full ranks	network volume, queueing, shard alignment
sequential device residency/offload	staged from host/NVMe	local after transfer	compute is available but device memory is not	transfer latency can dominate
remote teacher service	service owns teacher	RPC responses or generated text	teacher shared across jobs or API-only	retries, nondeterminism, rate limits, version pinning
offline producer	absent during student training	immutable cache shards	examples are stable and reuse is high	storage and invalidation

Placement changes the critical path only through a declared schedule. For $N_{\mathrm{valid}}$ target rows, vocabulary size $V$, target width $d_T$, and $b$ bytes per value, two logical payloads are

$$M_{\mathrm{logits}}=N_{\mathrm{valid}}Vb, \qquad M_{\mathrm{hidden}}=N_{\mathrm{valid}}d_Tb.$$

At $N_{\mathrm{valid}}=256$, $V=50{,}000$, $d_T=4096$, and $b=2$, full logits are $25.6$ MB while one hidden layer is $2{,}097{,}152$ bytes (2 MiB), a $12.21\times$ logical ratio. Multiple mapped hidden layers, attention relations, projection placement, sparse indices, padding, protocol headers, and collective algorithms change the comparison. These are logical payload bytes; physical wire traffic depends on route multiplicity and topology.

A separate teacher mesh does not merely send “the tensor.” It converts a source layout into the layout expected by the loss or student mesh. The cross-mesh resharding paper formalizes this as many-to-many multicast and studies schedules that can overlap communication. Its diagram is useful here because teacher/student meshes can differ in replica and shard axes even when their global tensor shapes match.

Source figure: On Optimizing the Communication of Model Parallelism, cross-mesh resharding examples, revision 2211.05322v2, cropped to the diagram. It establishes the layout-conversion problem; it is not a reproduced distillation topology or performance result.

An implementation contract therefore needs both global token semantics and layout metadata:

TargetShard = {
    "global_token_range": [v_start, v_end],
    "row_coordinate_hash": packed_row_hash,
    "source_mesh": {"dp": 4, "tp": 8, "pp": 1},
    "source_layout": "vocab_sharded",
    "destination_mesh": {"dp": 8, "tp": 4, "pp": 2},
    "destination_layout": "loss_owner_vocab_shard",
    "dtype": "float16",
    "target_spec_hash": target_spec_hash,
}

assert reshard_plan.covers_each_global_coordinate_once()
assert receive_row_hash == student_row_hash

Replica fan-out, all-to-all, point-to-point pipeline transfer, and host staging can all realize the logical conversion. The article does not prescribe one collective without a topology and measured link map.

Current framework behavior must be source- and version-scoped. For example, the NeMo Framework distillation guide currently describes loading both checkpoints, running forward passes for both models and backward only for the student. Its documented path requires the two models to support the same parallelism strategy; with pipeline parallelism, intermediate-state losses are documented only on the final pipeline stage. Those are constraints of that implementation path, not mathematical requirements of knowledge distillation.

Intermediate transfer can make placement harder than logit KD. A separate teacher pool must send selected [B,S,d_T] features or relations in addition to output supervision. Pipeline stages need to agree where mapped layers meet. A width projection may belong on teacher ranks, student ranks, or the transport boundary; that choice changes bytes, checkpoint ownership, and gradient flow. Only trainable student-side alignment modules should receive gradients.

There are three common pipeline alternatives:

Alternative	Where teacher target is produced	Where loss runs	Consequence
terminal-stage only	teacher final stage	student final stage	simplest transport; cannot match earlier states directly
mapped-stage rendezvous	corresponding teacher/student stages	each mapped student stage	lower target lifetime, more synchronization and loss reduction
target collector stage	teacher stages send selected states to one owner	dedicated or student stage	centralized alignment/projection, but extra activation traffic

If teacher and student have different depths, “same pipeline stage” is not a layer map. The semantic map from 0x04 and the physical route are separate objects. A trainable width projection placed on the teacher mesh also changes gradient ownership: either it must be part of the student optimization domain, or it must remain frozen and be identified in the target contract.

0x09E Offline Readers And Hybrid Fallback Are Stateful Systems

The Reader Owns A Coordinate Cursor, Not Just A File Name

An offline artifact is immutable, but its reader is stateful. It owns the manifest revision, current shard and entry, source-to-current data-parallel mapping, prefetch queue, decode workers, pinned-memory buffers, and the point at which a checksum-validated row becomes visible to training.

The pinned Megatron cached-logit implementation provides a concrete current trace: tar shards are grouped by source DP rank, remote objects can be prefetched, decode work can use a thread pool, and a DataLoader stages tensors through pinned host memory for non-blocking device copies. Its exact saved-DP to current-DP behavior is implementation-specific. A simpler teaching map for downscaling from saved DP 4 to current DP 2 is

$$\operatorname{owner}(r_{\mathrm{source}}) =r_{\mathrm{source}}\bmod DP_{\mathrm{current}},$$

which yields current rank 0 reading source ranks $[0,2]$ and current rank 1 reading $[1,3]$. This map is not universal. What is universal is the invariant: every committed logical row must have one declared owner under the new topology, and resume must preserve or explicitly document ordering semantics.

reader_state = {
    "manifest_id": "manifest-r6",
    "saved_dp": 4,
    "current_dp": 2,
    "cursor": {"shard": 7, "entry": 11},
    "source_rank_assignment": {0: [0, 2], 1: [1, 3]},
    "prefetch": [
        {"manifest_id": "manifest-r6", "shard": 8},
        {"manifest_id": "manifest-r6", "shard": 9},
        {"manifest_id": "manifest-r6", "shard": 10},
    ],
}

for item in reader_state["prefetch"]:
    assert item["manifest_id"] == reader_state["manifest_id"]
    validate_committed_checksum(item)

Changing DP degree without remapping can duplicate or skip rows. Restoring only the shard number without its entry offset can replay a shard prefix. Restoring prefetch buffers from another manifest can produce a few correctly shaped but semantically stale batches after restart.

A Hybrid Path Must Not Silently Mix Targets

A cache miss can fall back to online teacher inference, but the result needs the same contract checks as a cache hit:

key = build_cache_key(batch, frozen_target_contract)
cached = cache.lookup(key)

if cached is not None:
    assert cached.schema == expected_schema
    assert cached.teacher_revision == pinned_teacher_revision
    targets = cached.payload
    target_source = "cache"
else:
    targets = online_teacher(batch, frozen_target_contract)
    target_source = "online"
    cache.stage_for_atomic_publish(key, targets)

loss = distill(student(batch), targets)
telemetry.record(target_source, targets.revision, loss)

The fallback is valid only if all target-defining state is equal:

cached teacher revision       == live teacher revision
cached tokenizer/vocabulary   == live tokenizer/vocabulary
cached temperature/objective  == live temperature/objective
cached row/packing identity    == current row/packing identity
cached sparse-tail estimator  == live target estimator

If any field differs, the online result is a new artifact revision, not a cache fill. A retry policy must also distinguish a transient transport failure from a semantic miss. Retrying a pinned teacher request is operational; silently routing to a newer teacher is a method change.

Online raw logits and cached temperature-scaled probabilities are not interchangeable just because both have shape [B,S,V]. Nor are targets from two teacher checkpoints. Hybrid telemetry should report cache hit rate, source mix by data slice, target age, teacher revision, and loss by source. Otherwise a quality change caused by the online/cache mixture can be mistaken for an optimizer effect.

0x09F Checkpointing Must Capture More Than Student Weights

The deployable artifact is still the student, but a resumable training job needs the state of the entire supervision process:

student:
    weights, optimizer, scheduler, scaler, trainable alignment modules

data position:
    sampler state, source-mixture counters, packing cursor, accumulation step

teacher contract:
    teacher revision, tokenizer/vocabulary hash, target specification

cache contract:
    pinned manifest ID, completed producer shards, reader cursor/prefetch state

on-policy state:
    generation configuration, RNG, lambda schedule, rollout buffer
    generating student revision and accepted replay-age policy

distributed state:
    data/tensor/pipeline topology, shard metadata, loss normalizers

observability:
    valid-token denominator, source counts, dropped rows, cache hits/misses

Not every item belongs in one model checkpoint file. It must, however, be captured in the run checkpoint or immutable run manifest. Resume should fail loudly when the pinned cache manifest is unavailable or the teacher revision changes. Replaying a partially consumed data shard from its beginning may be acceptable if documented; silently skipping or duplicating unknown rows is not.

For on-policy methods, a resume boundary is also a policy boundary. Rollouts created before the checkpoint can be replayed only under an explicit off-policy age rule. Calling them current on-policy data after restoring a newer student changes the method.

The local fixture requires sixteen state classes. Teacher weights need not be copied into the run checkpoint when the teacher is an immutable external artifact, but teacher identity and target semantics do:

State class	Why it changes the resumed method if missing
student model, optimizer, scheduler, scaler	update rule and numerical trajectory
RNG, global step, accumulation cursor	sampling, dropout, schedule, and update boundary
sampler/packer cursor and source-mixture counters	occupied example and causal-row distribution
teacher contract and target-manifest ID	target distribution and payload semantics
target-reader cursor	exact persistent-row position
rollout policy state	prefix distribution and replay age
DP/TP/PP topology and shard metadata	ownership and reconstruction of state
loss-normalizer state	correct accumulated numerator/denominator
async checkpoint generation	which snapshot is complete and visible

Asynchronous Save Has Snapshot, Write, And Publication Boundaries

“Save started” is not “checkpoint resumable.” A correct asynchronous path has at least these states:

generation 12, training step 100
    -> freeze immutable logical snapshot
    -> stage device state to owned host/storage buffers
    -> write rank shards while training may advance to step 101
    -> verify all rank writes and metadata
    -> run cross-rank finalize callbacks
    -> atomically publish generation 12 manifest

The visibility predicate is

$$\operatorname{visible}(g) \iff \operatorname{staged}(g) \land \operatorname{written}(g) \land \operatorname{finalized}_{\mathrm{all\ ranks}}(g).$$

Training may mutate live step-101 state only after generation 12 owns immutable staging buffers or a copy-on-write equivalent. A directory containing some rank files is not a checkpoint. Rotation must not delete the last finalized generation because a newer generation has merely started.

The current pinned implementations expose compatible but distinct surfaces:

• PyTorch Distributed Checkpoint async_save distinguishes staging completion from upload completion and marks the API experimental;
• Megatron’s async request can be frozen and attaches finalize callbacks that run after distributed completion;
• NeMo’s distributed-checkpoint wrapper tracks pending async saves and warns when finalization has not completed.

These are version-scoped code observations, not one shared transaction API.

Resume Across A New Topology Is A Reshard Plus Data-Position Problem

Suppose the saved run uses DP 4, TP 2, PP 1 and the restarted run uses DP 2, TP 4, PP 1. Matching global world size is insufficient. Model and optimizer tensors need a representation that can be mapped from the saved ownership to the current ownership. The vocabulary hash, parameter names/shapes, and optimizer slot semantics must remain compatible. Separately, the data reader must remap DP ownership and restore its logical cursor.

The Universal Checkpointing paper represents model parameters independently of one training parallelism strategy, then reconstructs the requested target strategy. Its source figure makes the distinction between one logical parameter checkpoint and many runtime shard layouts explicit.

Source figure: Universal Checkpointing, overview figure, revision 2406.18820v3. It explains the paper’s parallelism-decoupled representation; its scale and performance results remain source-reported.

A fail-closed resume gate is broader than load_state_dict:

def admit_resume(saved, current):
    require(saved.checkpoint_state == "finalized")
    require(saved.student_schema.compatible_with(current.student_schema))
    require(resharder.supports(saved.topology, current.topology))
    require(reader.can_map(saved.data_cursor, current.dp))
    require(saved.teacher_contract == current.teacher_contract)
    require(saved.target_manifest_id == current.target_manifest_id)
    require(saved.tokenizer_hash == current.tokenizer_hash)
    require(saved.rollout_policy.admissible_at(current.student_revision))

Failure injection should interrupt after staging, after one rank write, before finalization, after publication, and during reader prefetch. Only the finalized generation may be selected; each accepted resume must reproduce the next logical row IDs and the same global loss numerator/denominator as an uninterrupted control run.

0x09G Pinned Runtime Code Trace And Evidence Ceiling

The source set is intentionally heterogeneous because no one repository owns the entire runtime:

Surface	Exact pin	What the code trace establishes	What it does not establish
Megatron-LM	commit c3d44dcf...	cached tar-shard reader, prefetch, DP resize path, TP-normalized sparse KL, async finalization primitives	universal cached-KD schema or production throughput
NVIDIA Model Optimizer	commit a21173a4...	co-resident student/teacher trainer path and ModelOpt checkpoint integration	every placement or framework backend
TRL GKD	commit 24db2a82...	generation, response masking, live teacher/student rescoring, global-item normalization hook	sequence-level estimator equivalence for arbitrary configs
Full-Chunked-KL-Loss	commit 38d09ce3...	chunked output projection, TP max/sum, backward recomputation	paper benchmark reproduction in this repo
PyTorch	commit 3b7cd3f2...	Stateful protocol and async checkpoint staging/upload futures	a distillation-specific run manifest
NeMo	commit b24cbb5e...	resumable dataloader inspection and pending distributed-checkpoint finalization	implementation-independent resume behavior

The evidence bundle pins every inspected file digest. Current documentation is also version-sensitive: older NeMo distillation documentation described same-parallelism co-residency and narrower PP/FSDP support, while newer recipe surfaces differ. Framework behavior must therefore be cited with a version or commit, not phrased as a permanent property of knowledge distillation.

0x09H Runtime Failure Lab

Failure	Observable symptom	Violated runtime contract	Repair
teacher is frozen but its graph is retained	memory resembles two-model training	teacher gradient ownership	no_grad, detach targets, audit saved tensors
teacher and student forwards overlap all targets	unexplained peak-memory spike	target lifetime	chunk/fuse losses or stream mapped targets
sharded softmax uses only local normalization	rank-dependent probabilities and loss	global vocabulary partition function	all-reduce max and shifted exponential sum
student partition sum uses a non-autograd collective	scalar forward is right but logit gradients are local	distributed backward ownership	autograd-aware SUM or fused backward with dense-gradient oracle
local KL terms are averaged before shard sum	loss changes with tensor-parallel degree	vocabulary reduction	sum vocabulary contributions, then reduce rows
rank-local means are averaged	padding pattern changes the objective	global token normalization	reduce loss numerator and valid-token denominator
separate rank pools assume matching shard IDs	correct shapes but mismatched tokens	vocabulary partition alignment	explicit repartition/token-ID routing
transport service is slower than student demand	prefetch queue grows until a periodic stall/OOM	queue stability	measure service rates, bound backlog, change placement or consumption
cache fallback uses a newer teacher	loss jumps only on cache misses	pinned teacher contract	reject or create a new manifest/run
host-offloaded targets miss prefetch deadlines	accelerator utilization collapses	input pipeline bandwidth	shard locally, overlap prefetch, measure stalls
partial cache shard is accepted	nondeterministic row count or decode error	atomic artifact publication	checksum and manifest-gated visibility
DP degree changes without reader remap	valid rows repeat or disappear after resume	logical data-position ownership	map every source rank/row once and persist cursor semantics
checkpoint restores student only	source mix and LR resume but data differs	full run state	restore sampler, cache, RNG, and supervision schedule
pending async generation is selected	some ranks or optimizer slots are absent	cross-rank checkpoint finalization	publish only a finalized manifest; retain prior finalized generation
staged snapshot aliases live tensors	saved step-100 state contains step-101 mutations	immutable snapshot generation	owned staging buffers or copy-on-write fence
old rollout buffer is called on-policy	prefix distribution lags current student	policy revision/age	expire, reweight, or label replay explicitly

0x09I Loop Boundary

The execution plan can now be audited independently of the loss name:

example distribution
    -> defines which inputs and prefixes are occupied

artifact schema and manifest
    -> define what teacher information survives and when it is valid

placement and tensor lifetime
    -> define peak memory, transport, and throughput

distributed normalization
    -> preserves the mathematical vocabulary objective across shards

checkpoint and recovery state
    -> preserve the actual training method across interruption

The executable Loop 9 fixture closes the following equalities and state transitions without claiming a production run:

online critical path                   = 10.5 s
offline amortized path at reuse 4      = 8.0 s
chunked liveness peak                  = 35.5 GiB
stable producer/transport service      = 2,500 rows/s
unstable backlog growth                = 200 rows/s
dense KL                               = sharded KL
dense q-p gradient                     = concatenated shard gradients
global token mean                      = 1.125 (not rank-mean 1.25)
saved DP 4 -> current DP 2              = {0:[0,2], 1:[1,3]}
checkpoint visibility                  = all-rank finalized only
saved DP4/TP2 -> current DP2/TP4        = admitted only with reshard + cursor map

The oracle also rejects 163 one-field mutations spanning pinned evidence, mode identity, service rates, shard boundaries, loss normalization, reader state, hybrid identity, snapshot publication, topology resume, and claim ceilings. This is evidence for the stated contract, not a substitute for fault injection on the intended trainer and storage stack.

Loop 10 can now compare paper variants without hiding these differences behind a single “KD” column. Loop 11 will own evaluation, instrumentation, promotion, rollback, and the student-only release contract; Loop 12 will own final source, formula, render, accessibility, and cross-article closure.

0x0A Paper-By-Contract And Equal-Budget Comparison

The phrase knowledge distillation hides several different interventions. One paper changes the target distribution. Another changes which prefixes are visited. Another adds a projection and matches hidden states. Another chooses a smaller architecture and an initialization. Another creates an offline corpus. Another inserts an assistant model or a pruning phase. Comparing those methods as if each were a loss-function toggle erases the mechanism that produced the result.

Owner question. Given two named distillation methods, which coordinates of the learning and execution contract differ, and under which explicitly frozen resource denominator would an empirical comparison be interpretable?

Loop 10 owns the comparison schema, the paper-by-contract matrix, the cost event ledger, equal-budget admission, composition accounting, and the evidence transfer firewall. It consumes the exact objectives, tensor traces, architectures, artifact schemas, and runtimes from 0x01-0x09. It does not choose a production method: 0x0B owns strategy selection, Loop 11 owns evaluation and promotion, and M8.9/M8.10 own deployment packaging when the student is composed with a low-bit representation.

Three evidence levels remain separate:

• the revision-pinned papers establish their own method and source-reported experiments;
• the fixed-computation study and its pinned code show one concrete comparison protocol, not a universal fairness theorem;
• the local oracle validates a small contract and accounting fixture. It does not train any paper model or reproduce any benchmark.

0x0AA A Method Name Is Not An Executable Contract

Represent a method $m$ by the tuple

$$\mathcal{K}_m = (O_m,P_m,I_m,A_m,S_m,R_m,Q_m,E_m),$$

where:

Coordinate	Meaning	Examples
$O_m$	matched object	softened logits, hidden rows, attention scores, value relations, decoded sequences
$P_m$	prefix and data distribution	fixed teacher-forced rows, teacher-decoded corpus, student-generated prefixes
$I_m$	teacher interface	text only, logits, selected hidden states, Q/K/V-derived relations, live trajectory scoring
$A_m$	alignment contract	token support, target shift, layer map, width projection, head repartition, mask
$S_m$	student construction	random initialization, inherited layers, compact pretraining, pruning, assistant path
$R_m$	runtime topology	online, offline, hybrid, generation plus scoring, multi-stage chain
$Q_m$	persistent training state	target shards, projections, assistant checkpoints, rollout revision, reader cursor
$E_m$	evidence regime	architecture family, task, data, compute denominator, hardware, evaluation protocol

The tuple is deliberately larger than a loss equation. For example, replacing forward KL by generalized JSD changes only part of $O_m$; switching from fixed rows to student prefixes changes $P_m$, $R_m$, and usually $Q_m$. Calling both changes gkd=True is not a complete implementation description.

The executable Loop 10 manifest requires every paper row to name all eight coordinates. Two rows may share a loss and still be different methods. Two papers may share a family label and still be incomparable because their student construction or evidence regime differs.

0x0AB Output And Sparse-Distribution Papers Change Different Objects

The first family starts at the output interface, but even these papers do not define one interchangeable objective.

Paper contract	$O_m$ and $P_m$	Interface and alignment	Construction, runtime, and evidence boundary
Ba and Caruana, 1312.6184v7	student approximates a teacher function on transfer examples	teacher output targets; shared output semantics	pre-Hinton function matching in the paper’s classification regime; it does not define temperature-scaled softmax KD
Hinton et al., 1503.02531v1	hard target plus softened class distribution on fixed examples	common class support and the same temperature convention	ordinary student update with online or materialized targets; architecture compression is external to the loss
Sparse Logit Sampling, 2503.16870v2	sparse stochastic estimator of the full teacher-distribution gradient	sampled indices, proposal probabilities, exact row identity	offline sparse artifact plus estimator metadata; deterministic normalized top-$k$ is not the same estimator

The output family therefore already contains three distinct claims:

1. a student can approximate a teacher’s function;
2. softened probabilities expose relative class evidence;
3. sparse storage can preserve a dense gradient only under a stated estimator.

The first claim does not imply the second, and neither implies the third. Loop 2 owns the dense probability algebra; Loop 8 owns the sparse payload and estimator proof. Loop 10 records their distinct contracts instead of deriving them again.

0x0AC Intermediate And Architecture Papers Change The Alignment Graph

Intermediate transfer is not one family of shape-compatible MSE losses. Each paper chooses a different teacher object, map, schedule, and student graph.

Paper contract	Matched object	Alignment hypothesis	Training and architecture contract
FitNets, 1412.6550v4	one teacher hint through a learned regressor	selected teacher/student layers plus spatial and channel compatibility	hint-regression stage precedes output training; the regressor is training state
Attention Transfer, 1612.03928v3	normalized activation-derived attention maps	selected residual groups and a paper-specific map operator	relation-like spatial transfer; not Transformer attention-probability matching
DistilBERT, 1910.01108v4	MLM target, softened output, and cosine hidden direction	BERT-family token rows and selected hidden states	six-layer inherited-width student initialized from alternating teacher layers; encoder pretraining recipe
Patient KD, 1908.09355v1	task, prediction, and normalized intermediate states	PKD-Skip or PKD-Last layer map	task distillation; a layer map is a method hypothesis rather than semantic ground truth
TinyBERT, 1909.10351v5	embeddings, hidden states, unnormalized attention scores, and prediction logits	layer map, learned width projections, token/head/mask identity	general distillation followed by task-specific distillation and augmentation
MiniLM, 2002.10957v2	last-layer QK attention and VV relation distributions	aligned token pairs and compatible relation heads	task-agnostic relation distillation; raw hidden width need not match
MiniLMv2, 2012.15828v2	query, key, and value relations after concatenation and repartition	relation-head count divides the concatenated width	relation-head choice changes the target geometry; it is not merely a kernel parameter

These seven rows span at least four different alignment graphs: direct hidden projection, activation-map normalization, attention-score matching, and relation-distribution matching. The paper-specific figures retained in Loop 4 show why a single intermediate_loss box would be misleading.

The visual is a readable nine-method subset, not the full executable manifest. The manifest also includes Ba and Caruana, FitNets, Attention Transfer, MiniLMv2, Minitron, and Sparse Logit Sampling so that every one of the 15 core papers is represented.

0x0AD Sequence, Occupancy, Capacity, And Structure Change The Process Graph

The remaining core papers alter where targets come from, how often the teacher runs, or which student graph is trained.

Paper contract	Changed process	Persistent artifact or state	Scope that must not be erased
Sequence-Level KD, 1606.07947v4	decode one or more teacher sequences, then train on those sequences	generated corpus, decode configuration, source-to-target identity	one decoded sequence is an approximation to a sequence distribution
Teacher Assistant KD, 1902.03393v2	train $T\rightarrow A$, then $A\rightarrow S$	assistant checkpoint and two run manifests	changes the capacity path and total training work, not the base KD target
MiniLLM, 2306.08543v6	sample student trajectories and optimize a reverse-sequence-KL estimator	rollout policy, behavior correction, reward-to-go state	token-local reverse KL without sequence credit is not the paper method
GKD, 2306.13649v3	mix fixed and student-generated prefixes, then teacher-score and differentiably student-rescore	generation schedule, mixture coefficient, divergence, response mask	sampled IDs are stopped; the live path has generation, teacher score, and student rescore passes
Minitron, 2407.14679v2	prune a pretrained teacher-derived graph, then recover it with KD	pruning/search decision, pruned checkpoint, recovery run	structural compression is owned by M6.11; KD is one recovery stage in a larger pipeline

This table prevents a common causal mistake. If Minitron and ordinary token KD produce different deployment models, the observed difference cannot be attributed to a divergence alone. If GKD and offline sequence KD see different prefix distributions, the result mixes objective and occupancy changes. If a teacher assistant helps, the experiment has also purchased an extra model and an extra optimization trajectory.

0x0AE Equal Budget Begins With Event Counts, Not A Single Word

For a candidate $m$, count the work-producing events before assigning costs:

$$\mathbf{r}_m = \left( n_{S,f}, n_{S,b}, n_{T,f}, n_{T,d}, n_{S,d}, B_{write}, B_{read}, M_{peak}, N_{runs} \right).$$

The subscripts denote student forward, student backward, teacher scoring forward, teacher decode, and student decode. The byte terms are artifact traffic, $M_{peak}$ is peak residency, and $N_{runs}$ records separate training stages. This is an event vector, not yet a scalar cost.

Given a declared unit-cost vector $\mathbf{u}$, an additive compute estimate is

$$C_m = \mathbf{u}_{compute}^{\mathsf T}\mathbf{r}_{m,compute}.$$

That equation follows by partitioning one method into mutually exclusive forward, backward, and decode events, then summing the work of each event. It does not convert bytes, peak memory, serial decode depth, or number of artifacts into FLOPs. Those remain separate coordinates.

Wall time is different again. If $G_m$ is the measured execution DAG and $w(v)$ is the observed duration of node $v$, then

$$W_m = \operatorname{CriticalPath}(G_m,w),$$

not the sum of all device FLOPs. Overlap, utilization, communication, input stalls, and synchronization can change $W_m$ without changing logical work. Consequently, same wall time and same compute are different controls.

0x0AF Five Common Denominators Answer Five Different Questions

Frozen denominator	What may vary	Question answered	Invalid conclusion
student updates or processed tokens	teacher work, wall time, storage	does added supervision help after the same number of student updates?	that the methods used equal total resources
unique examples	repetitions, updates, compute	which method extracts more from the same distinct data?	that each method had equal optimization opportunity
logical compute	processed tokens, updates, wall time	which method uses a fixed modeled FLOP allowance better?	that measured hardware time was equal
measured wall time on frozen hardware/software	tokens, updates, logical work	what fits into an operational time window on that stack?	that algorithmic compute was equal or portable
teacher calls or paid API tokens	student-only work, cached reuse, wall time	how much value is obtained from scarce teacher access?	that a no-teacher baseline is bounded by the same resource

Storage, peak memory, serial decode depth, hyperparameter search, and number of training stages can also be hard constraints. A candidate is feasible only if every declared resource fits:

$$\operatorname{feasible}(m;\mathbf{B}) = \bigwedge_j r_{m,j}\le B_j.$$

There is no requirement that one scalar rank all resources. In fact, a method can be compute-feasible and storage-infeasible, or fit total FLOPs while missing a wall-time deadline because generation is serial.

For a scalar additive budget $B_C$, if one update costs $c_m$, the maximum whole-update count is

$$U_m^{(C)} = \left\lfloor\frac{B_C}{c_m}\right\rfloor.$$

For a vector budget with additive per-update resource $r_{m,j}>0$,

$$U_m^{(\mathbf{B})} = \min_j \left\lfloor\frac{B_j-r^{once}_{m,j}}{r^{update}_{m,j}}\right\rfloor,$$

provided every one-time requirement $r^{once}_{m,j}$ fits. The minimum appears because the first exhausted resource stops the run.

0x0AG Fixed-Compute Evidence Is A Protocol, Not A Universal Ranking

Bui et al.’s revision-pinned study explicitly compares no-KD and several KD strategies with the same target architecture and a fixed wall-clock pretraining window. It also separates an unlimited-data regime from a limited-data regime where examples repeat. The paper reports that no-KD processes more tokens because it omits the teacher pass, and it scopes its conclusions to the studied encoder models, data, hardware, and protocol (arXiv:2404.19319v1).

The accompanying official repository is pinned here at commit dcc7b1d4abd071cde0ac7265dbb0ec03d95ddc8c. The code trace exposes the protocol rather than merely repeating the abstract:

Pinned code path	Observed contract
pretraining/schedules.py:90-104	TimeScheduler maps elapsed hours, not update count, onto the learning-rate schedule
pretraining/args/pretraining_args.py:124-135	total and early-exit training time are explicit run state
slurm/.../distillbert_6layer_1epoch.sh:46-55	teacher path, four-hour budget, architecture, method, and checkpoint time markers are configured together

This is strong evidence that time budget can be made executable. It does not prove that wall-clock matching is sufficient in every system: even the paper notes that wall time can vary, and its result does not cover decoder-only on-policy generation, offline target production, paid APIs, or heterogeneous clusters.

0x0AH Processed Tokens, Unique Data, And Repetition Must Be Separate

Let $U_m$ be student updates, $V_m$ valid rows per update, and $D$ the number of distinct eligible rows in the frozen corpus. Then

$$N_{processed,m}=U_mV_m,$$ $$N_{unique,m}=\min(N_{processed,m},D),$$

and the exposure ratio is

$$\rho_m=\frac{N_{processed,m}}{N_{unique,m}}.$$

If no-KD processes more rows under a time budget, it may see more unique data when $N_{processed}<D$. Once the corpus is exhausted, additional throughput increases repetition instead. A comparison that reports only processed tokens cannot distinguish those regimes.

The data identity also includes filtering, ordering, corruption, packing, and lineage-safe split rules from Loop 8. Two methods that read the same nominal corpus but apply different teacher-dependent filtering do not have the same training data.

0x0AI A Reproducible Toy Ledger Exposes The Trade-Offs

The local oracle uses intentionally artificial work units. It sets one student forward to 1 unit, one student backward to 2 units, one teacher score forward to 4 units, and declares separate teacher-decode, storage, peak-memory, and run-count coordinates. The values teach accounting; they are not measurements of any paper implementation.

With compute budget $B_C=8{,}400$, teacher-work budget $B_T=4{,}800$, serial-decode budget $B_D=1{,}800$, storage budget 32 GiB, and peak-memory budget 80 GiB, the fixture produces:

Candidate	Work/update	Updates under compute only	Updates under full vector budget	Binding resource or interpretation
no-KD	3.00	2,800	2,800	compute; no teacher coordinate is consumed
online dense token KD	7.00	1,200	1,200	compute and teacher work meet together
offline dense KD, reuse 4	4.00 amortized	2,100	0	64 GiB one-time target artifact exceeds storage budget
offline sampled sparse KD, reuse 4	4.00 amortized	2,100	2,100	compute; sparse artifact fits at 8 GiB
online hidden/relation KD	7.50	1,120	1,120	compute; added alignment work remains explicit
offline sequence KD	4.25 amortized	1,976	1,440	serial teacher decode is the binding coordinate
on-policy GKD	8.50	988	988	compute; student generation, teacher score, and rescore are all charged
teacher-assistant chain	12.00	700	700	two training runs and an assistant artifact are part of the method

The same-update comparison is different. At 1,000 updates, no-KD costs 3,000 units, online token KD costs 7,000, on-policy GKD costs 8,500, and the teacher-assistant chain costs 12,000. Equal update count is therefore not equal compute in this fixture.

The oracle also computes unique-row exposure with a 1.5-million-row corpus. A candidate can process more rows yet gain no new unique rows after the cap; its $\rho_m$ increases instead. This is why compute, data, and repetition must be reported together.

0x0AJ The Controlled Pair Must Freeze More Than The Student Shape

A controlled method comparison requires one immutable manifest shared by every candidate:

release target:
    student graph, parameter/storage identities, tokenizer, precision, context

initial state:
    initialization class, checkpoint digest, random-seed policy

training rows:
    source revision, split lineage, filtering, corruption, packing, masks

optimization:
    valid-row reduction, optimizer, schedule coordinate, clipping, updates

resource denominator:
    one primary budget plus every hard secondary limit

search allowance:
    candidate count, pilot tokens, failed runs, seed count, selection rule

evaluation:
    frozen tasks, prompts, metrics, decoding, hardware, software, repetitions

The no-KD control must use the same deployable student, not merely a model with a similar parameter count. Otherwise architecture and supervision change at the same time. If a paper method requires a distinct architecture, report that as a separate pipeline comparison and do not call the difference a pure KD effect.

For quality functional $Q$ under protocol $\Pi$, the distillation effect for a fixed student is

$$\Delta_{KD}(m;\Pi) = Q(S_{m};\Pi)-Q(S_{noKD};\Pi).$$

The notation keeps the protocol argument because changing the compute budget, data regime, initialization, or evaluation creates a different estimand. Loop 11 will own uncertainty, seed aggregation, metric direction, and promotion rules. Loop 10 only makes the pair admissible.

0x0AK Method Composition Adds Costs And Provenance

Methods can be composed, but their contracts do not disappear. A plausible pipeline is:

phase 0: choose or prune the student graph       -> S0
phase 1: transfer general representations       -> S1 + alignment state
phase 2: train on versioned teacher sequences   -> S2
phase 3: refine on student prefixes              -> S3 + rollout lineage
release: remove teacher-only state               -> deployable student bundle

For additive resources such as logical compute,

$$C_{pipeline}=\sum_{i=0}^{K}C_i.$$

For sequential phases whose live tensors do not overlap,

$$M_{peak,pipeline}=\max_i M_{peak,i},$$

whereas overlapping producer and consumer phases require an interval-liveness calculation from Loop 9. Persistent storage is the content-addressed union of artifacts rather than a blind sum:

$$A_{pipeline}=\left|\bigcup_i \mathcal{A}_i\right|_{bytes}.$$

Wall time is the critical path of the phase DAG. It is a sum only when every phase is serial and no production overlaps consumption.

A projection learned against one teacher revision cannot silently survive a teacher change. Teacher-generated sequences from phase 2 are not on-policy for $S_2$ or $S_3$. An assistant checkpoint is not free because it is absent from the final bundle. Composition adds identities, costs, and rollback points.

0x0AL Comparison Admission And Pareto Reporting

The comparison harness rejects undefined pairs before looking at quality:

def admit_comparison(shared, candidates, budgets):
    assert shared.same_release_student
    assert shared.same_tokenizer_and_context
    assert shared.same_split_lineage
    assert shared.same_evaluation_contract
    assert shared.primary_budget_coordinate in budgets
    assert shared.search_budget_is_counted

    reports = []
    for candidate in candidates:
        contract = validate_eight_coordinate_contract(candidate)
        events = count_runtime_events(contract)
        resources = price_declared_coordinates(events, shared.unit_costs)
        feasible = all(resources[name] <= budgets[name] for name in budgets)
        reports.append((contract, resources, feasible))

    return pareto_front(
        reports,
        minimize=["compute", "wall_time", "teacher_calls", "storage", "peak_memory"],
        maximize=["quality"],
    )

pareto_front is deliberate. A single score requires exchange rates between quality, teacher access, storage, latency, and engineering risk. Unless those weights are declared by the deployment owner, the honest result is a frontier, not a universal winner.

0x0AM Evidence Does Not Transfer Between Paper Contracts

The 15 core papers establish results in different regimes. DistilBERT, Patient KD, TinyBERT, and MiniLM study BERT-family encoders or encoder-style pretraining. Sequence-Level KD was developed for sequence generation. MiniLLM and GKD study autoregressive language-model distillation. Teacher Assistant KD’s original capacity-gap evidence is from image classification. Minitron combines pruning and recovery. Sparse Logit Sampling studies an estimator and storage path.

Therefore:

• an encoder result does not prove the same layer map is optimal in a causal decoder;
• a source-reported speedup includes one graph, runtime, hardware, workload, and denominator;
• an assistant that helps one capacity path does not establish that every extra stage helps;
• a divergence result under one prefix occupancy does not make prefix choice irrelevant;
• sparse-target accuracy does not transfer to a different support estimator;
• implementation parallelism is not a mathematical property of the objective;
• a pruned-and-recovered student is not a controlled test of KD alone.

0x0AN Paper-By-Contract Source Trace

The executable comparison manifest binds the complete core paper corpus:

1312.6184v7  Ba and Caruana function matching
1412.6550v4  FitNets
1503.02531v1 Hinton et al. classical KD
1606.07947v4 Sequence-Level KD
1612.03928v3 Attention Transfer
1908.09355v1 Patient KD
1909.10351v5 TinyBERT
1910.01108v4 DistilBERT
1902.03393v2 Teacher Assistant KD
2002.10957v2 MiniLM
2012.15828v2 MiniLMv2
2306.08543v6 MiniLLM
2306.13649v3 GKD
2407.14679v2 Minitron
2503.16870v2 Sparse Logit Sampling

Loop 10 adds 2404.19319v1 for the fixed-computation comparison protocol and pins its author repository at dcc7b1d4abd071cde0ac7265dbb0ec03d95ddc8c. All benchmark values remain source-reported. The local comparison fixture contains no copied leaderboard scores and cannot validate paper quality claims.

0x0AO Comparison Failure Lab

Failure	Broken invariant	Consequence	Recovery owner
same updates labeled equal compute	event vectors differ	teacher-heavy methods receive extra work	Loop 10 budget manifest
same wall time inferred from FLOPs	critical path was not measured	overlap and utilization disappear	Loop 9 runtime plus Loop 10 protocol
same nominal corpus after different filtering	row lineage differs	data effect is attributed to KD	Loop 8 artifact manifest
no-KD control uses another architecture	release student differs	architecture and transfer are confounded	Loop 5 construction manifest
top-$k$ renormalization labeled Sparse Logit Sampling	estimator identity differs	gradient claim is false	Loop 8 sparse-target contract
MiniLM row labeled hidden-state MSE	matched object differs	relation geometry disappears	Loop 4 target contract
sequence KD labeled on-policy	prefix producer differs	exposure-bias claim is unsupported	Loops 6 and 7
GKD omits student rescore	sampled IDs are treated as differentiable	wrong gradient path	Loop 7 three-pass trace
assistant checkpoint omitted from cost	extra run/artifact hidden	capacity-path method appears free	Loops 5 and 10
offline target production omitted	teacher work moved outside window	online/offline comparison is biased	Loops 8 and 9
hyperparameter trials omitted	search resources differ	selected result receives hidden budget	Loop 10 manifest; Loop 11 statistics
one source-reported result generalized across regimes	evidence coordinates differ	paper claim becomes universalized	Loop 10 evidence firewall
composed phases sum peak memory	liveness, not addition, defines peak	capacity estimate is wrong	Loop 9 interval ledger
one weighted score has undeclared exchange rates	preferences are hidden	apparent winner is policy-dependent	Loop 10 Pareto report

0x0AP Executable Closure And Boundary

The Loop 10 oracle validates 15 unique paper contracts, eight method coordinates, eight toy candidates, scalar and vector budget arithmetic, unique-data exposure, composition accounting, comparison admission, and claim ceilings. It rejects one-field mutations in source revisions, method coordinates, resource events, budgets, shared controls, and evidence scope.

What it proves is intentionally bounded:

proved locally:
    immutable paper-coordinate rows
    no duplicate full method contracts
    event-to-resource arithmetic
    scalar and vector budget admission
    processed-versus-unique row accounting
    additive/peak/union/critical-path composition rules

not proved:
    any paper benchmark
    any trained student quality
    any real GPU throughput or wall time
    any universal ranking of distillation methods
    any deployment promotion decision

Loop 11 can now evaluate candidates whose identities and budgets are actually comparable. 0x0B can choose a strategy without pretending that a paper name determines the target, data, runtime, cost, and release artifact by itself.

0x0B Strategy Admission Begins With A Release Identity

Strategy selection begins with the deployable student, not with a favorite paper. A method name such as MiniLM, sequence KD, or GKD does not identify the student graph, teacher interface, data occupancy, runtime, cost, or release artifact. Before selecting a target, freeze the student release tuple:

student release identity:
    architecture/config digest
    parameter and storage identities
    tokenizer and canonical token support digest
    context limit and generation interface
    serving precision and runtime revision
    target hardware and workload matrix
    latency, throughput, memory and quality gates

The candidate set then needs at least three distinct identities:

1. $S_0$: the same student graph trained without transferred supervision;
2. $S_{KD}$: that graph under the proposed transfer contract;
3. $T$: the teacher, reported as a quality and cost reference rather than as the control that isolates transfer value.

Comparing $S_{KD}$ only with $T$ answers how much capability remains below the teacher. It does not answer whether KD helped the chosen student. Comparing $S_{KD}$ with a different small architecture confounds architecture selection with transfer. Loop 10 admitted only candidates whose identities and budgets are comparable; Loop 11 decides whether one of those candidates has earned a student-only release artifact.

0x0BA Teacher Access Defines The Feasible Target Set

The teacher interface is a hard boundary, not an implementation convenience.

Available interface	Feasible supervision	What remains unavailable
generated text only	hard sequence artifacts, teacher-side filtering or reranking over returned text	exact token distributions and internal representations
sampled tokens plus sampled-token log-probabilities	sampled-action objectives and explicit sequence estimators	dense forward KL without another estimator
sparse top-$k$ logits plus tail metadata	the declared sparse or sampled estimator	exact dense KL by merely renormalizing retained entries
full logits on fixed rows	token KD, forward/reverse token divergences, offline dense caches	hidden, attention, and value relations
selected hidden, attention, or QKV states	intermediate and relational transfer in addition to output KD	any unstored layer, head, row, or tensor
live teacher on student trajectories	on-policy token or sequence training	deployment independence unless export proves it separately

If the API returns text only, there is no hidden-state matching problem to solve. If teacher and student token supports differ, same-index KL is undefined even when both responses are arrays called logits. A sequence objective or an explicit transport between token supports may be valid; a shape cast is not.

0x0BB Compatibility Chooses Between Raw States, Relations, And Outputs

Raw hidden matching is easiest to define when token rows correspond, mapped layers have a defensible role, and width can be aligned by a controlled projection. Relation matching can retain token-pair geometry when widths or head counts differ. Output or sequence transfer is usually the cleaner contract when architectures differ radically or internal access is absent.

More targets are not monotonically better. An intermediate objective can add teacher compute, activation retention, collectives, projections, fragile layer maps, and optimization conflict. It earns inclusion only if the same-student ablation later shows value under the same primary budget.

Constraint	Initial design	Escalate only when
shared tokenizer, full logits, modest teacher cost	masked token KD on fixed examples	self-generated trajectory failures remain
compatible stacks and internal access	output KD plus one justified state or relation target	a paired ablation supports the extra target
black-box text teacher	versioned sequence corpus	generation coverage, filtering and decoding-policy bias are measured
errors compound after student mistakes	fixed/student-prefix mixture	online generation and teacher scoring fit the vector budget
large capacity gap	repair temperature, data and optimization first	equal-budget direct KD remains optimization-limited
dense targets exceed storage	chunking, lower precision, or a verified sparse estimator	approximation error is checked against dense targets on an auditable subset
pruned or quantized student needs recovery	keep structural or numerical recipe fixed, then add scoped KD	same-artifact hard-label recovery is an insufficient baseline

0x0BC The Design Procedure Returns Contracts, Not A Paper Name

def design_distillation(requirements, teacher, student, budget):
    assert student.release_identity == requirements.student_release_identity
    assert requirements.has_same_architecture_no_kd_control

    plan = {
        "matched_objects": [],
        "prefix_distribution": None,
        "teacher_interface": teacher.interface,
        "execution_mode": None,
        "persistent_artifacts": [],
        "evaluation_manifest": requirements.frozen_evaluation_manifest,
    }

    if teacher.interface == "text_only":
        plan["matched_objects"] = ["teacher_decoded_sequence"]
        plan["prefix_distribution"] = "source_plus_teacher_target"
        plan["execution_mode"] = "versioned_offline_sequence_corpus"
        plan["persistent_artifacts"] = ["sequence_manifest", "filter_lineage"]
        return validate_plan(plan, budget)

    if teacher.token_support_hash != student.token_support_hash:
        assert requirements.has_explicit_cross_tokenizer_objective

    if teacher.has_full_logits:
        plan["matched_objects"].append("token_distribution")
        plan["prefix_distribution"] = "fixed_examples"
        plan["execution_mode"] = choose_online_offline_or_hybrid(budget)

    if requirements.needs_intermediate_transfer:
        target = choose_hidden_attention_or_relation_target(teacher, student)
        assert target.has_layer_width_head_row_and_mask_contract
        plan["matched_objects"].append(target)

    if requirements.needs_student_prefix_coverage:
        assert teacher.can_score_student_trajectories
        plan["prefix_distribution"] = requirements.frozen_prefix_mixture
        plan["execution_mode"] = "generate_teacher_score_student_rescore"

    return validate_plan(plan, budget)

The result is a testable design, not evidence that the design is useful. Selection still requires the same student, immutable evaluation rows, a fixed budget, paired uncertainty, runtime gates, and export closure.

0x0BD Transfer Cost And Student-Only Deployment Cost Are Different Ledgers

An expensive transfer can be rational when the student is served many times, but hidden production work remains part of the method:

$$C_{\mathrm{transfer}} =C_{\mathrm{teacher\ generation}} +C_{\mathrm{teacher\ scoring}} +C_{\mathrm{student\ training}} +C_{\mathrm{artifact\ I/O}} +C_{\mathrm{search}} +C_{\mathrm{evaluation}}.$$

The deployed path has another denominator:

$$C_{\mathrm{deploy}} =C_{\mathrm{student\ load}} +N_{\mathrm{requests}}C_{\mathrm{student\ inference}} +C_{\mathrm{student\ operations}}.$$

Counting the teacher in every future request understates the benefit of a teacher-independent student. Omitting target generation, assistants, failed search runs, and rollout production understates the cost of creating it. A term is zero only when the selected execution graph genuinely omits that event.

0x0BE The Evaluation Manifest Is Executable State

An evaluation result is meaningful only under an immutable protocol. Define

$$\Pi_{eval} =\left( h_{task},h_{data},h_{split},h_{prompt},h_{fewshot},h_{tokenizer}, G,\mathcal M,\mathcal B,H,R \right),$$

where $G$ is generation configuration, $\mathcal M$ metric definitions and directions, $\mathcal B$ case/seed blocking, $H$ hardware, and $R$ runtime. Every digest belongs in the result record. A task name alone does not identify the dataset revision, prompt, few-shot examples, filters, metric code, or generation policy.

The official lm-evaluation-harness task guide explicitly treats task YAML plus the code commit as shareable evaluation state. This article pins task_guide.md and evaluator.py at commit e90e5b656bcc91559f1f2928c431ef4801f15804. This is evidence for versioned configuration and result metadata, not a claim that one harness covers every safety, human, code, or long-generation evaluation.

def freeze_evaluation(spec):
    manifest = canonicalize({
        "task_revision": spec.task_revision,
        "dataset_revision": spec.dataset_revision,
        "split_hash": hash_rows(spec.case_ids),
        "prompt_hash": sha256(spec.prompt_template),
        "fewshot_ids": spec.fewshot_ids,
        "tokenizer_hash": spec.tokenizer_hash,
        "generation": spec.generation,
        "metric_versions": spec.metric_versions,
        "metric_directions": spec.metric_directions,
        "block_ids": spec.case_seed_block_ids,
        "hardware_runtime": spec.hardware_runtime,
    })
    assert lineage(manifest.test_rows).isdisjoint(training_and_target_rows)
    return sha256(manifest), manifest

0x0BF Leakage Is A Lineage Failure, Not Just An Exact String Match

Evaluation rows must be disjoint from student training data, teacher-target production rows, assistant training data, prompt-selection data, and hyperparameter-selection data. Exact row IDs are the first check; near duplicates, benchmark-derived synthetic rows, translated copies, and teacher artifacts generated from held-out prompts require lineage-aware exclusion.

The order matters:

training/search data -> candidate selection -> frozen validation decision
                    -> untouched final test -> release report

If the final test chooses temperature, layer map, prefix mixture, checkpoint, or decoding policy, it has become selection data. A content digest cannot undo that conceptual leak. The manifest must record which decisions each split was allowed to influence.

0x0C Paired Evaluation, Uncertainty, And Conjunctive Admission

Distillation creates two different estimands:

$$\Delta_{transfer}=M(S_{KD};\Pi_{eval})-M(S_0;\Pi_{eval}),$$ $$\Delta_{teacher}=M(S_{KD};\Pi_{eval})-M(T;\Pi_{eval}).$$

The first isolates transfer for a fixed student. The second reports the remaining teacher gap. Neither substitutes for the other. For a lower-is- better metric such as NLL or latency, reversing the sign informally is error prone, so normalize every metric to an improvement coordinate.

0x0CA Metric Direction Is Part Of The Formula

Let $s_m=+1$ when metric $m$ is maximized and $s_m=-1$ when it is minimized. For paired block $i$,

$$d_{i,m} =s_m\left(y^{KD}_{i,m}-y^{0}_{i,m}\right).$$

Now positive always means that KD improved the same student. The teacher gap uses the same orientation:

$$g_{i,m} =s_m\left(y^{KD}_{i,m}-y^{T}_{i,m}\right).$$

The block can be one test case, prompt, user, document, conversation, random seed, or a declared composite. It must be the same block for both candidates. Shuffling one candidate’s rows preserves both marginal means while destroying the paired estimand.

Peyrard et al. show why independent averages can hide the joint structure of scores observed on the same examples. Their paper is used here to motivate preserving pairs; it does not prescribe this article’s release gate (ACL Anthology paper and source Figure 2).

0x0CB Deriving The Paired Bootstrap Gate

The point estimate is

$$\bar d_m=\frac{1}{n}\sum_{i=1}^{n}d_{i,m}.$$

Resampling candidates independently would erase their shared case difficulty. Instead sample block indices $I_b=(i_1,\ldots,i_n)$ with replacement and carry the already-computed paired differences together:

$$\bar d_m^{*(b)} =\frac{1}{n}\sum_{j=1}^{n}d_{i_j,m}.$$

For a percentile interval with two-sided error rate $\alpha$,

$$CI_m= \left[ Q_{\alpha/2}\left(\{\bar d_m^{*(b)}\}\right), Q_{1-\alpha/2}\left(\{\bar d_m^{*(b)}\}\right) \right].$$

The local oracle uses only six synthetic blocks, so it enumerates all $6^6=46{,}656$ bootstrap index vectors exactly rather than adding Monte Carlo noise. This is a teaching choice, not a recommendation to evaluate a real model on six prompts.

For a non-inferiority margin $\delta_m\ge 0$, admit metric $m$ only when

$$LCB_m\ge-\delta_m.$$

A primary transfer metric can require $LCB_m>0$; a safety or long-generation guardrail may allow a predeclared small regression. The margin is product policy fixed before observing the candidate, not a value chosen to make the candidate pass.

The synthetic fixture produces:

Metric	Direction	Mean oriented transfer delta	95% percentile interval	Gate
task score	maximize	$+0.0217$	$[+0.0167,+0.0267]$	primary $LCB>0$ passes
NLL	minimize	$+0.0983$	$[+0.0750,+0.1183]$	guardrail passes
safety score	maximize	$+0.0067$	$[+0.0033,+0.0100]$	guardrail passes
long-generation score	maximize	$+0.0083$	approximately $[0,+0.0167]$	guardrail passes

These numbers are generated data. They validate code paths and signs, not KD quality.

0x0CC Bootstrap, Paired Tests, And Hierarchical Blocks Are Alternatives

No statistical procedure is universal:

Procedure	Useful when	Main contract or failure
paired mean and paired $t$ interval	block differences are an appropriate mean estimand and sample size supports the approximation	sensitive to heavy tails and dependence
paired percentile or BCa bootstrap	the statistic can be recomputed under block resampling	resampling unit must match dependence; small $n$ remains small $n$
sign or signed-rank family	direction or rank shift matters more than magnitude	answers a different estimand and carries its own assumptions
pairwise win probability or Bradley-Terry family	per-instance relative outcomes are primary	does not preserve the meaning of every task-native score automatically
hierarchical bootstrap	prompts nest in documents, users, tasks, or seeds	hierarchy must reflect the data-generating and deployment unit
multi-seed paired analysis	training randomness is material	three favorable seeds are not a population proof

Case resampling estimates evaluation-set uncertainty conditional on trained weights. Seed resampling estimates training-run uncertainty. Treating hundreds of prompts from one seed as hundreds of independent training replications is pseudoreplication. If both sources matter, report both levels or use a declared hierarchical design.

0x0CD Quality Is A Vector And Admission Is A Conjunction

Teacher agreement measures imitation, not correctness. Teacher and student can share the same error, so release evaluation needs labels, references, constraints, human judgments, or independent evaluators whose provenance is not the distilled teacher itself.

Surface	Required evidence	Hidden failure in one aggregate
token prediction	held-out NLL/perplexity under one tokenizer and reduction contract	rare-language or long-context regressions disappear in the mean
task quality	task-native exact match, F1, pass rate, rubric or preference with metric version	teacher agreement can increase while correctness decreases
calibration	NLL, Brier score, reliability curve or precisely defined ECE	better top-line quality need not mean calibrated confidence
long generation	quality by output length, repetition, EOS, constraints and recovery after early mistakes	fixed-prefix loss does not exercise student trajectories
capability slices	language, domain, prompt form, context length, code/tool mode	capacity can be reallocated between slices
safety/refusal	harmful-completion, jailbreak, over-refusal and policy slices	KD can transfer both desired and undesired behavior

Let $G_Q$, $G_R$, $G_A$, and $G_E$ be quality, runtime, artifact, and export gates. Admission is

$$G_{release}=G_Q\land G_R\land G_A\land G_E.$$

An excellent average task score cannot compensate for a failed safety floor, teacher dependency, tokenizer mismatch, or latency ceiling. Pareto reporting can rank candidates that pass hard gates; a weighted score must not turn a failed invariant into a tradeable preference.

0x0CE Runtime Evaluation Is A Student-Only Experiment

The release benchmark starts in a fresh process that cannot resolve teacher weights, teacher APIs, target caches, rollout workers, assistants, or training-only alignment modules. The measured identity is

$$I_{run}= (h_{weights},h_{config},h_{tokenizer},h_{runtime},H,h_{workload}).$$

Record checkpoint/config bytes, cold load, resident and peak memory, prompt and output length distributions, concurrency, batching, warmup, repetitions, prefill/decode timing, throughput, kernel selection, and fallbacks. The same candidate under another runtime or workload is another experiment.

def benchmark_release(bundle, workload):
    process = fresh_process(network_policy="deny_teacher_services")
    model = process.load_student_only(bundle)
    assert model.weights_hash == bundle.candidate_weights_hash
    assert process.loaded_components == {
        "student_model", "student_tokenizer", "runtime_kernel"
    }

    warmup(model, workload.warmup_cases)
    raw = [run(model, workload) for _ in range(workload.repetitions)]
    return aggregate_without_discarding_raw_samples(raw)

The teacher and student should share a controlled protocol when reporting a deployment gap, but each may also receive an operationally tuned batch size. Report the controlled and tuned comparisons separately. A paper’s speedup does not transfer across precision, hardware, runtime, prompt/output lengths, or concurrency without remeasurement.

0x0D Attribution, Teacher-Independent Export, And Failure Closure

Evaluation says whether the candidate passed. Instrumentation and ablation say which contract plausibly produced the change. Export then proves that the measured candidate became a self-contained student artifact.

0x0DA Distributed Telemetry Preserves Numerators And Denominators

A scalar loss cannot diagnose a multi-target run. Every component keeps its population:

hard CE: numerator / valid causal rows
token KD: numerator / valid shared-support rows
hidden target: numerator / valid mapped token-feature elements
attention or relation target: numerator / valid mapped pairs
sequence estimator: weighted return terms / valid response actions
runtime: teacher, generation, transport, cache wait, student fwd/bwd
correctness: teacher gradient tensors, padding contribution, dropped rows

For rank $r$ with numerator $N_r$ and denominator $D_r$, the global objective is

$$L_{global}=\frac{\sum_rN_r}{\sum_rD_r},$$

not

$$\frac{1}{R}\sum_r\frac{N_r}{D_r}.$$

The two are equal only under special denominator symmetry. The fixture uses 40 and 60 valid rows across two ranks and recovers hard CE $196/100=1.96$ and token KD $33/100=0.33$. It also requires zero teacher-gradient tensors and zero padding-loss numerator.

0x0DB A Factorial Ablation Exposes Interaction

One-at-a-time ablations can miss interactions between targets. For token KD $A\in\{0,1\}$ and hidden transfer $B\in\{0,1\}$, freeze architecture, initialization policy, data, optimizer, budget, seeds, and evaluation. The interaction is

$$I_{AB}=Q_{11}-Q_{10}-Q_{01}+Q_{00}.$$

In the synthetic fixture,

$$Q_{00}=0.700,\quad Q_{10}=0.725,\quad Q_{01}=0.711,\quad Q_{11}=0.742,$$

so the token effect at $B=0$ is $0.025$, the hidden effect at $A=0$ is $0.011$, and $I_{AB}=0.006$. The arithmetic shows that combined gain is not the sum of isolated gains. It does not prove a causal mechanism: hidden hyperparameter search, seed selection, compute drift, or data differences would invalidate that interpretation.

Other questions require their own controlled pair: direct versus assistant path including both runs; fixed versus student prefixes at matched resource budget; forward versus reverse KL on identical rows and estimator budget; dense versus sparse targets on a dense-audited subset; and pre-export versus exported student under the same runtime input fixture.

0x0DC A Release Bundle Is A Dependency Closure

The final training checkpoint is not the release artifact. Define the required student set

$$\mathcal A_{student} =\{config,weights,tokenizer,generation,documentation,evaluation,checksums,release\},$$

and the forbidden training set

$$\mathcal A_{train} =\{teacher,assistant,alignment,optimizer,scheduler,scaler, target\ cache,rollout\ cache,reader\ cursor\}.$$

A teacher-independent bundle requires

$$\mathcal A_{release}\supseteq\mathcal A_{student}, \qquad \mathcal A_{release}\cap\mathcal A_{train}=\varnothing.$$

Names alone are insufficient. Starting from the serving entrypoint, compute the reachable dependency graph $Reach(entry)$. No reachable node, import, credential, endpoint, tensor key, or lazy loader may require teacher-side state. An unused teacher file is still a packaging defect; an unbundled but reachable teacher API is a runtime defect.

The pinned Safetensors repository documents a tensor-only header/data format and forbids arbitrary executable payload in the tensor file itself (format at commit 6eb4dc9...). That narrows one serialization surface; it does not validate tensor meaning, model configuration, tokenizer compatibility, dependencies, or release provenance. Those remain manifest checks.

0x0DD Content Addressing And Export Parity Bind The Candidate

For canonical identity $I$ and sorted file records $(p_j,h_j)$, define

$$h_{bundle} =SHA256\left(canonical\left(I,\{(p_j,h_j)\}_{j=1}^{J}\right)\right).$$

The release identity binds student weights, config, tokenizer, evaluation manifest, generation policy, runtime compatibility, limitations, and rollback predecessor. A mutable directory name or latest tag is an alias, not the identity.

Export equivalence is another gate. For fixed parity inputs $x_i$ and chosen comparison function $z$,

$$\epsilon_{export} =\max_i\left\|z(f_{pre}(x_i))-z(f_{export}(x_i))\right\|,$$

and the exporter passes only if $\epsilon_{export}\le\epsilon_{tol}$ under a declared dtype and comparison surface. Token equality under greedy decoding is useful but can hide logit drift until a near-tie changes. Logit or hidden parity is stronger when the runtime exposes it; end-to-end generated parity is still needed at the actual interface.

def audit_student_bundle(bundle, expected):
    verify_every_file_digest(bundle)
    assert bundle.identity.weights_hash == expected.candidate_weights_hash
    assert bundle.identity.eval_manifest_hash == expected.eval_manifest_hash
    assert bundle.identity.tokenizer_hash == expected.tokenizer_hash
    assert required_paths(bundle) == expected.student_release_paths
    assert not forbidden_paths_tensor_keys_or_dependencies(bundle)

    graph = resolve_runtime_dependency_graph(bundle.entrypoint)
    assert not graph.reachable_nodes.intersection(expected.teacher_side_nodes)

    model = load_in_fresh_student_only_process(bundle)
    error = compare_pre_export_and_export(model, expected.parity_inputs)
    assert error <= expected.parity_tolerance
    return content_addressed_manifest(bundle)

0x0DE Local Promotion Is Not Production Rollout

Loop 11 owns a local, content-addressed state machine:

candidate -> evaluated -> bundled -> admitted to release registry

Every transition is monotonic and writes evidence before moving the registry pointer. The previous admitted student remains a valid predecessor. A failed evaluation or partial bundle never mutates the admitted pointer. Rolling back this local decision means restoring the pointer to the predecessor bundle and quarantining the failed identity.

This is not a canary, traffic ramp, SLO monitor, or production rollback. Those require stateful routing, live workload measurement, observability and recovery policy. M8.10 owns their whole-tuple release gate when this student is composed with a quantized/native low-bit deployment; M8.9 admits that representation. M6.10 retains the exact teacher-independent bundle and its local proof. Generic nonquantized production integration remains a separate, unassigned program boundary, not an implicit expansion of M8 ownership. M8.10’s release and rollback procedure is the closest material in this series; its low-bit parts do not apply.

The documentation boundary follows the same principle. Model Cards proposes reporting intended uses, evaluation procedures, performance conditions and limitations with released models (Mitchell et al.). A model card is necessary release context, not a cryptographic identity or substitute for raw evidence.

0x0DF Executable Evaluation And Release Oracle

The local Loop 11 oracle performs the following synthetic transaction:

1. validate pinned source and evaluation manifests
2. prove no-KD and KD candidates share the controlled student identity
3. preserve six paired blocks across four directed metrics
4. enumerate 46,656 exact paired bootstrap resamples per metric
5. apply primary and guardrail lower-confidence-bound gates
6. apply student-only latency, throughput, memory and load gates
7. globally reduce objective numerators and denominators
8. compute a two-factor ablation interaction
9. validate exactly nine release files and their digests
10. prove no teacher/training state is bundled or runtime-reachable
11. verify export parity and rollback predecessor identity
12. reject 26 semantic fault injections

The accepted toy candidate has task-score transfer $0.0217$, NLL improvement $0.0983$, a nine-file bundle digest, and maximum synthetic export error near $10^{-7}$. None of those values came from a trained model or real device.

0x0DG Cross-Contract Failure Lab

Failure	Why a shallow check passes	Observable consequence	Fail-closed repair
KD compared only with teacher	teacher gap is a real number	architecture and transfer effects remain confounded	require the same-architecture no-KD control
final test tunes the KD recipe	split names still look correct	optimistic release estimate	freeze decision rights and use untouched final test
metric direction omitted	score is finite	lower NLL appears as a regression or latency as an improvement	store direction and orient every paired delta
candidate rows shuffled independently	marginal means remain unchanged	pairing and uncertainty are wrong	bind stable block IDs before aggregation
prompts from one seed treated as independent runs	sample count looks large	training uncertainty is understated	separate case and seed levels
one aggregate passes	average improves	safety, long-generation or language slice regresses	conjunctive bucketed gates
teacher agreement used as correctness	imitation improves	shared teacher errors are rewarded	use independent task/reference evidence
per-rank means averaged	all ranks report a scalar	objective changes with padding and world size	reduce numerators and denominators globally
one-at-a-time ablation only	isolated effects look additive	target interaction is hidden	include factorial cells or state interaction is unmeasured
teacher weights omitted but endpoint remains	bundle bytes look small	serving still depends on teacher availability	traverse runtime dependency graph under denied teacher access
alignment projection ships accidentally	state dict loads	claimed student bytes and graph are wrong	allowlist student keys and deny training-only state
required projection is deleted	package is smaller	released function is incomplete	classify inference-required modules as student graph before costing
tokenizer changes after evaluation	model weights are unchanged	prompt rows and generation semantics drift	bind tokenizer digest into candidate, evaluation and release identities
bundle digest excludes metadata	weight hash matches	config or generation policy mutates silently	digest canonical identity plus every file record
export parity checks only one token	smoke test passes	later logits or generation diverge	use fixed multi-surface parity fixtures
failed candidate overwrites release pointer	publication job partially succeeds	no known-good local predecessor remains	write immutable bundle first, then atomic pointer update
source-reported speedup becomes release expectation	citation is valid	target runtime misses its workload gate	remeasure exact student/runtime/workload identity
local admission called production deployment	registry state is real	live traffic, state and rollback risks are unevaluated	preserve local proof; low-bit composition enters M8.9/M8.10, other production integration remains separate

A diagnostic should proceed from semantic identity to rows, objective, statistics, runtime, and release. Optimizer tuning cannot repair a mismatched token support, leaked split, destroyed pair, reversed metric, or teacher- dependent artifact.

0x0DH Loop 11 Claim Ceiling And Boundary

The loop proves locally:

proved by synthetic executable evidence:
    manifest hashing and split-lineage rejection
    controlled same-student identity
    metric-direction normalization
    exact paired-bootstrap arithmetic on six toy blocks
    conjunctive quality/runtime/artifact/export admission
    global numerator/denominator aggregation
    factorial interaction arithmetic
    release allowlist, denylist and runtime dependency closure
    content digest, parity threshold and rollback predecessor
    26 fail-closed semantic mutations

not proved:
    any student was trained
    any paper benchmark was reproduced
    any real GPU was benchmarked
    any real model was exported or published
    any method is universally best
    any production rollout or rollback was exercised
    article-wide publication closure

Loop 12 can now audit the complete article’s formulas, source pins, visual semantics, rendering, accessibility, internal links, boundaries, and final claim ceiling without changing the Loop 11 evaluation or release result.

0x0E Boundaries And Handoffs

Distillation can participate in several later workflows, but it does not own their mechanisms.

Boundary	This article contributes	Owning article
structural pruning	teacher targets may support post-pruning recovery; this article does not choose heads, channels, layers, or sparsity structure	M6.11 - Pruning, Structural Compression, and Executable Sparsity
quantization-aware recovery	teacher supervision may be combined with fake-quantized student training; fake-quant placement and low-bit optimization are separate	M8.7 - Quantization-Aware Training
native low-bit architectures	a low-bit student can be distilled, but its binary/ternary parameterization and training dynamics are not KD	M8.8 - Native Binary and Ternary Transformers
quantized artifact integration	when low-bit W/A/KV or a native low-bit representation is added, retain the student bundle/provenance and validate the composed format, loader and kernel	M8.9 - Quantized Artifacts and Runtime Integration
quantized deployment rollout	student-only local proof feeds, but does not replace, whole-tuple quality/resource/canary/rollback gates for that low-bit composition	M8.10 - Deployment Evaluation, Release, and Rollback

The same boundary applies to rationale, chain-of-thought, tool-trajectory, and cross-tokenizer distillation. They can be important transfer interfaces, but they add data-governance, observability, semantic-alignment, and sometimes safety contracts that this article has not derived. They should not be hidden inside a generic “sequence KD” label.

0x0F Claim Status And Evidence Discipline

This article uses five evidence labels:

Label	Meaning
Source-local	a claim or explanatory pattern present in the related Transformer Series articles or repository material
Externally verified	checked against a primary paper or current official documentation
Rewrite interpretation	a synthesis, engineering derivation, or implementation recommendation produced for this article
Source-reported	a result reported by its source under that source’s model, data, hardware, and evaluation setup
Uncertain	plausible or implementation-dependent, but not established strongly enough to generalize

The main claims resolve as follows:

Claim	Status	Scope
softened output distributions can transfer information beyond one hard label	Externally verified	classical KD under a shared output support
sequence-level KD can train on teacher-decoded sequences	Externally verified	the decoded sequence is an approximation/artifact, not the full teacher distribution
DistilBERT combines language modeling, distillation, and cosine objectives in a reduced-depth BERT student	Externally verified	encoder-specific recipe; published outcomes remain source-reported
Patient KD, TinyBERT, and MiniLM match different intermediate objects	Externally verified	their layer, width, head, token, and phase contracts differ
an intermediate teacher assistant can mitigate some capacity gaps	Externally verified, source-scoped	original evidence does not prove universal benefit for LLM distillation
MiniLLM uses a sequence reverse-KL approach and GKD trains on student-generated outputs with teacher feedback	Externally verified	exact estimators, mixtures, and implementations remain method-specific
the four-contract signature is sufficient to expose most hidden method differences	Rewrite interpretation	an audit framework, not a theorem from one paper
exact sharded vocabulary KL can avoid full-logit all-gather through global normalizers and local contributions	Rewrite interpretation grounded in distributed softmax identities	requires aligned token support, shard mapping, and autograd-aware collectives
sparse logit sampling can estimate a dense objective without treating naive top-$k$ renormalization as exact	Externally verified, recent and source-scoped	estimator assumptions and variance must be audited for the actual method
KD always makes a model smaller or faster	False if generalized	architecture and runtime create the saving; KD is a transfer process
on-policy or reverse-KL distillation is always superior	Uncertain if generalized	depends on model, prefix distribution, estimator, data, and budget

All performance numbers from papers are therefore source-reported unless a local reproduction states the student artifact, data, hardware, precision, runtime, batch/concurrency, and measurement protocol. No result in the references should be read as a universal deployment forecast.

The diagrams in this article are original explanatory redraws. Where a visual follows a paper-specific organization, its caption names that source. The Sequence KD, TinyBERT, MiniLM, MiniLLM, and GKD papers remain the authoritative descriptions of their own algorithms and figures.

0x10 Publish Gate And Final Contract

Before this article is considered complete, a reader should be able to audit the following without guessing.

0x10A Concept Gate

• Distillation is defined as transfer, not as a weight representation or compression format.
• The student architecture, precision, and runtime are identified as the source of deployment savings.
• Matched object, prefix distribution, teacher access, and execution plan remain separate contracts.
• Token KD, sequence artifacts, sequence objectives, and on-policy token objectives are not conflated.
• Paper variants are compared by what they change, not ranked under one unspecified benchmark.

0x10B Mathematical Gate

• Temperature softening, centered logits, and the $\tau^2$ gradient-scale convention are derived.
• Forward and reverse KL state which distribution supplies the expectation.
• Token rows, shifted labels, sequence trajectories, and sampled-prefix estimators are distinguished.
• Hidden, attention, and relation objectives state layer, width, head, token, position, and mask assumptions.
• Distributed vocabulary normalization and valid-token reduction preserve the intended global objective.

0x10C Implementation Gate

• The teacher is pinned, in evaluation mode, evaluated without gradients, and excluded from the released student.
• Tiny tensor traces expose shifts, masks, route/mapping metadata, and gradient ownership.
• Sampling stops gradients through discrete token IDs while rescoring the exact sampled prefixes.
• Online, offline, hybrid, and on-policy paths include their distinct runtime, cache, transport, and checkpoint state.
• Cached targets have versioned schemas, manifests, checksums, and explicit invalidation rules.
• Student-only export excludes teacher, assistant, cache, rollout, and training-only alignment state.

0x10D Evidence And Scope Gate

• Primary papers support paper-specific mechanism claims.
• Current framework behavior is cited as implementation-specific, not as a mathematical requirement.
• Benchmark and speed claims are labeled source-reported or locally reproduced with full context.
• Sparse-target claims are recent, source-scoped, and do not call naive top-$k$ renormalization exact.
• Pruning, QAT, native low-bit training, artifact integration, and rollout are handed to their owning articles.
• Semantic visuals state what they prove, include mobile variants, and do not substitute diagram similarity for algorithmic equivalence.
• Loop 12 independently closes formula and source identity, normalized browser rendering, accessibility, owner boundaries, local-route acceptance, and exact article-source rollback as publishable_with_declared_scope.

0x10E Answer To The Owner Question

The teacher transfers behavior by exposing a richer training signal than the student’s original hard labels or data provide. That signal may be a softened token distribution, generated sequence, hidden state, attention relation, value relation, or likelihood on a student-generated trajectory. The student becomes cheaper only because a separately chosen architecture and numerical representation make it cheaper.

The complete method is therefore not

KD = KL(teacher, student)

but

distillation method =
    deployable student specification
  + matched teacher object
  + supervised prefix/data distribution
  + teacher access and alignment contracts
  + estimator and optimization contract
  + artifact/runtime/checkpoint plan
  + student-only evaluation and release gate

If any term is omitted, the method may still run, but it is no longer clear what information was transferred, what objective was optimized, what resources were consumed, or whether the released artifact realizes the claimed saving.

0x11 Final Formula, Source, Rendering, Accessibility, Boundary, And Rollback Closure

Loop 12 is not another distillation strategy. It is a transaction over the article produced by Loops 1 through 11. A token-KD oracle can pass while a source revision drifts; a release allowlist can pass while its explanatory figure is missing on mobile; a mathematically correct section can still point to the wrong downstream owner. Publication therefore requires several independent surfaces to identify the same article candidate.

The parent is the exact Loop 11 Markdown digest. The final candidate changes only this closure section and the single final-gate row above. The rollback oracle removes those two additions in memory and requires the reconstructed bytes to hash to that parent. It never rewrites the working article.

0x11A One Candidate, Several Independent Evidence Surfaces

There is no aggregate documentation score. Every row below must pass, and all rows are joined by the final article digest.

Surface	Frozen publication identity	Acceptance condition
formulas	309 source-ordered display blocks plus their rendered inline peers	exact LaTeX text, order, section, line span, and digest; browser source/render parity
sources and claims	61 published external URLs, revision-pinned source artifacts, source traces, and claim matrices from all eleven loops	local source-artifact digests exact; article URL set exact; source-reported results remain scoped
figures and visuals	115 local asset routes yielding 72 rendered image elements, including 43 responsive pictures and 29 source-figure elements	every file exists and is digest-bound; every alternative is nonempty; responsive source selection matches viewport
executable evidence	eleven historical loop contracts and 1,331 mechanism, artifact, runtime, comparison, and release mutations	each mutation remains owned by its original oracle; the final audit does not flatten failure semantics
browser surface	desktop/mobile crossed with light/dark	MathJax parity, headings, anchors, language, themes, local routes, images, content types, and article-owned overflow pass
release and rollback	one final article digest and one exact Loop 11 parent digest	publication follows browser acceptance; rollback reconstructs the parent bytes without mutating the source file

The final registries are candidate registries. They do not rewrite historical Loop 1 through Loop 11 manifests whose article digests describe earlier states. This distinction preserves provenance: historical evidence remains replayable against the state it originally validated, while Loop 12 records the complete final state separately.

0x11B Normalized Browser And Accessibility Contract

Four profiles load the built local route: desktop light, desktop dark, mobile light, and mobile dark. Each profile waits for all MathJax sources to gain a rendered peer, resolves every article image, and audits the complete article DOM. Screenshots are viewport evidence; the checks are not limited to the screenshot rectangle.

math sources == rendered MathJax nodes
display-math sources == 309
rendered image elements == 72
responsive picture elements == 43
source-figure elements == 29
broken images == 0
empty image alternatives == 0
duplicate heading IDs == 0
broken internal anchors == 0
article-owned horizontal overflow == 0
article-owned console/page errors == 0
all local image and owner routes return the expected content type
mobile profiles select mobile sources; desktop profiles select desktop sources

The browser contract proves this repository snapshot under the four declared profiles. It does not prove every browser, assistive technology, zoom policy, site-shell arrangement, live external URL, or production deployment.

0x11C Concrete Owner Boundaries And Program Handoff

The final boundary audit resolves conceptual handoffs to actual local files, checks their routes, records current digests under an explicit policy, and rejects long exact prose duplication. These contracts remain non-transferable:

Relation	Owner	Contract preserved here
consumes	M3.9	teacher forcing, causal shifts, masks, and valid-token reduction
consumes	M3.8	output logits, shared vocabulary support, and token-row interface
consumes	M4.1	parameter, compute, memory, activation, and traffic accounting
consumes	M4.2	logical-versus-observed execution and measurement boundary
next program owner	M6.11	pruning scores, masks, graph surgery, sparse artifacts, recovery, and pruning-specific release proof
defers	M7.1	draft proposal, exact target verification, and speculative K/V state
defers	M8.7	fake quantization, STEs, observers, and low-bit optimization
defers	M8.8	binary and ternary student parameterization
conditionally consumes	M8.9	format, loader, executable ABI and kernel compatibility when composed with low-bit representation
conditionally consumes	M8.10	whole-tuple canary, monitoring and rollback for that low-bit composition; not universal student rollout ownership

M6.10 closes teacher-student transfer and a teacher-independent candidate artifact. It does not own which parameters M6.11 removes, how an M8 numerical representation is packed, or how M8.10 moves production traffic. In particular, the local registry admission in Loop 11 is a predecessor to deployment, not a claim that deployment occurred.

0x11D Release Verdict, Evidence Ceiling, And Narrow Rollback

The final article state is publication_closed with verdict publishable_with_declared_scope. Static formula, source, figure, visual, failure, boundary, and browser surfaces pass. All local article and asset routes are accepted. Live external availability remains unclaimed.

The rollback artifact is executable only at article-source scope: it proves that removing the Loop 12 span and gate row returns the exact Loop 11 Markdown bytes. It does not roll back a student checkpoint, tokenizer, teacher service, optimizer, target cache, rollout cache, registry generation, production service, or infrastructure deployment.

No teacher or student was trained. No framework model, GPU kernel, production scheduler, paper benchmark, live external-link monitor, or deployment was executed by Loop 12. The closure proves publication consistency for this repository snapshot; it does not establish empirical superiority of a distillation method or a real model release.

0xFF References

Primary Papers

1. Jimmy Ba and Rich Caruana. Do Deep Nets Really Need to be Deep?, NeurIPS 2014.
2. Adriana Romero et al. FitNets: Hints for Thin Deep Nets, ICLR 2015.
3. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, 2015.
4. Yoon Kim and Alexander M. Rush. Sequence-Level Knowledge Distillation, EMNLP 2016.
5. Sergey Zagoruyko and Nikos Komodakis. Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer, ICLR 2017.
6. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, 2019.
7. Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression, EMNLP-IJCNLP 2019.
8. Xiaoqi Jiao et al. TinyBERT: Distilling BERT for Natural Language Understanding, 2019.
9. Wenhui Wang et al. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, NeurIPS 2020.
10. Wenhui Wang et al. MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers, ACL-IJCNLP 2021.
11. Seyed-Iman Mirzadeh et al. Improved Knowledge Distillation via Teacher Assistant, AAAI 2020.
12. Yuxian Gu et al. MiniLLM: On-Policy Distillation of Large Language Models, ICLR 2024; this article pins the 2026-01-31 arXiv revision rather than relying on the floating abstract.
13. Rishabh Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes, ICLR 2024.
14. Saurav Muralidharan et al. Compact Language Models via Pruning and Knowledge Distillation, 2024.
15. Anshumann et al. Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs, 2025.
16. Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, AISTATS 2011.
17. Alexander Lin, Jeremy Wohlwend, Howard Chen, and Tao Lei. Autoregressive Knowledge Distillation through Imitation Learning, EMNLP 2020; referred to as ImitKD in this article.
18. Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. $f$-Divergence Minimization for Sequence-Level Knowledge Distillation, ACL 2023.

Current Implementation Documentation

• Hugging Face TRL. GKD Trainer, accessed 2026-07-21.
• Microsoft LMOps. MiniLLM reference implementation, pinned to commit 4f2a9deb5f08e459fd44c2e4792344d78ca89fc3.
• Hugging Face TRL. Pinned GKD trainer source, pinned to commit 24db2a825029f7e0e60f6d3c160485565a784503.
• NVIDIA NeMo Framework. Knowledge Distillation, accessed 2026-07-21.
• NVIDIA Megatron Core. Fused vocabulary-parallel cross entropy, accessed 2026-07-21.

Official documentation above describes current framework interfaces. It is version-scoped implementation evidence, not the primary proof of the paper algorithms.