M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation

0x00 Scaling Laws Are Experiments, Not Constants

M4.1 starts from a model configuration and derives an analytic resource ledger:

architecture and tensor shapes
    -> parameters
    -> activation elements
    -> FLOPs
    -> transferred and persistent bytes

That ledger answers what a proposed run should cost. It cannot answer what loss the run will reach. Before a large training run exists, a team must still choose its model size, data volume, training horizon, optimizer policy, and compute budget. The final run is often too expensive to use as the first experiment, yet its configuration must be selected before its result can be observed.

This creates M4.7’s owner question:

Given a controlled family of smaller training runs and a compute or data constraint, how should loss be defined, measured, fitted, numerically certified, and converted into a training-compute model-and-token allocation before extrapolation or lifecycle claims are made?

The common phrase scaling law can hide how empirical the answer is. The “law” is not a property read directly from the Transformer equation. It is a model fitted to observations produced by a particular experimental system:

model family
data distribution and tokenizer
optimizer and schedule policy
training-token horizon
parameter and compute accounting rules
evaluation split and metric
run inclusion and failure policy

Change one of those objects and the observations may belong to a new study family. A fitted relation can be stable and useful inside its validated range without becoming architecture-independent, data-independent, or timeless.

0x00A Why Predict Before The Full Run Exists?

Suppose a fixed training budget can be spent in several plausible ways:

larger model, fewer processed tokens
smaller model, more processed tokens
more unique data, less repetition
more pilot runs, less budget left for the final run
longer training, but a smaller model to serve afterward

“Bigger models usually improve” does not choose among them. It describes a direction while leaving the constrained allocation unanswered. Even a perfect analytic FLOP count cannot decide whether an additional unit of compute should buy more parameters or more training tokens; the decision needs measured loss response along both axes.

A scaling study therefore acts as an experimental bridge:

small and medium controlled runs
    -> measured held-out behavior
    -> an estimator with known residuals
    -> hidden-scale extrapolation test
    -> constrained allocation
    -> target-adjacent pilot
    -> final-run decision or refit

The bridge is useful because the destination is expensive. It is dangerous when the fitted curve is remembered but the run family that created it is forgotten.

0x00B A Training Curve Is Not A Scaling Curve

The term learning curve has been used for several different plots. Two are especially easy to conflate.

A training or optimization curve follows one run through time:

x-axis:
    optimizer steps, processed tokens, or elapsed training compute

y-axis:
    training loss, held-out loss, gradient scale, or another checkpoint metric

one point:
    one checkpoint from the same optimization trajectory

A cross-run scaling curve compares selected observations from multiple controlled runs:

x-axis:
    data volume, parameter count, analytic compute, or another scale variable

y-axis:
    a consistently measured held-out loss or task metric

one point:
    one run or one declared checkpoint-selection result

The distinction affects statistics and interpretation. Checkpoints from one training trajectory share initialization, optimizer state, data order, and all earlier updates. Treating every checkpoint as an independent run inflates the apparent sample count. Conversely, retaining only one final point per run can discard useful information when the method intentionally studies the envelope of a training trajectory. Either choice can be valid, but it must be declared.

The axes also answer different questions:

optimization curve:
    is this run converging, diverging, overfitting, or undertrained?

scaling curve:
    how does a controlled family change as a resource variable changes?

Later methods sometimes use many checkpoints from one run to estimate the best loss reachable at several compute budgets. That does not erase the distinction; it creates a fitting method whose dependence structure must be audited.

0x00C From Empirical Learning Curves To Neural Scaling Studies

Learning curves long predate large language models. The basic experimental question is familiar: as a learner receives more examples or more capacity, how quickly does its error decline, and where does improvement stop? What changed with modern deep learning was the scale of the experiment and the regularity observed across much larger run families.

In 2017, Hestness et al. measured generalization-error and model-size behavior across machine translation, language modeling, image classification, and speech recognition. Their systems were recurrent and convolutional rather than today’s decoder-only Transformers. That historical boundary is important: the paper is evidence for a broad empirical pattern and a measurement methodology, not a direct source of current language-model allocation constants.

The authors summarized a real-application learning curve using three regions. The following visual is a responsive reconstruction of their conceptual Figure 6, not a plot of the paper’s measured points.

Responsive reconstruction of Figure 6 in Hestness et al., “Deep Learning Scaling is Predictable, Empirically”. The three-region drawing is conceptual. The paper reported broad power-law regions in its tested applications but explicitly said that its real-application runs had not reached the final irreducible-error region.

Small-data region

With too little data, the learner cannot yet extract enough information to separate the task from a strong guess baseline. The curve can be flat or transitional rather than following the relation observed later.

This region is not merely “the left side of a line.” Several causes can overlap:

insufficient examples for the task
optimization settings that fail at the smallest scale
model capacity too small for the target relation
metric granularity or class imbalance
regularization tuned for another data regime

Fitting this transition together with a later regular region can change the estimated slope. A scaling study must first ask whether all observations belong to the same behavioral regime.

Power-law region

After the initial transition, Hestness et al. observed broad regions where generalization error looked approximately linear when both the scale variable and error were viewed in logarithmic coordinates. A straight segment in that view is useful because it provides a compact rate of diminishing returns and a candidate extrapolator.

The visual does not prove that the same straight segment continues forever. It records behavior over tested ranges. The paper also found that the steepness depended on the domain and metric, and its empirical exponents were not explained by the theoretical values discussed at the time. The safe lesson is:

regularity was measured;
its range and experimental family are part of the result;
the exponent must be estimated rather than assumed.

Irreducible-error region

Eventually, additional examples may stop reducing the measured error because the remaining error includes ambiguity in the data-generating process, labeling noise, incomplete inputs, evaluation noise, or limitations of the chosen problem statement. This motivates an error floor.

But a plateau is not automatically the irreducible error of the world. It can also be caused by an unsuitable architecture, optimizer failure, stale data, limited context, an evaluation ceiling, or an accounting bug. Hestness et al. described the final region as likely for real applications and tested a lower bound on toy problems; they did not claim that their real-application experiments had reached it.

That source-local distinction will matter later. A fitted floor is an experimental asymptote under a declared family, not automatically the true entropy of language or a proof that further algorithmic progress is impossible.

0x00D Why The Objective Changes The Decision

The measured observations do not choose an operating point by themselves. They can support different decisions depending on the objective and constraint.

training-only objective:
    minimize held-out loss under a fixed training-compute budget

data-constrained objective:
    allocate model size and repeated exposure when unique tokens are limited

deployment objective:
    meet a quality target while minimizing training plus lifetime serving cost

downstream objective:
    choose scale for a declared task suite rather than pretraining loss alone

Later language-model studies changed the experimental target from general learning-curve description to explicit parameter/data/compute allocation. Kaplan et al. and Hoffmann et al. are central examples, but their fitted conclusions cannot be compared as bare numbers: model-size definitions, compute accounting, training horizons, and optimizer policies condition the frontier. Data scarcity and inference demand change the optimization problem again; they do not merely add another caption to the same curve.

The complete empirical-scaling path therefore uses three article owners rather than asking one article to fit a law, prove its extrapolation, alter its data geometry, translate it into product metrics, and make a deployment decision:

Owner	Contracts it closes	Output handed forward
M4.7	measurement, family control, baseline functional form, training-compute allocation, fitting, and numerical certification	a versioned fitted surface, uncertainty draws, residual evidence, and declared support
M4.8	scale-ordered extrapolation, alternative curve families, unique-data limits, repetition, and effective sample size	an extrapolation-tested, constraint-aware performance surface
M4.9	deliberate over-training, downstream metric maps, serving demand, lifecycle cost, discrete candidate selection, and pilot policy	a versioned artifact decision with uncertainty, evidence, and abort conditions

A coefficient without these contracts is not a reusable result. It is a number detached from the experiment that gave it meaning.

0x00E Owns And Defers

This article owns:

empirical learning and scaling-curve regimes
model/data/compute/loss measurement contracts
controlled run families and run manifests
one-variable power laws, fitted floors, and joint model-data loss surfaces
compute-optimal parameter/token derivations
paper-by-contract comparison
fitting objectives, parameterization, initialization, and convergence evidence
residual audits, grouped uncertainty, and construction of scale-ordered holdouts
training-only allocation distributions and feasible-candidate regret

It defers:

M4.1:
    deriving Transformer parameters, activation elements, FLOPs, and bytes

M4.2-M4.6:
    mapping analytic work onto GPU kernels and hardware pipelines

M4.8:
    extrapolation model selection, broken laws, unique-data limits,
    repeated exposure, and effective-sample-size geometry

M4.9:
    deliberate over-training, downstream metric translation,
    lifecycle allocation, target pilots, and artifact decisions

M5:
    request scheduling, K/V state, serving placement, and runtime memory policy

M6:
    MoE active-parameter semantics, pruning, and distillation mechanisms

M8:
    quantized artifact construction, runtime qualification, release, and rollback

M4.7 consumes those boundaries rather than redefining them. For example, an MoE study must declare whether N means stored parameters, active parameters, or a FLOPs-per-token proxy. M6.1 owns the routed-capacity and active-parameter semantics; M4.7 does not reteach MoE routing. M4.7 ends with a numerically certified baseline law and its uncertainty objects. M4.8 owns the next question: whether that law survives outside its fitted scale and what must change when the available data is finite or repeatedly exposed.

0x00F Claim Status And Source Discipline

Status	Meaning in this article
Source-local	The explanatory seed comes from the local book or Transformer Series, but is checked before becoming evidence.
Externally verified	A mechanism or experimental contract is checked against the original paper or official project material.
Source-reported	A coefficient, run count, model size, benchmark, or conclusion remains attached to its source’s setup.
Rewrite derivation	Algebra or experimental pseudocode is reconstructed step by step from declared assumptions.
Interpretation	The article connects sources or proposes an operational rule not stated verbatim by one paper.
Uncertain	The available evidence does not justify a stronger claim.

The article will not use a paper’s fitted constant as a default configuration. It will first reconstruct how that paper created its observations.

0x00G Three-Article Reading Map

The three articles form one evidence pipeline, not three surveys of similar papers:

M4.7  construct and certify the baseline empirical law
  -> M4.8  test extrapolation and rebuild the law under changed data geometry
  -> M4.9  translate quality into downstream and lifecycle artifact decisions

Article	Start with	Finish with	Do not substitute
M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation	controlled run records and analytic resource definitions from M4.1	fitted parameters, uncertainty draws, residuals, support boundaries, and a training-compute allocation	fit convergence for extrapolation evidence
M4.8 - Extrapolation, Alternative Scaling Laws, and Data Constraints	M4.7 fit artifacts plus scale-ordered hidden groups and data-lineage metadata	selected curve family, extrapolation error, constraint-aware coordinates, and effective-data uncertainty	random-row interpolation for scale extrapolation
M4.9 - Over-Training, Downstream Metrics, and Lifecycle Allocation	an M4.8-certified quality surface plus measured M5 runtime and M8 artifact evidence	a discrete candidate, scenario costs, target-adjacent pilot, and proceed/abort/refit policy	training FLOPs for served cost or validation loss for product utility

Read M4.7 through M4.9 once for the mathematical chain. Revisit M4.9 after M5 and M8.10 when the goal is an executable deployment decision rather than a conceptual lifecycle comparison.

0x00H Pinned Source-Version Contract

Scaling papers are revised, and revisions can change coefficients, experiment counts, fitting details, or the strength of a conclusion. The quantitative and paper-comparison claims in this article were audited against the versions below. An unversioned arXiv link is useful for discovery; it is not the evidence identity used by this article.

Source	Audited version	Contract consumed here
Hestness et al.	arXiv:1712.00409v1	historical learning-curve regions and empirical power-law framing
Rosenfeld et al.	arXiv:1909.12673v2	joint model/data generalization surface lineage
Kaplan et al.	arXiv:2001.08361v1	early language-model scaling and compute-allocation contract
Hoffmann et al.	arXiv:2203.15556v1	three estimator families, Approach 3 surface, and Chinchilla frontier
DeepSeek-AI	arXiv:2401.02954v1	non-vocabulary FLOPs/token coordinate, IsoFLOP grid, and data-version comparison
Porian et al.	arXiv:2406.19146v4	cumulative head-cost, warmup, schedule, and optimizer reconciliation
Alabdulmohsin et al.	arXiv:2209.06640v2	extrapolation-aware fitting and robust-objective lineage handed to M4.8
Besiroglu et al.	arXiv:2404.10102v2	digitized-data replication and uncertainty critique of Chinchilla Approach 3

Every coefficient, run count, model range, token range, and benchmark result from these papers remains source-reported. The article’s algebra, estimators, executable examples, and engineering gates are rewrite derivations or interpretations unless a source is named at the claim.

0x01 The Measurement Contract

A plotted point looks simple. It may have one horizontal coordinate and one vertical coordinate. Operationally, it is a projection of a much larger record:

run configuration
    -> produced checkpoint
    -> evaluated checkpoint under a named protocol
    -> accounted parameter/token/compute values
    -> selected observation used by a fit

If two records use different definitions, placing them on the same axes does not make them comparable. The measurement contract defines the fields needed before any curve is fitted.

0x01A Core Symbols And Units

Symbol	Meaning	Unit or required convention
N	model-size variable used by the study	exact parameter convention must be named
N_all	all stored model parameters	parameters
N_nonemb	parameters excluding declared embedding/output terms	parameters plus exclusion rule
N_active	parameters activated for one token or declared routing average	parameters/token plus routing rule
M_tok	analytic forward FLOPs per token used as a model-scale proxy	FLOPs/token plus operation convention
D_total	total tokens processed by training	tokenizer-produced token occurrences
D_unique	unique token occurrences available before repetition	tokenizer-produced token occurrences
R_D	repeated exposure beyond the first pass	repeats or epochs, precisely defined
C_analytic	analytic training-compute estimate	FLOPs plus derivation and omissions
C_measured	measured or profiler-derived training compute	FLOPs or accelerator counters plus tool
L_train	training objective value	aggregation, units, masking, and smoothing rule
L_eval	held-out loss used by the scaling study	split, tokenizer, units, and checkpoint rule
Q	downstream quality metric	task suite, prompt/eval protocol, and aggregation
rho	processed-token multiplier relative to the declared N	dimensionless; only meaningful after N is fixed

These symbols are intentionally more explicit than a single N, D, C, and L. Papers often use those compact letters while assigning different objects to them. M4.7 will use the compact symbols only after mapping them to the expanded fields for the study being discussed.

0x01B Model Size Is A Policy, Not A File Label

“A 7B model” is enough for a product name and insufficient for a scaling fit. Possible model-size coordinates include:

all trainable parameters
non-embedding parameters
parameters excluding both token embeddings and output head
stored parameters in a sparse or MoE model
active parameters per token
analytic forward FLOPs per token

The choices can differ materially. Vocabulary size and weight tying change the embedding/head contribution. MoE increases stored capacity while activating only selected experts. A pruned checkpoint can contain dense tensors with zeros, a sparse representation, or physically smaller matrices. A formula fitted against one coordinate cannot silently be replotted against another.

The manifest therefore records both the value and its definition:

model:
  parameter_definition: "non-embedding trainable parameters"
  parameter_count: 123456789
  non_embedding_parameter_count: 120000000
  flops_per_token_definition: "dense forward, multiply and add counted separately"
  flops_per_token: 720000000

M4.1 owns the derivation of these analytic quantities. M4.7 owns the rule that every run must use the same derivation or be placed in a separate accounting stratum.

0x01C Processed Tokens Are Not Unique Data

D_total counts token occurrences consumed by optimization. D_unique counts the available token occurrences before repetition under the declared corpus and tokenizer. They are equal only when the run processes one pass without resampling, duplication, or curriculum effects.

same source documents + different tokenizer
    -> different token sequence and token count

same unique corpus + multiple epochs
    -> same D_unique, larger D_total

same target D_total + changed mixture weights
    -> different exposure distribution

A data-constrained study needs both fields because another pass over existing tokens need not provide the same return as new tokens. Even without repetition, deduplication, filtering, contamination removal, ordering, and mixture changes can alter the effective experiment. The corpus manifest and tokenizer hash are part of the point.

0x01D Compute Has Several Ledgers

The word compute can refer to at least four non-equivalent objects:

Ledger	What it measures	Why it can disagree with another ledger
C_analytic	modeled arithmetic from tensor shapes and training rules	may omit embeddings, attention terms, rematerialization, routing, or failed work
C_measured	profiler or accelerator-counter estimate	tool coverage and counter conventions differ
wall time	elapsed duration	depends on utilization, failures, communication, and hardware count
monetary/energy cost	resources paid or consumed	depends on prices, power, reservation, and infrastructure overhead

An analytic approximation is useful when applied consistently. It becomes misleading when two papers use different operation counts or when a model’s non-block work is no longer negligible. A scaling law against analytic FLOPs is not automatically a wall-time law.

The run manifest records both the estimate and the machine result when available. Later allocation sections will state which one defines the constraint.

0x01E Loss Needs A Tokenizer, Split, And Aggregation Rule

Cross-entropy is not fully defined by the word “loss.” A reproducible held-out measurement needs:

evaluation corpus and version
tokenizer and vocabulary
token or sequence averaging
ignored/masked positions
natural-log or base-2 units
checkpoint selection rule
context packing and truncation
data-contamination policy

Changing the tokenizer changes both the number of tokens and the units over which loss is averaged. Comparing raw per-token losses across tokenizers can therefore mix the dependent variable and the data coordinate at the same time. Similarly, the minimum held-out checkpoint, final checkpoint, and a schedule-matched checkpoint answer different questions.

Downstream quality Q is even less universal. A mean across tasks needs task weights, shot counts, prompts, decoding rules, and aggregation. M4.7 will not substitute an unspecified benchmark average for L_eval and call it the same scaling target.

0x01F What Counts As One Run?

A run is one optimization trajectory created by a declared initialization, data order, optimizer state, schedule, and configuration. Its checkpoints are repeated observations from that trajectory, not independent replicas.

independent seed rerun:
    new initialization and stochastic trajectory

checkpoint at a later token count:
    same trajectory with shared history

resume after preemption from exact state:
    continuation of the same run

restart after failure with changed batch or learning rate:
    new run or explicitly marked intervention

This identity rule matters when estimating uncertainty. Resampling checkpoints as though each were a new model can produce confidence intervals that are too narrow. A study may intentionally use checkpoints to build a compute envelope, but its fit and bootstrap must preserve trajectory grouping.

0x01G Family Controls And Retuning Policy

A scaling family is not created merely by giving models similar names. It declares what stays invariant and what changes with scale.

Axis	Fixed-family example	Valid scale-dependent policy	Invalid silent change
architecture	decoder-only block family and width/depth rule	depth and width follow a registered schedule	attention type changes midway without stratification
data	one corpus version and mixture	registered mixture schedule by scale	new filtered corpus for only the largest run
tokenizer	one tokenizer hash	none unless tokenizer is an explicit study axis	vocabulary changes but token counts are compared directly
optimizer	AdamW family	learning rate, batch, or beta follows a declared tuning rule	only failed large runs receive manual tuning
schedule	one schedule family	horizon and warmup scale by registered rule	cosine horizon leaks knowledge of the target run
precision	one numerical policy	explicit scale-dependent precision study	largest run silently changes loss scaling or quantization
evaluation	one held-out manifest and checkpoint rule	registered additional downstream suite	best checkpoint chosen differently by scale

Retuning is not automatically a confound. A fixed learning rate can itself become an unfair handicap as model size changes. The contract must state whether the scientific object is:

one fixed training recipe across scale
or
the best result under a declared scale-dependent tuning budget

Those are different experiments. The tuning budget, search space, and failure policy belong in the manifest.

0x01H Run Manifest Skeleton

The following schema is the minimum operational object that later fitting code should consume. It is intentionally more detailed than a plotting CSV.

run_id:
code_commit:
config_hash:

model:
  family:
  architecture:
  scaling_policy:
  parameter_definition:
  parameter_count:
  non_embedding_parameter_count:
  active_parameter_definition:
  active_parameter_count:
  flops_per_token_definition:
  flops_per_token:
  context_length:
  vocabulary_size:
  tied_embeddings:

data:
  corpus_manifest_hash:
  corpus_version:
  tokenizer_hash:
  tokenizer_name:
  unique_tokens:
  processed_tokens_target:
  processed_tokens_actual:
  epochs_or_repeat_count:
  mixture_weights:
  deduplication_policy:

optimization:
  optimizer:
  learning_rate:
  beta1:
  beta2:
  weight_decay:
  batch_tokens:
  gradient_accumulation:
  warmup_tokens:
  schedule:
  schedule_horizon_tokens:
  seed:
  precision:
  tuning_policy_id:

compute:
  analytic_training_flops:
  analytic_flops_definition:
  measured_training_flops:
  profiler_or_counter_source:
  accelerator:
  accelerator_count:
  wall_time:
  achieved_mfu:

evaluation:
  checkpoint_rule:
  selected_checkpoint_tokens:
  train_loss:
  heldout_loss:
  loss_units:
  heldout_manifest_hash:
  context_and_packing_policy:
  downstream_suite_hash:
  downstream_aggregation:

lineage:
  parent_run_id:
  resumed_from_checkpoint:
  trajectory_group_id:

status:
  completed:
  divergence:
  oom:
  preemption:
  excluded_from_fit:
  exclusion_reason:

Three details prevent common distortions:

1. processed_tokens_target and processed_tokens_actual are separate. A failed or early-stopped run did not consume its planned horizon.
2. Exclusion is recorded rather than implemented by deleting the row. Failed large runs are evidence about the training policy even when they cannot enter a loss fit.
3. trajectory_group_id keeps checkpoint-derived observations together during uncertainty estimation.

0x01I From Manifest To One Canonical Observation

Fitting code should not read arbitrary plotting columns directly. It should first validate the contract and construct a canonical observation.

def canonical_observation(run, study):
    assert run.status.completed
    assert not run.status.excluded_from_fit

    assert run.model.family == study.model_family
    assert run.model.parameter_definition == study.parameter_definition
    assert run.model.flops_per_token_definition == study.flops_definition

    assert run.data.corpus_manifest_hash == study.corpus_manifest_hash
    assert run.data.tokenizer_hash == study.tokenizer_hash

    assert run.optimization.tuning_policy_id == study.tuning_policy_id
    assert run.evaluation.heldout_manifest_hash == study.heldout_manifest_hash
    assert run.evaluation.loss_units == study.loss_units

    return {
        "run_id": run.run_id,
        "trajectory_group_id": run.lineage.trajectory_group_id,
        "N": select_model_scale(run.model, study.model_scale),
        "D_total": run.data.processed_tokens_actual,
        "D_unique": run.data.unique_tokens,
        "C_analytic": run.compute.analytic_training_flops,
        "C_measured": run.compute.measured_training_flops,
        "L_eval": run.evaluation.heldout_loss,
    }

The assertions are not incidental data cleaning. They define the study. If a field differs for a scientifically intentional reason, the correct action is to register a sensitivity axis or a separate stratum, not to remove the assertion and merge the points.

0x01J Measurement Failure Lab

Failure	Plot still looks plausible?	What became invalid?	Required response
embeddings included only for small models	often	horizontal coordinate changes definition with scale	recompute one parameter convention
largest runs use a new tokenizer	often	token counts and loss units both change	separate family or evaluate in common units
planned tokens used after early stop	yes	data and compute coordinates are overstated	use actual processed tokens
checkpoints treated as independent seeds	yes	uncertainty is underestimated	group by trajectory
failed large runs deleted	yes	survivorship hides instability	retain status and report failure frontier
held-out split changes with corpus version	yes	vertical coordinate changes target distribution	use one frozen evaluation manifest
analytic FLOPs compared with billed GPU-hours	sometimes	two cost ledgers are conflated	keep separate axes or define conversion
best checkpoint selected only for some scales	yes	checkpoint rule favors part of the family	apply one registered selection rule

These failures are dangerous precisely because a smooth line can survive them. Visual regularity is not evidence that the measurement contract was coherent.

0x02 From A Regular Learning Curve To A Power Law

The measurement contract tells us what one observation means. It does not yet specify the shape connecting observations. We now consider the simplest useful candidate for one positive scale variable x:

x may be:
    training examples under one fixed-model learning-curve study
    processed tokens under one declared training policy
    parameter count under one architecture-scaling policy
    analytic compute under one accounting convention

L(x) is:
    one consistently measured loss or error for that run family

This section does not claim that all four choices share one exponent. It asks what a power law means after the independent variable and metric have already been fixed.

0x02A Deriving The Functional Form From Relative Scale

Let E be a candidate limiting loss for the declared family, and define the excess loss above that floor:

$$G(x)=L(x)-E.$$

Suppose a regular scaling region has the following property: multiplying x by the same factor r multiplies excess loss by the same factor, regardless of the starting value of x. Then there is a positive function q such that:

$$G(rx)=q(r)G(x).$$

Apply one scale change s and then another scale change r:

$$G(rsx)=q(r)G(sx)=q(r)q(s)G(x).$$

Applying the combined scale change in one step also gives:

$$G(rsx)=q(rs)G(x).$$

Therefore:

$$q(rs)=q(r)q(s).$$

For a positive continuous function, this multiplicative consistency has a power-form solution. Because loss should decline rather than grow in the region being modeled, write its exponent as -alpha, with alpha > 0:

$$q(r)=r^{-\alpha}.$$

Choose a positive reference scale x_ref and call the excess loss there A_ref = G(x_ref). Substituting r=x/x_ref yields:

$$G(x)=A_{ref}\left(\frac{x}{x_{ref}}\right)^{-\alpha}.$$

Adding the candidate floor back gives the one-variable law:

$$\boxed{ L(x)=E+A_{ref}\left(\frac{x}{x_{ref}}\right)^{-\alpha} } \qquad A_{ref}>0,\ \alpha>0.$$

Renderer-safe form:

G(x) = L(x) - E
G(r*x) = r^(-alpha) * G(x)

L(x) = E + A_ref * (x / x_ref)^(-alpha)
where A_ref > 0 and alpha > 0

This is a derivation from a scale-covariance assumption, not a proof that a neural network must obey the result. Measurements decide whether the assumption is adequate over a declared interval. Small-data transitions, capacity limits, optimizer failures, and metric boundaries can all violate it.

0x02B What The Three Parameters Mean

Parameter	Meaning inside the fitted family	What it is not
E	limiting loss approached by the candidate model as x grows	automatically the true entropy, Bayes risk, or universal task floor
A_ref	excess loss at the declared reference scale x_ref	invariant to changing the unit or reference used for x
alpha	dimensionless elasticity controlling relative improvement under multiplicative scaling	guaranteed to transfer across data, model, optimizer, metric, or architecture changes

The normalized form makes units explicit. If the unnormalized expression A*x^(-alpha) is used and the unit of x changes from tokens to millions of tokens, the numerical value of A changes. A_ref avoids that ambiguity by pinning the amplitude to a named reference point.

For a decreasing loss, A_ref and alpha are positive. A higher-is-better metric such as accuracy cannot be inserted blindly with the same signs. The study must model an error, a gap to a ceiling, or derive a different monotone form.

The candidate floor also must lie below every observation used by a logarithmic excess-loss fit:

$$E < \min_i L(x_i).$$

That inequality guarantees only that the logarithm exists on observed points. It does not identify the correct floor.

0x02C The Log-Coordinate Derivation

Subtract the floor before taking logarithms:

$$L(x)-E=A_{ref}\left(\frac{x}{x_{ref}}\right)^{-\alpha}.$$

Then:

$$\log(L(x)-E) = \log A_{ref} - \alpha\log\left(\frac{x}{x_{ref}}\right).$$

Renderer-safe form:

log(L(x) - E)
    = log(A_ref) - alpha * log(x / x_ref)

The logarithm base does not change alpha as long as the same base is used on both axes.

In these coordinates:

horizontal coordinate:
    log(x / x_ref)

vertical coordinate:
    log(L - E)

intercept at x = x_ref:
    log(A_ref)

slope:
    -alpha

The exponent is therefore a logarithmic derivative:

$$\alpha = -\frac{d\log(L-E)}{d\log x}.$$

Equivalently, increasing scale by a factor r leaves the following fraction of the previous excess loss:

$$\frac{L(rx)-E}{L(x)-E}=r^{-\alpha}.$$

This ratio is the practical meaning of the exponent. It concerns the remaining gap above E, not the total loss.

alpha	Excess loss remaining after 2x scale	Excess loss remaining after 10x scale
0.05	96.6%	89.1%
0.10	93.3%	79.4%
0.20	87.1%	63.1%

These are mathematical examples, not coefficients recommended for language models. They show why a visually shallow slope can still compound over several orders of magnitude, and why the same slope implies diminishing absolute gains as the excess loss becomes smaller.

Differentiate in the original coordinate:

$$\frac{dL}{dx} = -\frac{\alpha}{x}(L-E).$$

The absolute improvement from one additional unit of x shrinks both because 1/x shrinks and because the remaining excess loss shrinks. Per logarithmic unit of scale:

$$-\frac{dL}{d\log x}=\alpha(L-E).$$

That identity is the local diminishing-return ledger for the one-variable model.

0x02D Why log L Is Not log(L-E)

If E=0, plotting log L against log x can produce a straight line. With a nonzero floor:

$$\log L(x) = \log\left[ E+A_{ref}\left(\frac{x}{x_{ref}}\right)^{-\alpha} \right],$$

which is not linear in log x. Its local slope is:

$$\frac{d\log L}{d\log x} = -\alpha\frac{L-E}{L}.$$

Far above the floor, (L-E)/L may be close to one and the distinction can look small. Near the floor, the factor approaches zero and log L bends toward a horizontal line even when log(L-E) remains exactly linear.

Renderer-safe distinction:

correct straight-line coordinate:
    log(L - E) versus log(x / x_ref)

generally curved when E is nonzero:
    log(L) versus log(x)

local slope of log(L):
    -alpha * (L - E) / L

Fitting a straight line to log L silently changes the model. It tends to make the apparent slope shallower as observations approach a nonzero floor.

0x02E A Wrong Floor Changes The Apparent Exponent

Suppose the data were generated relative to floor E, but an analyst subtracts a candidate floor E_0. The transformed quantity is:

$$L(x)-E_0 = (E-E_0) + A_{ref}\left(\frac{x}{x_{ref}}\right)^{-\alpha}.$$

Its local log-log slope becomes:

$$\frac{d\log(L-E_0)}{d\log x} = -\alpha\frac{L-E}{L-E_0}.$$

This yields three diagnostic cases:

Candidate floor	Slope behavior	Visual symptom
E_0 = E	constant slope -alpha	transformed points can form a straight line
E_0 < E	magnitude becomes too small at large x	apparent flattening and an underestimated asymptote
E_0 > E	magnitude becomes too large, then the log becomes undefined	apparent steepening followed by invalid excess loss

The floor and exponent are therefore coupled. If all observed losses remain far above the floor, many candidate E_0 values can produce almost straight lines with different slopes. A low interpolation residual does not resolve the ambiguity.

0x02F Worked Example: Similar Fits, Different Futures

Construct five observations at x = 1, 2, 4, 8, 16 from:

$$L(x)=1.6+1.7x^{-0.22}.$$

Now compare two fits:

fit A:
    E = 1.6
    A = 1.7000
    alpha = 0.2200

fit B:
    E fixed to 1.2
    log-excess least-squares fit gives
    A = 2.0912
    alpha = 0.1665

x	constructed observation / fit A	fit B with lower floor
1	3.300000	3.291210
2	3.059561	3.063278
4	2.853129	2.860190
8	2.675893	2.679238
16	2.523725	2.518008

Fit B’s relative RMSE over the observed window is about 0.21%. On that small range, the two curves are visually difficult to distinguish. Extrapolation exposes the difference:

Extrapolation scale	fit A	fit B	Difference relative to fit A
1,024	1.969984	1.859476	-5.6%
1,000,000	1.681367	1.409619	-16.2%

This is a constructed teaching example, not paper data. Its purpose is to show the geometry behind the methodological result emphasized by Alabdulmohsin et al.: parameters that interpolate a finite learning curve best need not be the parameters that extrapolate best. Their paper evaluates this issue across multiple domains and proposes a more flexible monotone estimator; M4.7 defers that estimator comparison to the later functional-form section.

The conclusion is not that fit A would be known in a real study. It is that observations restricted to x <= 16 contain weak information about a floor that controls behavior far outside that range. Floor uncertainty must become prediction uncertainty.

0x02G Code Trace: Profile The Floor, Then Test Scale Holdouts

For a fixed candidate floor, the remaining parameters can be estimated with a linear regression in transformed coordinates:

def fit_given_floor(observations, floor, x_ref):
    # observations: independent or trajectory-aware records with x > 0
    assert floor < min(point.loss for point in observations)

    u = [log(point.x / x_ref) for point in observations]
    v = [log(point.loss - floor) for point in observations]

    intercept, slope = ordinary_least_squares(u, v)
    alpha = -slope
    amplitude_at_reference = exp(intercept)

    if alpha <= 0:
        return InvalidFit("loss does not decrease as a positive power law")

    return PowerLawFloorFit(
        floor=floor,
        amplitude=amplitude_at_reference,
        alpha=alpha,
        x_ref=x_ref,
    )

Profiling several candidate floors reveals whether interpolation identifies a narrow region:

def profile_candidate_floors(fit_points, scale_holdout, floor_grid, x_ref):
    reports = []

    for floor in floor_grid:
        if floor >= min(point.loss for point in fit_points):
            continue

        fit = fit_given_floor(fit_points, floor, x_ref)
        reports.append({
            "floor": floor,
            "alpha": fit.alpha,
            "interpolation_log_rmse": log_rmse(fit, fit_points),
            "interpolation_original_rmse": original_rmse(fit, fit_points),
            "extrapolation_log_rmse": log_rmse(fit, scale_holdout),
            "residuals_by_log_scale": residuals(fit, fit_points + scale_holdout),
        })

    return reports

This is a diagnostic, not the final fitting algorithm. Later sections will add nonlinear positive parameterization, robust objectives, initialization grids, trajectory-aware bootstrap, and competing functional forms. The important scale-holdout rule is already visible: the largest observations used to choose a functional form or floor must be hidden from parameter fitting.

The fitting objective also matters. Least squares on log(L-E) emphasizes relative error in excess loss. Least squares on L emphasizes absolute error in total loss. Neither is automatically correct for every decision. A study must report the objective rather than describe both as “the best fit.”

0x02H Regime And Identifiability Failure Modes

Failure mode	What happens to the simple law	Diagnostic
small-data guess region included	early points bend away from the regular slope	repeat fits after increasing the lower cutoff
transition region included	one exponent averages two behaviors	inspect local slopes and residual sign changes
floor not approached	E, amplitude, and exponent trade off	profile candidate floors and propagate the range
finite model capacity plateau	fitted E may describe one model size, not the task	separate fixed-capacity and best-family curves
optimizer failure at large scale	apparent floor is a training-policy ceiling	retain failed runs and retune under a declared policy
metric ceiling or quantization	measured score saturates before model behavior	test a smoother underlying loss or error metric
too little log-scale span	many slopes interpolate similarly	add orders of magnitude or narrow the claim
mixed data/model families	residuals encode a hidden intervention	stratify by manifest field or model the intervention
dependent checkpoints treated as replicas	fit looks precise without independent evidence	group by trajectory during validation and bootstrap

Hestness et al. already separated the small-data, regular power-law, and irreducible-error regions. Alabdulmohsin et al. sharpen the fitting consequence: if the observed data include transitional behavior, a power-law-plus-floor form can fit the finite window yet infer the wrong asymptotic exponent. The answer is not to discard the simple model automatically. It is to validate the region and compare extrapolation behavior.

0x02I Minimum Audit Before Trusting A One-Variable Law

1. Freeze the measurement and family contracts from 0x01.
2. Plot total loss against scale in ordinary coordinates.
3. Plot log(L-E_0) against log(x/x_ref) for a profile of candidate floors.
4. Inspect local slopes and residuals across the scale axis.
5. Refit after excluding the small-data and transition region.
6. Hide the largest contiguous scales before fitting.
7. Report interpolation and extrapolation error separately.
8. Propagate floor/exponent uncertainty to the target-scale prediction.
9. Reject the law if residual structure or holdout error remains material.

Randomly withholding points inside the observed range mainly tests interpolation. A scaling claim needs a scale holdout: one or more contiguous larger values of x that the estimator did not see. The distance from the fit range to the intended target must also be reported. Predicting the next factor of two is a different claim from predicting six additional orders of magnitude.

0x03 Why A Joint Model-Data Law Has Three Terms

The one-variable analysis in 0x02 treated one scale coordinate at a time. That view is incomplete for training allocation. A model can stop improving because its function class is too restricted, because its finite training stream and optimizer did not find the best predictor in that class, or because the evaluation distribution has an irreducible conditional uncertainty. Increasing only N or only D cannot remove all three limits.

The common joint law is:

$$\widehat L(N,D) = E +A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha} +A_D\left(\frac{D}{D_{ref}}\right)^{-\beta}.$$

This section derives where the three terms come from and, equally important, which step of the argument is only an approximation. It does not yet optimize N and D under a compute budget. Section 0x04 adds that constraint only after the loss surface itself is auditable.

0x03A Start With Population Risk, Not A Curve Fit

Let X denote the available token context, Y the next token, and P the frozen evaluation distribution over (X,Y). A probabilistic predictor f maps each context to a distribution over the vocabulary. For next-token cross-entropy, define the population risk:

$$R_P(f) = \mathbb E_{(X,Y)\sim P} \left[-\log f(Y\mid X)\right].$$

Renderer-safe form:

R_P(f) = E_(X,Y ~ P)[-log f(Y | X)]

The Bayes predictor for this contract is the true conditional distribution:

$$f^*(\cdot\mid x)=P(\cdot\mid x), \qquad E=R_P(f^*).$$

Under this exact setup, E is the conditional entropy H_P(Y|X) in the units set by the logarithm. That statement is narrower than “the entropy of natural language.” Change the tokenizer, context available in X, evaluation corpus, loss weighting, or logarithm base and the numerical risk changes. In a real fit, the estimated asymptote may also absorb model-family and training-policy limits, so it must not be declared Bayes risk merely because the symbol E was used.

Now introduce an architecture-scaling policy pi_arch. It maps a scale choice to a complete model configuration: depth, widths, head counts, MLP ratio, vocabulary treatment, parameter tying, and any other shape decision. Let H_N(pi_arch) be the predictor family produced by that policy at the declared parameter count N. The best population-risk predictor available in that family is:

$$f_N^* = \operatorname*{arg\,min}_{f\in H_N(\pi_{arch})}R_P(f).$$

Finally, a training algorithm does not see P directly. Given a training stream S_D containing D processed tokens and a declared training policy pi_train, it returns an actual predictor:

$$\bar f_{N,D} = \mathcal A_{\pi_{train}} \left(H_N(\pi_{arch}),S_D\right).$$

This last object includes optimizer, batch schedule, learning-rate schedule, precision, initialization, stopping rule, data order, and random seed. It is not automatically the exact empirical-risk minimizer. Hoffmann et al. make this distinction explicit for their data-rich, approximately single-epoch training setup.

0x03B The Exact Part: A Telescoping Risk Decomposition

Add and subtract the risk of f_N^*:

$$\begin{aligned} R_P(\bar f_{N,D}) &=R_P(f^*) \\ &\quad+\left[R_P(f_N^*)-R_P(f^*)\right] \\ &\quad+\left[R_P(\bar f_{N,D})-R_P(f_N^*)\right]. \end{aligned}$$

Define the two gaps:

$$G_{model}(N) = R_P(f_N^*)-R_P(f^*),$$ $$G_{train}(N,D,\pi_{train},S_D) = R_P(\bar f_{N,D})-R_P(f_N^*).$$

Then:

$$L(N,D)=E+G_{model}(N)+G_{train}(N,D,\pi_{train},S_D).$$

Renderer-safe decomposition:

observed population loss
    = Bayes/source-distribution risk
    + finite-function-class gap
    + finite-data-and-training gap

L(N, D)
    = E + G_model(N) + G_train(N, D, pi_train, S_D)

The addition is an identity given the definitions. The names describe three different reasons for excess loss:

Term	Exact object	What can make it large?	What scaling may reduce it?
E	R_P(f*)	irreducible uncertainty under the frozen prediction contract	changing N or D cannot reduce it
G_model	R_P(f_N*) - R_P(f*)	restricted architecture family, finite representation capacity	a better or larger declared model family
G_train	R_P(fbar_N,D) - R_P(f_N*)	finite sample, finite horizon, optimizer error, schedule mismatch, stochasticity	more or better data, more steps, or a better training policy

Calling the last term a data term is convenient but incomplete. It contains both statistical and optimization effects. If two runs process the same number of tokens under different warmup fractions or stopping rules, they need not have the same G_train.

One can introduce the exact empirical-risk minimizer fhat_N,D and telescope again:

$$G_{train} = \left[R_P(\widehat f_{N,D})-R_P(f_N^*)\right] +\left[R_P(\bar f_{N,D})-R_P(\widehat f_{N,D})\right].$$

This separates an empirical-sample object from the algorithm output, but it does not automatically produce two nonnegative population-risk terms. The algorithm output can generalize better than an exact empirical minimizer. If the second gap is instead measured with empirical risk to guarantee nonnegativity, generalization-gap terms must be added when returning to population risk. Collapsing these effects into one fitted D term is a modeling choice, not proof that statistical estimation and optimizer convergence share one exponent.

0x03C The Approximate Part: Power Laws And Separability

To obtain the familiar formula, make two empirical approximations over a declared scale region.

First, approximate the function-class gap by a power law:

$$G_{model}(N) \approx A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha}.$$

Second, make the much stronger claim that the finite-data-and-training gap is approximately independent of N and follows its own power law:

$$G_{train}(N,D,\pi_{train},S_D) \approx A_D\left(\frac{D}{D_{ref}}\right)^{-\beta}.$$

Substitution gives:

$$\boxed{ \widehat L(N,D) = E +A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha} +A_D\left(\frac{D}{D_{ref}}\right)^{-\beta} }$$

with positive A_N, A_D, alpha, and beta.

Renderer-safe form:

L_hat(N, D)
    = E
    + A_N * (N / N_ref)^(-alpha)
    + A_D * (D / D_ref)^(-beta)

Hoffmann et al. write the equivalent unnormalized source form:

$$\widehat L(N,D)=E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}.$$

Their Appendix D.2 motivates the model term through function approximation and the final term through finite-data stochastic optimization. It then fits the exponents rather than fixing them to a theoretical rate. The reported 0.34 model exponent and 0.28 data exponent belong to that paper’s run family, metric, and fitting procedure; they are not constants of transformers.

The normalized form is preferable for teaching and implementation because the amplitudes have visible units:

Quantity	Definition and units
L, E, A_N, A_D	the declared loss unit, such as nats per evaluation token
N	dimensionless count under one parameter convention
D	dimensionless processed-token count under one tokenizer and repetition convention
N_ref, D_ref	named positive reference counts with the same conventions as N, D
alpha, beta	dimensionless elasticities of their respective reducible-loss terms

In the unnormalized source form, A has units of loss * parameter^alpha and B has units of loss * token^beta. Changing from tokens to billions of tokens changes their numerical values. It does not change the underlying surface if the conversion is performed correctly.

The argument therefore has three different epistemic statuses:

exact under the definitions:
    risk telescoping into E + model gap + training gap

empirical model choice:
    each gap behaves as a power law over the selected region

additional separability choice:
    the training gap depends only on D and not on N

Only the first line is an identity.

0x03D A One-Variable Slice Does Not Reveal The Global Floor

The joint form changes the interpretation of every one-variable curve from 0x02. Hold processed tokens fixed at D_0:

$$\widehat L(N,D_0) = \underbrace{ E+A_D\left(\frac{D_0}{D_{ref}}\right)^{-\beta} }_{E_{N\text{-slice}}(D_0)} +A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha}.$$

The apparent floor of the parameter-scaling curve is not E; it is E plus the finite-data/training term. Likewise, at fixed model size N_0:

$$\widehat L(N_0,D) = \underbrace{ E+A_N\left(\frac{N_0}{N_{ref}}\right)^{-\alpha} }_{E_{D\text{-slice}}(N_0)} +A_D\left(\frac{D}{D_{ref}}\right)^{-\beta}.$$

This gives two concrete predictions of the separable model:

train longer at fixed model family
    -> parameter-scaling slices should approach a lower asymptote

increase model capacity at fixed training policy
    -> data-scaling slices should approach a lower asymptote

For a normalized teaching surface with E=1.5, A_N=0.8, A_D=0.6, and alpha=beta=0.5:

Slice	Apparent floor at reference scale	Apparent floor at 16x other resource
vary N, hold D	1.5 + 0.6 = 2.10	1.5 + 0.6/sqrt(16) = 1.65
vary D, hold N	1.5 + 0.8 = 2.30	1.5 + 0.8/sqrt(16) = 1.70

This is why a floor estimated from one fixed-data model sweep must not be inserted as the global E in a joint law. The slice has not observed the limit in the other axis.

0x03E Separability Is A Falsifiable Surface Claim

Write a more general surface as:

$$L(N,D)=E+u(N)+v(D)+w(N,D).$$

The additive law sets the interaction w(N,D) to zero. This says more than “both axes show power laws.” It says that the absolute loss improvement from changing N is the same at every D, and the absolute improvement from changing D is the same at every N.

In continuous log coordinates, additive separability implies:

$$\frac{\partial^2 L} {\partial\log N\,\partial\log D} =0.$$

On an observed 2x2 rectangle with N_1 < N_2 and D_1 < D_2, the equivalent finite-difference contrast is:

$$\Delta_{rect} = L(N_2,D_2)-L(N_2,D_1)-L(N_1,D_2)+L(N_1,D_1).$$

For an additive surface, Delta_rect = 0 apart from noise. Consider:

Joint loss	D_small	D_large
N_small	2.20	2.10
N_large	2.05	1.95

Then:

Delta_rect = 1.95 - 2.05 - 2.10 + 2.20 = 0.00

If the bottom-right observation were 1.90, the contrast would be -0.05. The combined increase in model and data scale would then improve loss more than the two additive effects predict. A positive contrast would indicate the opposite interaction.

This diagnostic requires off-diagonal observations. If every run follows only one path such as D = rho*N, the model and data terms remain confounded: many pairs of exponents and amplitudes can explain the same diagonal. A smooth compute frontier is not evidence that the full two-dimensional surface is separable.

Do not test this claim using the slope of log L; the floor and the other term enter its denominator. Under the normalized additive law, the relevant absolute log-coordinate derivatives are:

$$-\frac{\partial L}{\partial\log N} = \alpha A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha},$$ $$-\frac{\partial L}{\partial\log D} = \beta A_D\left(\frac{D}{D_{ref}}\right)^{-\beta}.$$

The first expression must not change systematically with D; the second must not change systematically with N. Residual heatmaps, rectangle contrasts, and slope-by-slice plots are therefore direct separability checks.

0x03F Rosenfeld: The Model Axis Is A Scaling Policy

Rosenfeld et al. approached the joint surface from a different direction. They began with empirical observations: at fixed dataset size, error often decreases with model scale and saturates; at fixed model size, error often decreases with dataset scale and saturates; and the observed surface is reasonably smooth. Their more general intermediate form allowed amplitudes and exponents to depend on the other axis:

$$\epsilon(m,n) \approx a(m)n^{-\alpha(m)} +b(n)m^{-\beta(n)} +c_\infty.$$

The simpler additive form follows only after treating a, b, alpha, and beta as constants. Rosenfeld et al. also add an envelope for the transition near random-guess error; that is a different boundary condition from the large-scale cross-entropy floor emphasized by the Chinchilla-style law.

Contract axis	Rosenfeld-style error landscape	Hoffmann/Chinchilla Approach 3
response	task error or language-model test loss across several older domains	smoothed language-model training loss, treated as an unbiased test-loss estimate in a data-rich regime
model coordinate	scale m under an explicit width, depth, or other architecture policy	declared parameter count N for one transformer family
data coordinate	dataset examples or words under each task contract	processed training tokens D
motivation	match fixed-axis power-law saturation and smooth joint geometry	telescope expected risk into Bayes, function-class, and stochastic/training gaps
transition treatment	explicit convenient envelope near random-guess behavior	additive large-scale floor plus two reducible terms
strongest warning	a scale number is meaningful only with the policy that constructs the model	the fitted decomposition depends on training and measurement contracts

The scaling-policy point is essential for transformers. Two configurations can have the same total parameter count while assigning capacity differently to depth, residual width, attention heads, MLP width, embeddings, or experts. They are not automatically two realizations of one N axis. A study must either:

freeze one architecture-scaling policy
    -> fit one joint surface

compare several policies as separate strata
    -> test whether coefficients and residuals transfer

register policy identity as an intervention
    -> model the intervention explicitly

Rosenfeld’s result is therefore not evidence that any arbitrary set of models can be collapsed onto parameter count. It is evidence for a constructive joint landscape under declared scaling policies in the paper’s tested settings.

0x03G Tensor And Resource Handoff To M4.1

The loss law consumes scalar coordinates, but those coordinates are produced by concrete tensors and runtime conventions. M4.1 owns the architecture-specific accounting:

model configuration theta
    -> exact stored/trainable parameter count N(theta)
    -> non-embedding count N_no_embed(theta)
    -> active parameter count for routed models
    -> forward FLOPs/token M_tok(theta)
    -> optimizer and activation memory ledgers

training manifest
    -> processed tokens D_total
    -> unique tokens D_unique
    -> repetition and sequence packing

evaluation manifest
    -> loss L in one declared unit and distribution

The empirical joint law must choose one model coordinate. Replacing total parameters with non-embedding parameters, active MoE parameters, or FLOPs per token changes the independent variable and requires a refit. Equal N does not imply equal M_tok; equal analytic training FLOPs does not imply equal wall time or memory traffic.

Section 0x04 introduces a compute constraint such as C = k*N*D. That expression is only a dense-training approximation after N, D, and k have been declared. It must not be smuggled into the definition of the joint loss surface.

0x03H Code Trace: Evaluate Terms And Test A Grid

Keep normalized coordinates in the fit object so amplitudes remain interpretable:

from dataclasses import dataclass

@dataclass(frozen=True)
class JointLaw:
    floor: float
    model_amplitude: float
    data_amplitude: float
    model_exponent: float
    data_exponent: float
    model_reference: float
    token_reference: float

    def terms(self, parameter_count, processed_tokens):
        assert parameter_count > 0
        assert processed_tokens > 0

        model_gap = self.model_amplitude * (
            parameter_count / self.model_reference
        ) ** (-self.model_exponent)

        data_training_gap = self.data_amplitude * (
            processed_tokens / self.token_reference
        ) ** (-self.data_exponent)

        return {
            "floor": self.floor,
            "model_gap": model_gap,
            "data_training_gap": data_training_gap,
            "predicted_loss": self.floor + model_gap + data_training_gap,
        }

The evaluator does not establish that the terms are correct. The run grid must challenge the structural assumption:

def rectangle_contrast(loss, n_small, n_large, d_small, d_large):
    return (
        loss[n_large, d_large]
        - loss[n_large, d_small]
        - loss[n_small, d_large]
        + loss[n_small, d_small]
    )


def audit_joint_grid(records):
    assert len({r.parameter_count for r in records}) >= 3
    assert len({r.processed_tokens for r in records}) >= 3
    assert all_same(records, "architecture_policy_hash")
    assert all_same(records, "training_policy_hash")
    assert all_same(records, "tokenizer_hash")
    assert all_same(records, "evaluation_manifest_hash")

    grid = aggregate_independent_seeds(records)
    contrasts = all_observed_rectangle_contrasts(grid)
    model_slopes_by_data_slice = absolute_log_n_slopes(grid)
    data_slopes_by_model_slice = absolute_log_d_slopes(grid)

    return {
        "rectangle_contrasts": contrasts,
        "model_slopes_by_data_slice": model_slopes_by_data_slice,
        "data_slopes_by_model_slice": data_slopes_by_model_slice,
        "coverage": observed_grid_coverage(grid),
    }

Three values per axis are a minimal implementation check, not a universal sample-size prescription. Replicated seeds and wider log-scale coverage are needed to distinguish interaction from noise and regime transition. The later fitting loop will add positive parameterization, robust objectives, initialization grids, scale holdouts, and uncertainty propagation.

0x03I Joint-Surface Failure Lab

Failure	Smooth fit still possible?	Structural problem	Required diagnostic
only diagonal N,D pairs	yes	model and data terms are not separately identified	add off-diagonal runs and rectangle contrasts
same N, different shapes	yes	parameter count hides architecture-policy changes	stratify by policy or use a registered intervention
fixed steps instead of fixed tokens	yes	sequence length and batch changes alter D	record actual non-padding processed tokens
warmup fraction changes with scale	yes	training gap acquires hidden N,D dependence	report schedule fractions and residuals by scale
data quality changes at large D	yes	D is no longer one homogeneous resource	separate corpus version/mixture or model quality
repeated tokens treated as fresh data	yes	statistical return and optimizer horizon are conflated	retain D_unique, D_total, and repetition count
one slice floor used as global E	yes	the unscaled other resource remains in the asymptote	fit a joint grid and profile global floor uncertainty
interaction hidden by total-loss RMSE	yes	diagonal residual cancellation masks w(N,D)	inspect heatmaps, slices, and mixed differences
compute or FLOPs/token substituted for N	yes	independent variable changes without a refit	derive the coordinate from M4.1 and refit the law

The additive form is useful because it is interpretable and analytically tractable. Those benefits do not excuse a failed interaction audit. When residual structure is material, the appropriate response is a richer model, a narrower regime, or a separate family, not a more confident compute-optimal recommendation.

0x04 Deriving Compute-Optimal Allocation

The joint surface predicts loss for a pair (N,D). A training decision adds a different object: a feasible set of pairs that consume the available compute. The compute-optimal point is the lowest predicted loss on that constraint. It is not the model with the lowest unconstrained loss, the largest model that fits in memory, or a universal token multiplier.

0x04A State The Compute Contract Before Optimizing

For a dense transformer family, a common approximation is:

$$C=kND,$$

where:

Symbol	Meaning
C	declared training-compute budget in FLOPs
N	the parameter coordinate used by the fitted loss law
D	actual processed, non-padding training tokens
k	training FLOPs per parameter-token under the declared convention

The familiar k=6 approximation treats a dense transformer’s forward pass as roughly 2N FLOPs per token and the backward pass as roughly twice the forward cost. It is a useful accounting shorthand, not an architectural identity. Embeddings, output projections, attention’s sequence-dependent work, parameter reuse, routed experts, auxiliary losses, and the exact backward implementation can break proportionality between training FLOPs per token and N.

M4.1 owns the exact alternative:

$$C_{exact}(\theta,D) = D\,M_{train/token}(\theta),$$

where theta is an executable architecture and M_train/token is derived from its tensor operations. Closed forms below require M_train/token(theta) = kN with approximately constant k across the fitted family. If that relation is not adequate, retain the loss law but solve the constrained problem numerically with the exact cost ledger.

Because both reducible terms decrease as their resource increases, the simple data-rich problem uses the full budget. This equality need not hold after adding finite unique data, memory, time, instability, or lifecycle-cost constraints; later sections will treat those as changed decision problems.

0x04B Normalize The Axes And The Budget

Use the normalized joint law from 0x03:

$$\widehat L(N,D) = E +A_N\left(\frac{N}{N_{ref}}\right)^{-\alpha} +A_D\left(\frac{D}{D_{ref}}\right)^{-\beta}.$$

Define dimensionless coordinates:

$$n=\frac{N}{N_{ref}}, \qquad d=\frac{D}{D_{ref}}, \qquad c=\frac{C}{kN_{ref}D_{ref}}.$$

The compute constraint becomes:

$$nd=c.$$

The reducible loss above the floor is:

$$F(n,d) = \widehat L-E = A_Nn^{-\alpha}+A_Dd^{-\beta}.$$

Renderer-safe contract:

n = N / N_ref
d = D / D_ref
c = C / (k * N_ref * D_ref)

constraint: n * d = c
reducible loss: F = A_N*n^(-alpha) + A_D*d^(-beta)

This normalization does not alter the optimum. It makes every power act on a dimensionless ratio and keeps A_N and A_D in loss units.

0x04C Substitute The Constraint And Differentiate

From nd=c:

$$d=\frac{c}{n}.$$

Substitute into the reducible loss:

$$\begin{aligned} g_c(n) &=A_Nn^{-\alpha} +A_D\left(\frac{c}{n}\right)^{-\beta} \\ &=A_Nn^{-\alpha}+A_Dc^{-\beta}n^\beta. \end{aligned}$$

The first term decreases as the model grows. The second increases because a larger model leaves fewer training tokens at fixed compute. Differentiate with respect to n:

$$\frac{dg_c}{dn} = -\alpha A_Nn^{-\alpha-1} +\beta A_Dc^{-\beta}n^{\beta-1}.$$

Set the derivative to zero and multiply by positive n:

$$\alpha A_Nn^{-\alpha} = \beta A_Dc^{-\beta}n^\beta.$$

Since d=c/n, this becomes the central marginal-balance condition:

$$\boxed{ \alpha A_Nn^{-\alpha} = \beta A_Dd^{-\beta} }$$

or, naming the two reducible terms F_N and F_D:

$$\alpha F_N=\beta F_D.$$

Renderer-safe decision contract:

fixed-compute constraint:
    n * d = c

marginal balance:
    alpha * A_N * n^(-alpha)
        = beta * A_D * d^(-beta)
    equivalently: alpha * F_N = beta * F_D

The raw model and data loss terms are not generally equal. Their exponent-weighted marginal returns are equal.

0x04D Why The Stationary Point Is The Unique Global Minimum

The derivative gives a candidate point. To establish that it is the optimum, move to the log-model coordinate z=log n. Along the fixed-compute line, log d=log c-z, so:

$$h_c(z) = A_Ne^{-\alpha z} +A_Dc^{-\beta}e^{\beta z}.$$

Its second derivative is:

$$\frac{d^2h_c}{dz^2} = \alpha^2A_Ne^{-\alpha z} +\beta^2A_Dc^{-\beta}e^{\beta z} >0.$$

Therefore h_c is strictly convex. It also diverges when z goes to either extreme: an arbitrarily tiny model makes the model term diverge, while an arbitrarily large model leaves vanishing D and makes the data/training term diverge. The stationary point is consequently the unique global minimum of the continuous positive problem.

This proof relies on the additive positive-power form. Interactions, saturation, bounded data, or a nonmultiplicative cost function can create a different geometry.

0x04E The Balance Condition Is A Resource-Exchange Test

At fixed compute, increase log n by a small amount delta. Then log d must decrease by the same amount. The directional derivative is:

$$\frac{dh_c}{dz} = -\alpha F_N+\beta F_D.$$

This gives an operational interpretation:

alpha * F_N > beta * F_D
    -> model limitation dominates at the current split
    -> increase N and reduce D

alpha * F_N < beta * F_D
    -> data/training limitation dominates
    -> reduce N and increase D

alpha * F_N = beta * F_D
    -> first-order loss change is zero
    -> the compute line is tangent to the loss contour

At the optimum, the shares of total reducible loss follow directly. Let F*=F_N*+F_D*. From alpha*F_N*=beta*F_D*:

$$F_N^*=\frac{\beta}{\alpha+\beta}F^*, \qquad F_D^*=\frac{\alpha}{\alpha+\beta}F^*.$$

The slower-decaying resource term can occupy a larger share at the optimum. Equal shares occur only when alpha=beta.

0x04F Solve For The Continuous Frontier

Starting from the balance equation:

$$\alpha A_Nn^{-\alpha} = \beta A_Dc^{-\beta}n^\beta,$$

collect powers of n:

$$n^{\alpha+\beta} = \frac{\alpha A_N}{\beta A_D}c^\beta.$$

Define:

$$G = \left( \frac{\alpha A_N}{\beta A_D} \right)^{\frac{1}{\alpha+\beta}}.$$

Then the normalized optimum is:

$$\boxed{ n^*(c)=G\,c^{\frac{\beta}{\alpha+\beta}} }$$

and, because d*=c/n*:

$$\boxed{ d^*(c)=G^{-1}c^{\frac{\alpha}{\alpha+\beta}} }.$$

Returning to physical coordinates:

$$N^*(C) = N_{ref}G \left( \frac{C}{kN_{ref}D_{ref}} \right)^{\frac{\beta}{\alpha+\beta}},$$ $$D^*(C) = D_{ref}G^{-1} \left( \frac{C}{kN_{ref}D_{ref}} \right)^{\frac{\alpha}{\alpha+\beta}}.$$

The exponents are:

$$a=\frac{\beta}{\alpha+\beta}, \qquad b=\frac{\alpha}{\alpha+\beta}, \qquad a+b=1.$$

The sum equals one because the assumed cost is multiplicative in N and D. It is not a general law for objectives with different cost geometry.

Hoffmann et al. use the unnormalized form E+A/N^alpha+B/D^beta. Since A=A_N N_ref^alpha and B=A_D D_ref^beta, the same solution becomes:

$$N^*(C) = \left(\frac{\alpha A}{\beta B}\right)^{\frac{1}{\alpha+\beta}} \left(\frac{C}{k}\right)^{\frac{\beta}{\alpha+\beta}},$$ $$D^*(C) = \left(\frac{\beta B}{\alpha A}\right)^{\frac{1}{\alpha+\beta}} \left(\frac{C}{k}\right)^{\frac{\alpha}{\alpha+\beta}}.$$

Setting k=6 recovers the paper’s closed-form convention.

0x04G Derive The Optimal Loss Exponent

Substitute the frontier into each reducible term:

$$F_N^*(c) = A_NG^{-\alpha} c^{-\frac{\alpha\beta}{\alpha+\beta}},$$ $$F_D^*(c) = A_DG^\beta c^{-\frac{\alpha\beta}{\alpha+\beta}}.$$

Both terms acquire the same compute exponent. Therefore:

$$\widehat L^*(C)-E = K \left( \frac{C}{kN_{ref}D_{ref}} \right)^{-\gamma},$$

where:

$$\gamma = \frac{\alpha\beta}{\alpha+\beta} = \left(\frac{1}{\alpha}+\frac{1}{\beta}\right)^{-1},$$ $$K=A_NG^{-\alpha}+A_DG^\beta.$$

The compute exponent is smaller than either one-variable exponent. Additional compute must be split between model capacity and training tokens, so the frontier improves more slowly than a curve where the other resource is assumed unlimited.

0x04H The Token Ratio Is Usually Not Constant

Divide the physical-coordinate solutions:

$$\boxed{ \frac{D^*}{N^*} = \frac{D_{ref}}{N_{ref}} \left(\frac{\beta A_D}{\alpha A_N}\right)^{\frac{2}{\alpha+\beta}} \left( \frac{C}{kN_{ref}D_{ref}} \right)^{\frac{\alpha-\beta}{\alpha+\beta}} }.$$

Its compute elasticity is:

$$\frac{d\log(D^*/N^*)}{d\log C} = \frac{\alpha-\beta}{\alpha+\beta}.$$

Renderer-safe frontier contract:

s = alpha + beta
G = (alpha * A_N / (beta * A_D))^(1 / s)

n_opt(c) = G * c^(beta / s)
d_opt(c) = G^(-1) * c^(alpha / s)

N_opt(C) = N_ref * n_opt(C / (k * N_ref * D_ref))
D_opt(C) = D_ref * d_opt(C / (k * N_ref * D_ref))

D_opt / N_opt
    = (D_ref / N_ref)
    * (beta * A_D / (alpha * A_N))^(2 / s)
    * (C / (k * N_ref * D_ref))^((alpha - beta) / s)

Therefore:

Exponent relation	Frontier behavior
alpha = beta	token/parameter ratio is constant with compute
alpha > beta	optimal token/parameter ratio grows with compute
alpha < beta	optimal token/parameter ratio shrinks with compute

Even in the equal-exponent case, the constant is not universal. With alpha=beta=p:

$$\frac{D^*}{N^*} = \frac{D_{ref}}{N_{ref}} \left(\frac{A_D}{A_N}\right)^{1/p}.$$

It depends on amplitudes, reference units, parameter convention, tokenizer, data distribution, metric, architecture policy, and training policy. A ratio such as 20 tokens per parameter can describe one chosen run or a local rule of thumb; it does not follow universally from compute optimality.

Using the source-reported rounded Approach 3 exponents from Hoffmann et al., alpha=0.34 and beta=0.28:

model frontier exponent:
    a = 0.28 / (0.34 + 0.28) = 0.4516

data frontier exponent:
    b = 0.34 / (0.34 + 0.28) = 0.5484

ratio elasticity:
    b - a = 0.0968

10x more compute:
    N* multiplies by about 2.83
    D* multiplies by about 3.54
    D*/N* multiplies by about 1.25

The paper summarizes the fitted frontier with rounded exponents near 0.46 and 0.54. The important local lesson is not the final decimal place. It is that approximately equal scaling can still imply a slowly changing ratio.

0x04I Worked Example: Same Compute, Different Splits

Use a normalized teaching law:

E = 1.5
A_N = A_D = 0.8
alpha = beta = 0.5
c = 10,000

Here G=1, so:

$$n^*=d^*=\sqrt{10{,}000}=100.$$

At the optimum:

model gap = 0.8 / sqrt(100) = 0.08
data gap  = 0.8 / sqrt(100) = 0.08
loss      = 1.5 + 0.08 + 0.08 = 1.66

Compare three allocations with the same product n*d=10,000:

Allocation	n	d	Model gap	Data/training gap	Total loss
data-heavy	25	400	0.16	0.04	1.70
balanced optimum	100	100	0.08	0.08	1.66
model-heavy	400	25	0.04	0.16	1.70

For the model-heavy point, alpha*F_N=0.02 while beta*F_D=0.08. The directional test says to reduce model scale and spend the released compute on more tokens. The data-heavy case produces the opposite recommendation.

0x04J Geometric View: Tangency In Log Coordinates

In ordinary (N,D) coordinates, fixed compute is a hyperbola. In (log N, log D) coordinates, it is a line with slope -1:

$$\log D = \log\left(\frac{C}{k}\right)-\log N.$$

An iso-loss contour contains all model/token pairs with the same predicted loss. The compute-optimal point is where the lowest reachable contour is tangent to the compute line. At that point, moving modelward or dataward along the budget has zero first-order benefit, exactly reproducing alpha*F_N=beta*F_D.

The source paper visualizes the fitted contours, empirical runs, efficient frontier, and IsoFLOP slices directly:

Article-authored reconstruction of Hoffmann et al., Training Compute-Optimal Large Language Models, Figure 4. It preserves the source argument connecting fitted contours, empirical support, an efficient frontier, and IsoFLOP minima, but it does not reproduce the paper artwork, exact coordinates, or data points and is not a new measurement. The reported frontier remains conditional on the paper’s model, data, schedule, loss, and FLOP conventions.

The original figure is evidence for the paper’s fitted frontier, not a proof that another run family has the same contour geometry or coefficients.

0x04K From A Continuous Optimum To An Executable Run

N* and D* are continuous real numbers. Transformer architectures, global batches, sequence packs, optimizer schedules, and data shards are discrete. The correct deployment step is a constrained neighborhood search using the exact M4.1 cost ledger:

def continuous_dense_optimum(law, compute_budget, k):
    c = compute_budget / (
        k * law.model_reference * law.token_reference
    )
    total_exponent = law.model_exponent + law.data_exponent
    gain = (
        law.model_exponent * law.model_amplitude
        / (law.data_exponent * law.data_amplitude)
    ) ** (1.0 / total_exponent)

    n_star = gain * c ** (law.data_exponent / total_exponent)
    d_star = gain ** -1 * c ** (
        law.model_exponent / total_exponent
    )

    return (
        law.model_reference * n_star,
        law.token_reference * d_star,
    )

Then round through executable configurations rather than rounding N alone:

def choose_executable_run(law, continuous, budget, architecture_policy,
                          tensor_cost_model, runtime_constraints):
    n_star, d_star = continuous
    candidates = architecture_policy.neighbors(n_star)
    reports = []

    for config in candidates:
        ledger = tensor_cost_model.training_ledger(config)
        max_tokens = floor(budget / ledger.flops_per_processed_token)
        tokens = round_down_to_data_and_batch_quantum(max_tokens, config)

        if tokens <= 0:
            continue
        if not runtime_constraints.memory_fits(config, ledger):
            continue
        if not law.within_validated_region(config.parameter_count, tokens):
            continue

        predicted = law.terms(config.parameter_count, tokens)
        reports.append({
            "config": config,
            "tokens": tokens,
            "exact_training_flops": (
                tokens * ledger.flops_per_processed_token
            ),
            "predicted_loss": predicted["predicted_loss"],
            "model_gap": predicted["model_gap"],
            "data_training_gap": predicted["data_training_gap"],
        })

    return min(reports, key=lambda row: row["predicted_loss"])

The recommendation should retain several nearby candidates if fit uncertainty is larger than their predicted loss difference. A single decimal-rich N* is false precision when the executable frontier is coarse.

The curvature in log-allocation coordinates quantifies local rounding cost. If delta is a small shift in log n along the fixed-compute line:

$$h_c(z^*+\delta) \approx h_c(z^*) +\frac{1}{2} \left(\alpha^2F_N^*+\beta^2F_D^*\right)\delta^2.$$

There is no first-order penalty at the tangent. This often makes a neighborhood of practical shapes statistically indistinguishable, which is another reason to optimize runtime feasibility and uncertainty rather than worship the continuous point.

0x04L Compute-Optimal Failure Lab

Failure	Formula still returns a number?	Why the decision is invalid	Required response
k=6 used with inconsistent N	yes	loss and compute axes count different parameters	reconcile with M4.1 or refit on FLOPs/token
exact cost not proportional to N	yes	fixed-compute path is not ND=constant	solve numerically with C_exact(theta,D)
optimum outside fitted N,D range	yes	allocation is unsupported extrapolation	report distance or add bracketing runs
only one D/N diagonal was fitted	yes	amplitudes/exponents are not separately identified	obtain off-diagonal observations first
unique-data cap ignored	yes	recommended D may require harmful repetition	change to a data-constrained objective
schedule does not match D*	yes	fitted training term no longer describes the run	construct schedule from actual horizon
continuous N* rounded without reevaluation	yes	shape and exact token budget change together	enumerate executable neighbors
memory or wall time treated as FLOPs	yes	the budget is not the operational constraint	add runtime feasibility and cost ledgers
fixed token ratio imposed before fitting	yes	conclusion is encoded as a constraint	fit both axes and test ratio elasticity
coefficient uncertainty omitted	yes	close candidates receive unjustified ranking	propagate bootstrap/posterior draws to allocation

The closed form is valuable precisely because every assumption is visible. When one fails, keep the loss evidence but change the optimization problem.

0x04M Compute-Optimal Allocation Reader Check

Before moving from the joint loss surface to compute-constrained allocation, the reader should now be able to answer:

• Is the plot an optimization trajectory or a cross-run scaling study?
• Which region of the conceptual learning curve is being modeled?
• Which parts of the Hestness three-region picture were observed and which were hypothetical for its real-application experiments?
• What exact objects do N, D_total, D_unique, compute, and loss denote?
• Which fields make two observations members of one controlled family?
• Are checkpoints, seeds, failures, and exclusions represented honestly?
• Which later objective will convert the fitted relation into a decision?
• Which relative-scale assumption yields the power-law form?
• Why is A_ref tied to a declared reference scale?
• Why must the floor be subtracted before the log-linear transformation?
• What multiplicative reduction in excess loss does alpha imply?
• How does a wrong floor bend the apparent log-log slope?
• Do similar interpolation errors conceal divergent target-scale predictions?
• Were the largest contiguous scales hidden during fitting?
• What distribution and loss make R_P(f) the population-risk object?
• When can E be interpreted as conditional entropy, and when is it only a fitted asymptote?
• Which predictor is Bayes-optimal, which is best in H_N, and which one was actually produced by the training algorithm?
• Which part of the three-term decomposition is an identity?
• Which steps introduce power-law and separability assumptions?
• Why does a fixed-D parameter sweep have an apparent floor above global E?
• What are the units of both normalized amplitudes and reference scales?
• Does the run matrix contain off-diagonal N,D combinations?
• Are rectangle contrasts and absolute slice slopes consistent with an additive surface?
• What full architecture-scaling policy gives meaning to N?
• Which M4.1 parameter and FLOP convention supplies the model coordinate?
• Why is k=6 an approximation rather than a transformer identity?
• How do n, d, and c remove unit ambiguity from the derivation?
• Which substitution turns the two-variable constraint into g_c(n)?
• Why does strict convexity in log n establish a unique continuous optimum?
• Why are alpha*F_N and beta*F_D balanced instead of the raw terms?
• Can the constants in N*(C) and D*(C) be reconstructed from A_N, A_D, the references, exponents, k, and C?
• Why do the model and data frontier exponents sum to one here?
• How is gamma=alpha*beta/(alpha+beta) obtained for optimal loss?
• Under exactly which condition is the token/parameter ratio constant?
• Why does a constant ratio still depend on amplitudes and conventions?
• Does the continuous optimum lie inside the validated loss-law region?
• Which executable architecture and token quanta replace the real-valued point?
• Is the exact M4.1 cost ledger close enough to k*N for the closed form to remain decision-useful?

If those questions cannot be answered, the study is not ready to trust a compute-optimal recommendation.

0x05 Three Ways To Estimate The Same Frontier

The derivation in 0x04 assumed that the loss surface was already known. A real scaling study has the opposite problem: it has a finite collection of expensive, noisy training records and must infer the frontier from them. Hoffmann et al. did this three ways:

Approach 1: compare interpolated checkpoints along training trajectories
Approach 2: compare final losses along deliberately constructed IsoFLOP slices
Approach 3: fit one parametric surface to final run records from both families

All three target the same decision,

$$(N_{\mathrm{opt}}(C),D_{\mathrm{opt}}(C)) = \arg\min_{N,D:\,\operatorname{FLOPs}(N,D)=C} L(N,D),$$

but they do not observe the same statistical object. An interpolated checkpoint on a shared optimizer trajectory is not a completed fixed-budget run. A local quadratic valley is not a global loss surface. A robust joint fit is not a model-free summary of the observations.

This distinction is the point of having three estimators. Their agreement is useful triangulation only after their different contracts and shared evidence are made explicit.

0x05A The Observation Contract Changes Before The Fit Does

The following table records what enters each estimator. The numbers and procedures attributed to the paper are source-reported; the explicit invariants and implementation boundaries are rewrite interpretation.

Contract axis	Approach 1: curve envelope	Approach 2: IsoFLOP valleys	Approach 3: joint surface
primary observation	smoothed loss checkpoints along complete training runs	smoothed final loss of each run	final losses from the Approach 1 and 2 experiment families
controlled coordinate	model size plus one of four training horizons	one of nine fixed training-FLOP budgets	joint (N,D) run geometry
source scale	models from about 70M to over 10B; four horizons per size	models up to 16B; budgets from 6e18 to 3e21 FLOPs	valid final records from both families
schedule contract	cosine cycle tied to each run horizon; learning rate drops 10x	cycle length matches the token/FLOP target	inherits the final-run policies that generated its records
preprocessing	Gaussian smoothing with a source-reported window of 10 steps, then interpolation	final loss after smoothing	transform positive coefficients to log space; compute predicted log loss with log-sum-exp
estimator	pointwise minimum across supported trajectories at 1,500 log-spaced budgets	one parabola per budget, then its minimum	Huber objective on log-loss residuals, optimized by multi-start L-BFGS
immediate output	sampled N_opt(C_j), D_opt(C_j) envelope	one local optimum per sampled budget	coefficients E,A,B,alpha,beta and a global analytic frontier
dependence risk	many checkpoints and budget-grid winners reuse one run	points are separate runs, but share recipes and budget construction	reuses final records from the first two experiment families
main failure	schedule mismatch or treating checkpoints as independent runs	no observed rise on both sides of the fitted minimum	functional-form, initialization, residual, and robust-loss sensitivity

The complete paper study reports more than 400 models from below 70M to above 16B parameters and from 5B to above 400B training tokens. That overall range must not be attached to every estimator individually: Approach 1 and Approach 2 use different upper model scales, and Approach 3 combines their final records.

0x05B Approach 1: A Schedule-Qualified Training-Curve Envelope

Approach 1 asks a checkpoint-level question:

At cumulative compute $C_j$, which available, correctly scheduled trajectory has reached the lowest smoothed training loss?

For a run $r$, let the recorded sequence be

$$\mathcal T_r= \left\{ (C_{r,s},D_{r,s},L_{r,s}) \right\}_{s=1}^{S_r},$$

where $s$ is checkpoint index, $C_{r,s}$ is cumulative training compute, $D_{r,s}$ is cumulative processed tokens, and $L_{r,s}$ is the declared loss. After smoothing and interpolation inside the run’s observed support, define

$$\widetilde L_r(C) = \mathcal I_r\!\left( \operatorname{Smooth}(\mathcal T_r) \right), \qquad C\in[C_r^{\min},C_r^{\max}].$$

The envelope winner is then

$$r^*(C_j) = \arg\min_{r:\,C_j\in[C_r^{\min},C_r^{\max}]} \widetilde L_r(C_j).$$

The winning run and interpolation location provide a frontier record:

$$\left( C_j, N_{r^*(C_j)}, D_{r^*(C_j)}(C_j), \widetilde L_{r^*(C_j)}(C_j) \right).$$

Hoffmann et al. trained four separate horizons for each parameter count. Across those horizons, the cosine cycle lengths span a factor of 16 and the learning rate falls by a factor of 10. The source reports Gaussian smoothing with a 10-step window and evaluates the envelope at 1,500 logarithmically spaced FLOP values. It also reports that all selected envelope points occur within the last 15% of their runs.

That last observation matters. A checkpoint at token count $D'$ inside a run whose cosine schedule targets a much later $D$ has not experienced the same learning-rate history as a run designed to end at $D'$. The paper’s separate horizons make schedule length part of the controlled treatment instead of pretending every intermediate checkpoint is a valid short-run endpoint.

Reconstructing The Envelope

def build_training_curve_envelope(runs, compute_grid):
    curves = []

    for run in runs:
        assert within_schedule_tolerance(
            run.schedule_target_tokens,
            run.target_tokens,
        )
        assert strictly_increasing(run.cumulative_flops)
        assert strictly_increasing(run.cumulative_tokens)

        smoothed_loss = gaussian_smooth(
            run.training_loss,
            window_steps=run.declared_smoothing_window,
        )
        curves.append({
            "run_id": run.run_id,
            "N": run.parameter_count,
            "C_min": run.cumulative_flops[0],
            "C_max": run.cumulative_flops[-1],
            "loss_at": interpolate_inside_support(
                run.cumulative_flops,
                smoothed_loss,
            ),
            "tokens_at": interpolate_inside_support(
                run.cumulative_flops,
                run.cumulative_tokens,
            ),
        })

    winners = []
    for C in compute_grid:
        candidates = [
            curve for curve in curves
            if curve["C_min"] <= C <= curve["C_max"]
        ]
        if len(candidates) < 2:
            continue

        winner = min(candidates, key=lambda curve: curve["loss_at"](C))
        winners.append({
            "C": C,
            "N": winner["N"],
            "D": winner["tokens_at"](C),
            "loss": winner["loss_at"](C),
            "source_run_id": winner["run_id"],
        })

    return winners

The power-law frontier is fitted after the winners exist:

$$\log N_j = q_N+a\log C_j+\epsilon_{N,j},$$ $$\log D_j = q_D+b\log C_j+\epsilon_{D,j}.$$

If $D_j$ is instead recomputed from the approximation $C_j=6N_jD_j$, then $a+b=1$ is imposed algebraically. If N and D are both taken from run records and fitted separately, their estimated exponents need not sum to exactly one because of smoothing, interpolation, cost-accounting error, and fit residuals. A reproduction must state which path it uses.

Why 1,500 Winners Are Not 1,500 Replicates

The compute grid makes the frontier visually and numerically dense. It does not create new experimental information. Adjacent winners can come from nearby checkpoints on the same run, and long budget intervals can select the same run:

one optimizer trajectory
    -> many dependent checkpoints
    -> one interpolated curve
    -> many neighboring C_j queries
    -> many repeated source_run_id values

Treating those 1,500 records as independent observations would make ordinary standard errors far too small. The provenance field in the pseudocode is not administrative decoration; it is required for grouped diagnostics and resampling.

The source figure shows the full construction: raw curves on the left, then model and token coordinates extracted from the envelope.

Article-authored reconstruction of Hoffmann et al., Training Compute-Optimal Large Language Models, Figure 2. It preserves the source method from shared training trajectories to the lower envelope and fitted allocation coordinates, but all curves and points are schematic rather than copied or digitized measurements. The dense envelope reuses a much smaller number of trajectories and must not be read as independent replication at every compute query.

0x05C Approach 2: Bracket The IsoFLOP Valley

Approach 2 changes the experiment before changing the fit. Choose a target compute budget $C_m$, vary model size, and assign each model a token horizon that satisfies the same cost contract:

$$D_{m,i} = \frac{C_m}{kN_{m,i}}$$

under the simple approximation, or solve the exact M4.1 cost ledger for the nearest executable token horizon. Every point should finish a schedule matched to that horizon. The comparison is then between final losses, not arbitrary checkpoints from one long trajectory.

For teaching, define a centered log-model coordinate

$$z_i=\log\frac{N_{m,i}}{N_{\mathrm{ref}}}$$

and fit a local quadratic at budget $C_m$:

$$L_m(z)=u_mz^2+v_mz+w_m.$$

Differentiate:

$$\frac{dL_m}{dz}=2u_mz+v_m.$$

When $u_m>0$, the stationary point is a minimum at

$$z_m^*=-\frac{v_m}{2u_m},$$

which maps back to

$$N_m^*=N_{\mathrm{ref}}e^{z_m^*}, \qquad D_m^*=\frac{C_m}{kN_m^*}.$$

The Hoffmann paper states that it fits a parabola to each IsoFLOP curve, but the audited paper text does not specify the coordinate in which that parabola is fitted. Figure 3 displays model size on a logarithmic axis. The z=log N quadratic above is therefore a transparent rewrite interpretation for teaching and implementation, not a claim about source code that the paper does not expose. A reproduction must declare whether its quadratic is in N, log N, or another scaled coordinate because the fitted vertex changes.

A Vertex Is Not Enough: Require Bracketing

An accepted valley needs more than u > 0:

positive curvature:
    u_m > 0

interior optimum:
    min(z_i) < z_m* < max(z_i)

observed support on both sides:
    at least one sampled z_i < z_m*
    at least one sampled z_i > z_m*

local shape evidence:
    losses fall toward the center and rise after it

distance guard:
    z_m* is not supported only by far-away endpoints

A small worked contrast makes the difference concrete. Let $z=\log(N/N_{\mathrm{ref}})$.

Slice	Sampled z	Sampled loss	Quadratic result	Decision
bracketed	[-1, 0, 1]	[2.12, 2.00, 2.08]	u=0.10, v=-0.02, z*=0.10	accept; the minimum lies inside the observations
one-sided	[-2, -1, 0]	[2.50, 2.28, 2.12]	u=0.03, v=-0.13, z*=2.17	reject; all observed losses are still falling toward larger models

The second fit has positive curvature and a finite vertex, yet the implied model is

$$N^*=e^{2.1667}N_{\mathrm{ref}}\approx8.73N_{\mathrm{ref}},$$

far outside the largest sampled model. It is an extrapolated guess, not an observed IsoFLOP valley.

Executable Slice Fitting

def fit_isoflop_slice(records, C_target, N_ref):
    rows = select_same_budget_records(records, C_target)
    rows = sorted(rows, key=lambda row: row.parameter_count)

    assert every_run_finished_its_matched_schedule(rows)
    assert cost_spread(rows) <= declared_budget_tolerance

    z = log(array([row.parameter_count for row in rows]) / N_ref)
    loss = array([row.final_smoothed_loss for row in rows])
    u, v, w = least_squares_quadratic(z, loss)

    if u <= 0:
        return reject("slice is not locally convex")

    z_star = -v / (2 * u)
    if not (z.min() < z_star < z.max()):
        return reject("minimum is outside sampled model range")

    if not observed_points_on_both_sides(z, z_star):
        return reject("minimum is not bracketed")

    N_continuous = N_ref * exp(z_star)
    N_executable = choose_nearby_architecture(N_continuous)
    D_executable = solve_tokens_under_exact_cost(C_target, N_executable)

    return {
        "C": C_target,
        "N_continuous": N_continuous,
        "N_executable": N_executable,
        "D_executable": D_executable,
        "quadratic": (u, v, w),
        "support": (z.min(), z.max()),
    }

Hoffmann et al. report nine compute budgets from 6e18 to 3e21 FLOPs and state that model coverage was chosen to expose a clear minimum in every slice. The source figure shows why that experimental requirement matters: each colored profile descends and then rises across model size.

Article-authored reconstruction of Hoffmann et al., Training Compute-Optimal Large Language Models, Figure 3. It preserves the source experiment’s nine-budget range from 6e18 to 3e21 FLOPs and the requirement to bracket a minimum, but its curves and points are schematic rather than copied or digitized data. A visible valley is direct evidence only inside the sampled model-size range; the later power law still extrapolates those local optima to larger compute.

0x05D Approach 3: Fit A Robust Joint Loss Surface

Approach 3 uses all final losses from the Approach 1 and Approach 2 experiments. It does not place every intermediate Approach 1 checkpoint into the joint fit. Its model is the surface motivated in 0x03:

$$\widehat L(N,D) = E+\frac{A}{N^\alpha}+\frac{B}{D^\beta}.$$

The positive amplitudes and floor can be represented by

$$A=e^a, \qquad B=e^b, \qquad E=e^e.$$

This gives a numerically stable predicted log loss. Start from

$$\widehat L = e^e+e^{a-\alpha\log N}+e^{b-\beta\log D}.$$

Taking the logarithm produces a log-sum-exp:

$$\log\widehat L(N,D) = \operatorname{LSE} \left( e, a-\alpha\log N, b-\beta\log D \right),$$

where

$$\operatorname{LSE}(x_1,x_2,x_3) = m+\log\sum_{j=1}^{3}e^{x_j-m}, \qquad m=\max_jx_j.$$

Subtracting $m$ prevents direct exponentiation of large log-coordinates. This is not merely an algebraic convenience: a naive implementation of the surface can overflow or lose the smaller terms before optimization begins.

For run $i$, define the log-loss residual

$$r_i = \log\widehat L(N_i,D_i)-\log L_i.$$

Using the common Huber convention,

$$h_\delta(r)= \begin{cases} \frac{1}{2}r^2, & |r|\le\delta,\\ \delta\left(|r|-\frac{1}{2}\delta\right), & |r|>\delta. \end{cases}$$

The fitted objective is

$$\min_{a,b,e,\alpha,\beta} \sum_i h_\delta(r_i).$$

Renderer-safe robust-fit contract:

for each included observation i:
    residual_i = log(predicted_loss_i) - log(observed_loss_i)

fit positive E, A, B, alpha, beta by minimizing:
    sum_i huber_delta(residual_i)

the result is accepted only after multistart convergence,
finite-parameter, residual, and held-out checks

Hoffmann et al. report delta=1e-3, L-BFGS, and a grid of initial values:

alpha in {0.0, 0.5, 1.0, 1.5, 2.0}
beta  in {0.0, 0.5, 1.0, 1.5, 2.0}
e     in {-1.0, -0.5, 0.0, 0.5, 1.0}
a     in {0, 5, 10, 15, 20, 25}
b     in {0, 5, 10, 15, 20, 25}

The Cartesian product contains

$$5\times5\times5\times6\times6=4{,}500$$

initializations. The source reports selecting the best local minimum and notes that the selected initialization was not on the boundary of the sweep. A reproduction still needs convergence, finite-value, positive-exponent, and residual checks; “L-BFGS returned” is not a sufficient fit certificate.

def stable_log_prediction(theta, N, D):
    a, b, e, alpha, beta = theta
    terms = stack([
        full_like(N, e),
        a - alpha * log(N),
        b - beta * log(D),
    ], axis=-1)
    return logsumexp(terms, axis=-1)


def fit_joint_chinchilla_surface(final_records, starts, delta=1e-3):
    N = array([row.parameter_count for row in final_records])
    D = array([row.processed_tokens for row in final_records])
    log_observed = log(array([row.final_smoothed_loss for row in final_records]))

    def objective(theta):
        residual = stable_log_prediction(theta, N, D) - log_observed
        return huber(residual, delta=delta).sum()

    candidates = []
    for start in starts:
        result = lbfgs(objective, start)
        if not result.converged or not isfinite(result.objective):
            continue
        if result.alpha <= 0 or result.beta <= 0:
            continue
        candidates.append(result)

    best = min(candidates, key=lambda result: result.objective)
    attach_residuals_and_support_metadata(best, final_records)
    return best

Once the coefficients are fitted, the analytic allocation in 0x04 maps them to $N^*(C)$ and $D^*(C)$. Approach 3 therefore uses observations efficiently, but every frontier point inherits the separable functional form, the log-loss residual definition, the Huber threshold, the optimizer behavior, and the run inclusion policy.

The source-reported fit is

$$\widehat L(N,D) = 1.69 +\frac{406.4}{N^{0.34}} +\frac{410.7}{D^{0.28}}.$$

These coefficients belong to the paper’s parameter units, tokenization, data, model family, schedule, and loss contract. They are not default constants for a new training stack.

0x05E What The Three Estimates Actually Agree On

The source-reported frontier exponents are:

Source method	$a$ in $N_{opt}\propto C^a$	$b$ in $D_{opt}\propto C^b$	Immediate interpretation
minimum over training curves	0.50 (0.488, 0.502)	0.50 (0.501, 0.512)	approximately equal growth from the checkpoint envelope
IsoFLOP profiles	0.49 (0.462, 0.534)	0.51 (0.483, 0.529)	approximately equal growth from bracketed local valleys
parametric loss surface	0.46 (0.454, 0.455)	0.54 (0.542, 0.543)	a somewhat more data-heavy extrapolated frontier
Kaplan et al. comparison row	0.73	0.27	model scale grows much faster than data under that paper’s contract

All numbers in this table are source-reported from Hoffmann et al. Table 2. Parentheses are the paper’s 10th and 90th bootstrap percentiles, based on 100 resamples containing 80% of the dataset. They are not newly recomputed intervals.

The result is stronger than “Chinchilla says 20 tokens per parameter” and narrower than “model and data must always grow equally”:

1. Across the paper’s tested contract, three transformations of related evidence estimate model and data exponents near one half.
2. Approach 1 and Approach 2 produce especially similar frontier slopes even though one uses trajectory checkpoints and the other uses completed fixed-budget runs.
3. Approach 3 predicts smaller models at high compute than the other two. The source attributes this partly to larger residuals at low compute, robust treatment of those points, and observed negative curvature in the empirical frontier.
4. The methods share run families. Agreement is therefore not equivalent to three independent laboratories reproducing the result.
5. As derived in 0x04H, near-half exponents do not by themselves establish a universal token/parameter constant. The ratio also depends on amplitudes, units, and any difference between alpha and beta.

The three methods also extrapolate at different stages:

Method	Interpolation step	Extrapolation step
curve envelope	interpolate loss inside each run’s compute support	fit a frontier power law beyond observed budgets
IsoFLOP	fit a local valley inside sampled model sizes	fit a power law through local optima and project to larger budgets
joint surface	interpolate a declared parametric surface through all final records	evaluate the same surface and analytic frontier outside the run cloud

“Direct” never means “no extrapolation.” Approach 2 directly observes local allocation valleys at nine budgets, but its Gopher-scale recommendation is still a projection from those valleys.

0x05F Uncertainty Must Follow The Run Hierarchy

The natural hierarchy is:

experimental recipe and data snapshot
    -> model configuration / target horizon
        -> independent training run
            -> dependent checkpoints
                -> smoothed and interpolated curve
                    -> queried budget-grid winners

Resampling below the independent-run level creates pseudo-replication. For Approach 1, sampling 1,500 envelope points as though they were independent would preserve almost none of the uncertainty associated with rerunning a model. For Approach 3, treating final records from the two experiment families as unrelated ignores shared architecture, data, optimizer, and accounting choices.

The paper reports sampling 80% of its dataset 100 times to obtain Table 2’s percentiles. The exact grouping unit is not detailed in the main table caption. This article therefore records the reported intervals but does not reinterpret them as configuration-grouped or seed-aware confidence intervals.

A new study should define the resampling unit before seeing the result:

def grouped_frontier_bootstrap(run_table, fit_estimator, trials, rng):
    groups = group_by_independent_run_or_configuration(run_table)
    estimates = []

    for _ in range(trials):
        sampled_groups = rng.choice(groups, size=len(groups), replace=True)
        sampled_rows = concatenate(group.rows for group in sampled_groups)
        estimates.append(fit_estimator(sampled_rows))

    return summarize_percentiles(estimates)

This is a recommended implementation contract, not a reconstruction of an unspecified source bootstrap. Section 0x07 treats grouped bootstrap, held-out large-scale validation, fit stability, and decision intervals in depth.

0x05G Training-System Integration: Preserve Provenance Before Fitting

The estimators cannot be reconstructed from a CSV containing only N, D, and one loss column. The run system should retain at least:

identity:
    study_id, recipe_version, run_id, seed, parent_configuration

model coordinate:
    architecture tuple, parameter convention, exact N, active N if sparse

data coordinate:
    tokenizer, data snapshot, mixture, unique tokens, processed tokens

schedule coordinate:
    target tokens, warmup, maximum LR, cosine horizon, terminal LR

cost coordinate:
    analytic FLOPs, measured FLOPs if available, cumulative device time

observation coordinate:
    step, cumulative tokens, raw training loss, smoothed loss,
    held-out loss if measured, run status

fit provenance:
    inclusion reason, smoothing policy, interpolation policy,
    source_run_id, source_checkpoint_ids, estimator version

Approach-specific planners then produce different run matrices:

def plan_curve_envelope(model_sizes, target_horizons):
    return [
        RunSpec(
            N=N,
            target_tokens=D,
            schedule_target_tokens=D,
            record_dense_checkpoints=True,
        )
        for N in model_sizes
        for D in target_horizons[N]
    ]


def plan_isoflop_slices(compute_budgets, architecture_candidates):
    specs = []
    for C in compute_budgets:
        for architecture in architecture_candidates[C]:
            D = solve_tokens_under_exact_cost(C, architecture)
            specs.append(RunSpec(
                architecture=architecture,
                target_tokens=D,
                schedule_target_tokens=D,
                isoflop_budget=C,
            ))
    return specs

The first planner buys dense information along trajectories but couples its observations. The second buys cleaner local allocation comparisons but spends a new training run at every (C,N) point. A practical study can combine them, but its budget report should show which runs primarily support interpolation, which bracket valleys, and which are reserved for held-out validation.

0x05H Estimator Failure Lab

Failure	Incorrect interpretation	Observable symptom	Required response
longest schedule reused as every shorter run	any checkpoint is a valid endpoint	early checkpoint LR is much higher than in a horizon-matched run	train or calibrate separate schedule horizons
interpolation outside run support	every requested compute has a curve value	winner comes from extrapolated tail	reject unsupported candidate at that budget
dense grid treated as replication	1,500 budget winners imply 1,500 samples	tiny standard errors despite few source runs	retain source_run_id; group uncertainty above checkpoints
envelope winner loses provenance	only (C,N,D,L) is stored	cannot identify repeated source trajectories	persist run and checkpoint lineage
one-sided IsoFLOP slice	positive quadratic curvature proves an optimum	vertex lies outside sampled N range	add models on the missing side or reject the slice
parabola coordinate omitted	quadratic in N equals quadratic in log N	optimum moves when units or transform change	declare coordinate and center/scale it
fixed-budget spread hidden	all slice runs have exactly equal compute	residual cost differences correlate with model size	enforce tolerance or model actual C_i
intermediate A1 checkpoints added to A3	more rows always strengthen joint fit	long runs dominate through many dependent points	follow declared final-record inclusion or model dependence explicitly
Huber threshold omitted	”robust fit” is reproducible	coefficients change materially across library defaults	publish residual definition, Huber convention, and delta
best optimizer return accepted blindly	multi-start guarantees the global fit	boundary exponents, non-finite terms, structured residuals	check convergence, positivity, support, starts, and residuals
three estimates called independent	method count equals replication count	agreement is overstated despite shared runs	describe correlated triangulation
source exponents copied as defaults	the paper’s coefficients are architecture-independent	new pilot residuals drift with scale or data	refit under the new measurement contract

0x05I Frontier Estimation Review Gate

Before treating the three methods as evidence for one frontier, answer:

• What exact observation enters each estimator: checkpoint, final run, or reconstructed surface value?
• Which training horizons have their own matched learning-rate schedules?
• Does every interpolation stay inside one run’s observed compute support?
• How many independent runs produce the dense Approach 1 envelope?
• Can every envelope winner be traced back to a run and checkpoint pair?
• Which coordinate is used for each IsoFLOP quadratic?
• Is every accepted quadratic convex and bracketed by observed model sizes?
• How far does the projected frontier lie beyond the nine source budgets?
• Does the joint fit use final losses only, as the source approach does?
• Can the log-sum-exp objective be reconstructed without numerical overflow?
• Which Huber convention, threshold, initialization grid, and optimizer checks define the robust fit?
• Are the source-reported bootstrap percentiles clearly distinguished from a newly validated grouped uncertainty interval?
• Where do the three methods reuse experiment families?
• Are method-specific exponents shown before any consensus summary?
• Does the executable run planner preserve schedule, cost, and fit provenance?

The three-estimator analysis establishes that Chinchilla’s headline allocation is not one curve fit. It is a comparison among a schedule-qualified checkpoint envelope, directly bracketed fixed-compute valleys, and a robust functional surface. Section 0x06 compares why Kaplan, Chinchilla, DeepSeek, and later controlled reproductions recover different frontiers when parameter counts, FLOP conventions, warmup, optimizer scaling, and data versions change.

0x06 Kaplan Versus Chinchilla Is A Contract Disagreement

The familiar summary says that Kaplan allocates compute mostly to model size, whereas Chinchilla allocates it approximately equally to model size and data. That summary is numerically memorable and experimentally incomplete. The papers did not merely fit different slopes to exchangeable observations. They used different definitions of model size, different FLOP ledgers, different training-horizon logic, different optimizer controls, and different routes from run records to a frontier.

The contrast is still scientifically important. It becomes more useful when it is posed as a set of falsifiable contract differences:

paper name
    is not the independent variable

parameter accounting + FLOP accounting + schedule policy
    + optimizer policy + data version + metric + estimator
        -> observed loss geometry
        -> selected frontier points
        -> fitted allocation exponent

Porian et al. made this causal framing unusually concrete. They started from a Kaplan-like reproduction and changed the output-head compute, warmup, schedule, and optimizer policy in stages. The fitted model-allocation exponent moved from roughly the Kaplan regime to roughly the Chinchilla regime. DeepSeek LLM offers a complementary lesson: replace parameter count with non-vocabulary FLOPs per token, scale batch size and learning rate before running IsoFLOP experiments, and the fitted allocation still changes with the data version.

0x06A An Exponent Is Not A Paper Identity

Suppose a study reports a compute-optimal model frontier

$$N^*(C)=K_N C^a.$$

If that same study also imposes the exact product contract $C=kND$, then the corresponding data allocation is

$$D^*(C) = \frac{C}{kN^*(C)} = \frac{1}{kK_N}C^{1-a}.$$

Under this particular contract, the exponents sum to one:

$$a+b=1.$$

This identity is algebra, not an empirical discovery. It stops being exact if the cost model has scale-dependent attention, vocabulary, sparsity, recomputation, or hardware terms that cannot be absorbed into a constant $k$. It also says nothing about whether the observed optimum is unbiased by the training policy.

The source-reported frontier estimates already show why a paper should not be reduced to one coefficient:

Source result	Model exponent	Data exponent	What the number summarizes
Kaplan empirical fixed-batch frontier	0.88	0.12 under the product proxy	observed compute with the paper’s non-embedding ledger
Kaplan critical-batch-adjusted frontier	0.73	0.27	estimated minimum compute after an empirical batch-efficiency correction
Chinchilla Approach 1	0.50	0.50	minimum over schedule-qualified training curves
Chinchilla Approach 2	0.49	0.51	minima of nine directly sampled IsoFLOP valleys
Chinchilla Approach 3	0.46	0.54	constrained optimum of a robust joint loss surface
DeepSeek current in-house data	0.524 for $M$	0.476	IsoFLOP frontier using non-vocabulary FLOPs/token $M$

The first two Kaplan rows are not interchangeable. Kaplan defines $C_{min}$ as the compute that would be required far below the critical batch size and uses an empirical batch-efficiency model to adjust observed training. Chinchilla’s three rows are not independent replications either; 0x05 showed how they transform overlapping experiment families in different ways. DeepSeek’s model coordinate is $M$, not parameter count $N$.

The fair question is therefore not “which exponent is correct?” It is:

If the same model family is trained under the same data, accounting, schedule, optimizer, and estimator contract, which frontier remains?

0x06B Parameter Count And FLOPs Warp Both Axes

The four studies use four related but non-identical size ledgers.

Kaplan’s non-embedding proxy

For its standard Transformer geometry, Kaplan et al. approximate the non-embedding parameter count as

$$N_K \approx 12n_{layer}d_{model}^2,$$

excluding vocabulary and positional embeddings. The forward cost per token is approximated by

$$C_{forward} \approx 2N_K+2n_{layer}n_{ctx}d_{attn},$$

but the context-dependent term is then omitted from the headline training ledger, giving

$$C_K\approx 6N_KD.$$

The output projection into the vocabulary also does work. In the paper’s tied embedding setup it had no separately counted parameter matrix, and its FLOPs were omitted from the non-embedding proxy. This omission is small for a wide, large model and can dominate a narrow, small model.

Chinchilla’s all-FLOP ledger

Hoffmann et al. count embedding parameters in model size and include embedding, attention, dense-block, and final-logit work in their detailed FLOP ledger. They still use $C\approx6ND$ to derive and display the frontier because the detailed count is close over their experiment family. Their Appendix F reports exact-to-proxy ratios from 0.99 to 1.10 for models from 73M to 6.8B parameters. That empirical closeness is a property of that architecture range, not permission to assume the proxy is exact for every model.

Porian’s switchable head contract

Porian et al. normally define $N_P$ as all linear weights except embeddings, including the untied output head. Only the Kaplan-reproduction stage subtracts $d_{model}n_{vocab}$ from this count. They report that omitting head work underestimates their $6ND$ proxy by roughly 10% for their larger models and by roughly 90% for their smaller models. The error is therefore scale-dependent, which is exactly the kind of error that can rotate a log-log frontier.

DeepSeek’s non-vocabulary FLOPs/token

DeepSeek LLM replaces parameter count with a model-work coordinate $M$. Under the source architecture assumptions, its three compared representations are

$$6N_1=72n_{layer}d_{model}^2,$$ $$6N_2 = 72n_{layer}d_{model}^2 + 6n_{vocab}d_{model},$$

and

$$M = 72n_{layer}d_{model}^2 + 12n_{layer}d_{model}l_{seq}.$$

$N_1$ excludes embeddings. $N_2$ is the complete parameter count. $M$ includes the attention operation’s sequence-dependent work but excludes vocabulary computation, which the paper does not treat as model capacity. Its compute constraint becomes

$$C=MD.$$

This is neither “all FLOPs” nor “all parameters.” It is an intentional hybrid: count sequence-dependent Transformer work, exclude the vocabulary projection, and use the resulting per-token work as the model-scale coordinate.

The scale dependence is not theoretical trivia. One DeepSeek table gives:

Architecture point	$N_1$	$N_2$	$M$ FLOPs/token	$6N_1/M$	$6N_2/M$
8 layers, width 512	25.2M	77.6M	352M	0.43	1.32
24 layers, width 2048	1.21B	1.42B	9.66B	0.75	0.88
80 layers, width 8192	64.4B	65.3B	419B	0.92	0.94

At the smallest point, $6N_1$ misses more than half of $M$, while $6N_2$ overshoots it by 32%. At the largest point both proxies are much closer. A fit over the full range therefore sees a scale-dependent horizontal displacement.

Article-authored reconstruction of DeepSeek-AI, DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, Figure 6. The source reports that $6N_1D$ overpredicts large-scale performance, $6N_2D$ underpredicts it, and $MD$ gives the closest prediction for its two large validation runs. The diagram preserves that three-coordinate comparison but uses schematic curves and markers rather than paper artwork or digitized values.

Deriving the local slope distortion

Let $N$ and $C$ denote latent reference coordinates with a local frontier $N\propto C^a$. A paper reports a transformed size $\widetilde N=g(N)$ and a compute proxy

$$\widetilde C=\frac{C}{r(N)},$$

where $r(N)$ is the scale-dependent ratio between reference and reported compute. Along the frontier,

$$\frac{d\log C}{d\log N}=\frac{1}{a}.$$

The locally observed exponent is therefore

$$\widetilde a = \frac{d\log\widetilde N}{d\log\widetilde C} = \frac{d\log g/d\log N} {1/a-d\log r/d\log N}.$$

If $g(N)=N$ and $r$ is constant, the slope is unchanged. If either transform varies with scale, the reported exponent can move even when the underlying run ordering does not. Porian’s stages change both the reported size and compute coordinates when the head is restored, so this expression explains the mechanism but should not be used to attribute the entire measured shift to one axis analytically.

0x06C A Fixed Warmup Becomes A Scale-Dependent Intervention

Kaplan’s default schedule used a 3,000-step linear warmup and a batch of $2^{19}$ tokens per update. Its warmup therefore processed

$$W_K = 3000\times2^{19} \approx 1.57\times10^9\ \text{tokens}.$$

For a run trained on $D=BS$ tokens, the fraction of training spent in warmup is

$$\phi_{warmup} = \frac{S_w}{S} = \frac{BS_w}{D} = \frac{W}{D}.$$

Under the proxy $C=6ND$,

$$\phi_{warmup} = \frac{6NW}{C}.$$

A fixed 3,000 steps is consequently not a fixed perturbation. At one compute budget it penalizes candidate models with fewer total updates. Across compute budgets, it can consume most or all of the token horizon near the small-scale end of the inferred frontier.

This distorts an IsoFLOP valley asymmetrically. A candidate that would otherwise finish with low loss may end before the optimizer reaches its intended peak learning rate. The fit does not know that the loss is schedule-censored; it records the point as evidence that this (N,D) allocation is poor. At low compute, the selected solution is pushed toward enough tokens to escape warmup. As compute grows and warmup becomes a smaller fraction, optimal tokens grow more slowly, so the fitted model-size exponent becomes steeper.

Porian et al. first replace the fixed warmup-token count with

$$W_P(N)=N.$$

The warmup fraction becomes

$$\phi_{warmup}=\frac{N}{D}=\frac{1}{\rho}, \qquad \rho=\frac{D}{N}.$$

Their observed optimal token horizons are then at least about five times the interpolated warmup duration. In the later target-matched cosine experiment, they cap warmup at

$$W=\min(N,0.2D),$$

so no run can spend more than 20% of its target tokens warming up. These are source-specific heuristics, not universal prescriptions. The general rule is to make warmup fraction an audited variable rather than a hidden consequence of scale.

0x06D Schedule Matching Was Not The Whole Explanation

Chinchilla ties the cosine cycle to each target token/FLOP horizon. It also argues that a schedule calibrated to a much longer run impairs a shorter checkpoint. This motivated a plausible explanation for the Kaplan-Chinchilla gap: the shorter candidate endpoints did not receive an endpoint-matched learning-rate decay.

Porian et al. test that hypothesis after correcting head FLOPs and warmup. In their Kaplan-like schedule, cosine decay reaches zero only after about 131B tokens. Their compute-optimal candidates in the studied range do not exceed 10B tokens, so they experience less than about 1.5% of that long decay.

They then train a separate target-matched cosine schedule for each model and FLOP budget, decaying to 1% of the maximum learning rate at the target. The source-reported exponent moves only from about 0.60 to about 0.57. The fit becomes cleaner, and loss improves more at larger scales, but most of the allocation discrepancy remains.

That result separates two questions:

Question	Porian result
Does target-matched cosine decay improve final loss?	yes, especially as compute grows
Is target-matched decay necessary to recover a near-Chinchilla allocation exponent?	no in this experiment; tuned constant-learning-rate runs reached about 0.50
Is schedule policy irrelevant?	no; warmup and endpoint semantics materially alter which run records are comparable

The safe conclusion is narrower than either slogan. Learning-rate decay affects quality and run economics, but it was not the dominant cause of the frontier slope in this controlled sequence.

0x06E Batch Size, Learning Rate, And beta2 Form One Policy

At fixed token horizon $D$ and batch size $B$, the optimizer sees

$$S=\frac{D}{B}$$

parameter updates. A large fixed batch can leave the low-compute candidates with only a few hundred updates. Those points are not simply smaller versions of a well-trained large run; they inhabit a different optimization regime.

Changing batch size also changes gradient noise, so the learning rate and AdamW second-moment decay cannot be treated as independent constants. AdamW updates the second moment approximately as

$$v_t = \beta_2v_{t-1} + (1-\beta_2)g_t^2.$$

Ignoring startup bias correction, the effective exponential averaging horizon is on the order of

$$\tau_{v}\approx\frac{1}{1-\beta_2}.$$

The rough horizons for beta2 = 0.95, 0.99, and 0.999 are therefore 20, 100, and 1,000 updates. Smaller batches make $g_t^2$ noisier. Porian’s source reports that 0.95 performed poorly for its smaller-batch sweeps and that adding 0.99 and 0.999 exposed cleaner batch-size and learning-rate trends. This is an observed interaction in that setup, not a rule to maximize beta2 for every small batch.

For models from 5M through 108M parameters, with a 220M validation sweep, the paper fits source-specific policies close to

$$B_{seq}(N)\approx0.00037N^{0.703} \approx 160\left(\frac{N}{108\text{M}}\right)^{2/3},$$

and

$$\eta(N) \approx 3.7N^{-0.36} \approx 0.0047\left(\frac{N}{108\text{M}}\right)^{-1/3}.$$

The coefficients inherit the source’s model, sequence, batch, optimizer, and 20-tokens-per-parameter tuning contract. They are examples of a scaling policy, not reusable defaults.

DeepSeek independently fits policies directly against compute:

$$\eta_{opt}(C)=0.3118C^{-0.1250}, \qquad B_{opt}(C)=0.2920C^{0.3271}.$$

It labels any configuration within 0.25% of the minimum generalization error as near-optimal, fits over compute budgets from $10^{17}$ to $2\times10^{19}$ FLOPs, and validates the policy around $10^{20}$ FLOPs. The broad near-optimal band matters: a policy law describes an acceptable region, not a perfectly identified point.

0x06F The Porian Sequence Is Cumulative Evidence

The source study runs more than 900 trainings and repeats the staged comparison on OpenWebText2 and RefinedWeb. Its table reports the following model-frontier exponents:

Cumulative experiment contract	OpenWebText2	RefinedWeb	What changed from the previous stage
Kaplan-like reproduction	0.864	0.835	fixed large batch, long fixed warmup/decay, head excluded
count output-head FLOPs	0.699	0.706	head restored to size and compute proxy
correct warmup	0.603	0.602	warmup tokens scaled with model size
target-matched cosine decay	0.574	0.571	schedule endpoint matched to each budget; no optimizer retuning
optimizer tuning, no decay	0.518	0.497	batch, learning rate, and beta2 selected by scale; constant LR

The rows are a cumulative intervention sequence, not a factorial causal decomposition. For example, the optimizer-tuned row inherits the accounting and warmup corrections but changes several optimizer variables together. The difference between adjacent exponents is evidence that the combined contract matters; it is not an isolated average treatment effect that can be transferred to another architecture.

Source: Porian et al., Resolving Discrepancies in Compute-Optimal Scaling of Language Models, Figure 1, licensed under CC BY 4.0. The desktop image retains the attributed source figure; the mobile image is an attributed responsive adaptation of its five RefinedWeb panels. The paper also reports closely related OpenWebText2 results.

The most important interpretation is not that every modern study must recover 0.50. It is that an allocation exponent can be a property of the training policy family being optimized. A frontier over fixed hyperparameters answers:

Which (N,D) is best under these fixed hyperparameters?

A frontier over tuned scale-dependent hyperparameters answers:

Which (N,D) is best after allowing the training recipe to change with scale?

Those are different optimization problems.

0x06G DeepSeek Adds Data Version To The Contract

DeepSeek’s main IsoFLOP study uses eight compute budgets from $10^{17}$ to $3\times10^{20}$ FLOPs, roughly ten model/data allocations per budget, compute-scaled hyperparameters, and a separate 100M-token validation set. Its reported metric is bits per byte, which reduces tokenizer-vocabulary dependence relative to plain token-average loss but does not remove data-distribution dependence.

For the then-current in-house data, the source fit is

$$M_{opt}(C)=0.1715C^{0.5243}, \qquad D_{opt}(C)=5.8316C^{0.4757}.$$

During model development, the training data changed. The authors reran scaling studies on three data versions and reported:

Data contract	Model-work exponent	Data exponent	Source characterization
early in-house data	0.450	0.550	earlier, lower-quality internal pipeline
current in-house data	0.524	0.476	improved internal pipeline
OpenWebText2	0.578	0.422	smaller corpus with more meticulous processing

The paper interprets the ordering as higher-quality data favoring more model work at scale. That is a source-reported interpretation, not a controlled proof that allocation exponent is a scalar data-quality meter. Data versions can differ in source mixture, duplication, filtering, tokenizer interactions, domain difficulty, repetition, and validation match. Porian et al. later obtain a lower OpenWebText2 exponent under another controlled setup and conjecture that repetition in the DeepSeek experiment may contribute. That conjecture is plausible and unresolved by these two studies alone.

The operational lesson is stricter:

do not attach one scaling-law fit to a mutable dataset name

attach it to:
    immutable data snapshot
    mixture weights
    filtering and dedup versions
    tokenizer and byte/token metric
    repetition policy
    validation distribution

When any of these changes, the old law becomes prior evidence for a new pilot, not a drop-in allocation oracle.

0x06H Paper-By-Contract Comparison

Contract axis	Kaplan	Chinchilla	Porian	DeepSeek LLM
owner question	compute-efficient loss through model, steps, batch, and data laws	optimal model/data allocation under fixed training FLOPs	which controlled choices reconcile the Kaplan and Chinchilla frontiers?	how should model work, data, and major hyperparameters scale in the target pipeline?
model coordinate	non-embedding parameters	complete declared parameter count	linear parameters excluding embeddings; head included except reproduction	non-vocabulary FLOPs/token $M$
compute coordinate	observed $C$ and batch-adjusted $C_{min}$, both non-embedding proxies	detailed all-FLOP ledger; approximately $6ND$ on frontier	approximately $6ND$ with head-count intervention	exactly $MD$ under the paper’s $M$ definition
data	WebText2	MassiveText; C4/GitHub checks	OpenWebText2 and RefinedWeb	changing in-house versions and OpenWebText2
metric	cross-entropy in nats on WebText2 test distribution	smoothed training loss for fits, plus downstream evaluation	log loss on source-specific validation data	validation bits per byte
horizon	fixed long schedule, early stopping, critical-batch correction	target token/FLOP horizons and matched cosine cycles	long Kaplan-like schedule, corrected warmup, matched decay, then tuned constant LR	multi-step schedule intended to support continuation
optimizer scaling	learning-rate rules and post-hoc critical-batch model	model-dependent settings; schedule matched to horizon	direct sweeps for batch, LR, and beta2 by scale	$B(C)$ and $\eta(C)$ fitted before allocation study
frontier evidence	learning-time laws plus theoretical/empirical compute allocation	three frontier estimators	repeated IsoFLOP fits after cumulative interventions	IsoFLOP profiles under $M$, plus data-version comparisons
headline conclusion	model grows faster than data	model and data grow approximately equally	accounting, warmup, and tuning recover a near-Chinchilla exponent	near-equal main fit, but data version changes allocation
scope warning	adjusted and unadjusted compute are different objects	three methods share evidence and differ numerically	stage differences are cumulative and source-specific	$M$, bits per byte, and data versions are source-specific

No row can be removed without changing the interpretation. In particular, comparing 0.73, 0.49, and 0.524 while hiding the model coordinate makes the table look more commensurate than it is.

0x06I Tensor And Training-System Trace

A scaling study should generate its accounting from the same architecture configuration that creates the model. The following sketch keeps three ledgers rather than forcing one number to serve every paper contract:

from dataclasses import dataclass


@dataclass(frozen=True)
class DecoderGeometry:
    layers: int
    width: int
    sequence_length: int
    vocabulary_size: int


@dataclass(frozen=True)
class ScaleLedger:
    non_embedding_parameters: int
    complete_parameters: int
    output_head_parameters: int
    attention_flops_per_token: int
    vocabulary_flops_per_token: int
    total_training_flops_per_token: int
    non_vocabulary_training_flops_per_token: int


def deepseek_style_scale(geometry: DecoderGeometry):
    L = geometry.layers
    d = geometry.width
    s = geometry.sequence_length
    V = geometry.vocabulary_size

    six_n1 = 72 * L * d**2
    six_n2 = six_n1 + 6 * V * d
    non_vocab_flops = six_n1 + 12 * L * d * s

    return {
        "six_non_embedding_parameters": six_n1,
        "six_complete_parameters": six_n2,
        "non_vocabulary_flops_per_token": non_vocab_flops,
        "proxy_ratio_n1": six_n1 / non_vocab_flops,
        "proxy_ratio_n2": six_n2 / non_vocab_flops,
    }

This function is deliberately named deepseek_style_scale, not exact_transformer_flops. Its formula inherits a particular architecture and operation convention. M4.1 owns the complete operator ledger, including multiply-add convention, attention kernels, recomputation, sparsity, and parallel-system overhead.

The experiment contract must be equally explicit:

@dataclass(frozen=True)
class ScalingContract:
    contract_id: str
    architecture_family: str
    size_ledger: str
    flop_ledger: str
    data_snapshot: str
    tokenizer_hash: str
    validation_snapshot: str
    loss_unit: str
    batch_policy: str
    learning_rate_policy: str
    beta2_policy: str
    warmup_policy: str
    endpoint_policy: str
    frontier_estimator: str


def materialize_run(contract, architecture, compute_budget):
    flops_per_token = exact_cost_from_architecture(
        architecture,
        ledger=contract.flop_ledger,
    )
    target_tokens = compute_budget // flops_per_token
    batch_tokens = choose_batch(
        contract.batch_policy,
        architecture,
        compute_budget,
    )
    updates = target_tokens // batch_tokens
    warmup_updates = choose_warmup(
        contract.warmup_policy,
        architecture,
        target_tokens,
        batch_tokens,
    )

    assert 0 <= warmup_updates < updates

    return RunSpec(
        contract_id=contract.contract_id,
        architecture=architecture,
        compute_budget=compute_budget,
        flops_per_token=flops_per_token,
        target_tokens=target_tokens,
        batch_tokens=batch_tokens,
        target_updates=updates,
        warmup_updates=warmup_updates,
        warmup_fraction=warmup_updates / updates,
        learning_rate=choose_lr(contract.learning_rate_policy,
                                architecture, compute_budget),
        beta2=choose_beta2(contract.beta2_policy,
                           architecture, batch_tokens),
        data_snapshot=contract.data_snapshot,
    )

To reproduce the staged discrepancy rather than merely redraw it, construct cumulative contracts and rerun the full frontier estimator after each stage:

def porian_style_contract_sequence(base):
    yield base.replace(
        contract_id="kaplan_like",
        size_ledger="linear_weights_without_head_or_embeddings",
        flop_ledger="6N_without_head",
        warmup_policy="fixed_3000_updates",
        learning_rate_policy="fixed_long_cosine",
        batch_policy="fixed_2^19_tokens",
    )

    yield previous().replace(
        contract_id="count_head",
        size_ledger="linear_weights_with_head_without_embeddings",
        flop_ledger="6N_with_head",
    )

    yield previous().replace(
        contract_id="correct_warmup",
        warmup_policy="tokens_equal_model_size",
    )

    yield previous().replace(
        contract_id="target_matched_cosine",
        learning_rate_policy="cosine_to_1_percent_at_target",
        warmup_policy="min_model_size_or_20_percent_tokens",
    )

    yield contract("correct_warmup").replace(
        contract_id="scaled_optimizer_constant_lr",
        learning_rate_policy="fitted_by_model_size_constant",
        batch_policy="fitted_by_model_size",
        beta2_policy="sweep_0.95_0.99_0.999_by_batch",
    )


for contract in porian_style_contract_sequence(base_contract):
    runs = execute_isoflop_grid(contract)
    frontier = fit_bracketed_frontier(runs)
    publish(contract, runs, frontier)

The pseudocode’s previous() notation is schematic; production code should materialize immutable contract objects and parent IDs. The important invariant is that a fitted exponent always points back to the exact run-generating contract.

Runtime and budget integration

Hyperparameter scaling is not free metadata. It requires sweeps whose compute must appear in the study budget. Porian reports that its hyperparameter sweep cost is comparable to its scaling-law fit experiment, while using a constant instead of cosine schedule avoids the roughly twofold cost of training a separate decayed run for every model-budget pair in that stage. These economics belong in the method comparison:

Design choice	Benefit	Cost / risk
one fixed recipe	cheapest and operationally simple	frontier includes scale-dependent undertraining
tune every (N,D) point	closest to a fully optimized surface	combinatorial sweep cost and selection noise
fit $B(N),\eta(N)$ or $B(C),\eta(C)$	amortizes tuning across the grid	assumes smooth transfer and requires held-out validation
target-matched cosine	clear endpoint semantics and better final loss	separate schedule/run per target budget
constant or horizon-free schedule	supports continuation and checkpoint reuse	can have worse final loss; still needs warmup and scale tuning
exact operator FLOPs	preserves cross-shape accounting	tied to implementation and convention; harder to compare historically
parameter proxy	simple and easy to communicate	scale-dependent error can rotate the frontier

0x06J Contract Failure Lab

Failure	Incorrect conclusion	Observable symptom	Required response
paper name used as contract	Kaplan says 0.73, Chinchilla says 0.5	adjusted and raw Kaplan estimates are mixed	record the exact row, compute object, and estimator
all parameters treated as active work	$6ND$ is exact for every shape	proxy/exact ratio drifts with width or sequence length	log exact per-token FLOPs and proxy ratio per run
output head omitted at small scale	tiny models are compute-cheap	omission grows as width shrinks or vocabulary dominates	count head FLOPs or exclude affected scales with justification
fixed warmup called constant	every run has the same warmup treatment	warmup fraction correlates with compute or model size	report warmup tokens, updates, and fraction
unmatched decay endpoint	an early checkpoint is a completed short run	candidate ends near peak LR or before meaningful decay	train a matched endpoint or label checkpoint semantics
batch size tuned without LR or beta2	batch is a throughput-only knob	loss degrades at small batches despite more updates	sweep coupled optimizer variables and inspect moment statistics
adjacent Porian stages read as isolated effects	exponent difference is the causal effect of one variable	each stage inherits prior changes and may alter several knobs	call the sequence cumulative; run factorial ablations if needed
DeepSeek $M$ called exact all-FLOP compute	vocabulary and runtime work are counted	ledger excludes logits or system overhead	name it non-vocabulary FLOPs/token and link M4.1
data versions pooled	more runs improve one universal fit	residuals cluster by data snapshot	fit versions separately and treat snapshot as contract state
data exponent called a quality score	higher model exponent proves better data	filtering, repetition, or validation match changed too	use controlled data ablations; keep quality claim source-scoped
hyperparameter sweep compute hidden	the frontier study cost is only final grid runs	tuning budget rivals measurement budget	report tuning, fit, validation, and final-pilot compute separately
0.5 called universal	every architecture should scale equally	residuals or held-out pilots drift under a new family	refit and validate under the target contract

0x06K Paper-Contract Comparison Review Gate

Before comparing two allocation laws, answer:

• Does each $N$ include embeddings, positional parameters, and the output head?
• Does each compute ledger include attention, vocabulary logits, and the backward-pass convention?
• Is the proxy-to-exact FLOP ratio approximately constant over the fitted scale range?
• Is the reported Kaplan result observed compute or critical-batch-adjusted minimum compute?
• Which Chinchilla estimator produced the quoted exponent?
• How many tokens and updates does warmup consume at every candidate point?
• Is the learning-rate endpoint matched to the candidate’s actual horizon?
• How many optimizer updates remain after choosing batch size?
• Were learning rate, batch size, and AdamW beta2 tuned jointly or held fixed?
• Is hyperparameter tuning included in the experiment’s compute budget?
• Are Porian’s stages described as cumulative interventions?
• Is DeepSeek’s $M$ identified as non-vocabulary FLOPs/token rather than all training FLOPs?
• Is bits per byte kept distinct from token-average cross-entropy in nats?
• Are data snapshot, mixture, deduplication, repetition, tokenizer, and validation distribution immutable inside one fit?
• Can every fitted frontier be traced to a contract hash and run manifest?
• Does a held-out larger-scale pilot validate the final contract rather than a nearby but different one?

The paper-contract comparison does not resolve compute-optimal scaling into one eternal exponent. It resolves an apparent paper contradiction into measurable experimental choices. Section 0x07 treats the fitting procedure itself with the same discipline: positive parameterization, robust objectives, grouped bootstrap, optimization convergence, and held-out extrapolation.

0x07 Fitting Is An Algorithm

The joint law in 0x05D can be written in one line, but its estimate is not a five-number tuple produced by an interchangeable call to minimize. A fitted scaling law is the output of a numerical and statistical pipeline:

run manifest + inclusion policy
    -> observation and seed weighting
        -> coordinates and positive parameterization
            -> residual definition and robust objective
                -> initialization + optimizer + stopping contract
                    -> convergence and residual diagnostics
                        -> uncertainty procedure
                            -> held-out prediction and allocation decision

Changing any arrow can move the predicted frontier without changing the printed functional form. This section therefore owns the estimation algorithm: how the same declared loss surface is fitted, rejected, resampled, and tested. It does not yet decide whether a different functional form should replace the surface; M4.8 Sections 0x01 and 0x02 own extrapolation and alternative-law selection.

The governing distinction is:

Object	Question it answers	What it does not establish
objective value	which parameters fit the included rows under one declared loss?	that the optimizer reached the best basin
convergence certificate	is the returned point numerically stable under the declared algorithm?	that the functional form is correct
residual audit	what structure did the fitted form leave unexplained?	future-scale prediction accuracy by itself
bootstrap distribution	how does the fitted decision move under a declared resampling population?	uncertainty from unobserved regime changes
scale-ordered holdout	does the frozen estimator predict larger hidden runs?	performance beyond the largest validated scale
target-adjacent pilot	does the rounded recommendation survive one costly local test?	a universal law for another training contract

0x07A Center The Coordinates Before Enforcing Positivity

Section 0x05 used the source-compatible form

$$\widehat L(N,D) =E+AN^{-\alpha}+BD^{-\beta}.$$

The raw coordinates can span many orders of magnitude. Let $N_0$ and $D_0$ be fixed reference scales, preferably geometric means of the fit set, and define

$$n=\log\frac{N}{N_0}, \qquad d=\log\frac{D}{D_0}.$$

Because $N=N_0e^n$,

$$\frac{A}{N^\alpha} = \frac{A}{N_0^\alpha}e^{-\alpha n} =A_0e^{-\alpha n}, \qquad A_0=\frac{A}{N_0^\alpha}.$$

The data term transforms identically with $B_0=B/D_0^\beta$. The model is unchanged:

$$\widehat L(n,d) =E+A_0e^{-\alpha n}+B_0e^{-\beta d}.$$

Centering moves the amplitude parameters from values tied to the arbitrary units parameters=1 and tokens=1 to reducible-loss contributions near the middle of the run cloud. It usually reduces amplitude/slope correlation and makes optimizer tolerances easier to interpret. It does not add evidence or repair a narrow scale range.

Use unconstrained raw parameters while keeping every physical iterate valid:

$$A_0=e^a, \quad B_0=e^b, \quad E=e^e, \quad \alpha=\varepsilon+\operatorname{softplus}(u), \quad \beta=\varepsilon+\operatorname{softplus}(v),$$

where $\varepsilon$ is a small declared positive lower bound. The prediction is then

$$\log\widehat L_i = \operatorname{LSE} \left( e, a-\alpha n_i, b-\beta d_i \right).$$

This recommended implementation is intentionally stricter than a literal reproduction of Hoffmann et al., whose initialization grid included zero for the directly optimized exponents. Post-fit clipping is not equivalent. If an optimizer explores $\alpha<0$ and that changes its path or stopping point, replacing the returned value with zero does not solve the constrained problem.

The log-sum-exp also exposes the local geometry. Define component shares

$$w_E=\frac{E}{\widehat L}, \qquad w_N=\frac{A_0e^{-\alpha n}}{\widehat L}, \qquad w_D=\frac{B_0e^{-\beta d}}{\widehat L},$$

so $w_E+w_N+w_D=1$. The derivatives of the predicted log loss are

$$\frac{\partial\log\widehat L}{\partial e}=w_E, \quad \frac{\partial\log\widehat L}{\partial a}=w_N, \quad \frac{\partial\log\widehat L}{\partial b}=w_D,$$

and

$$\frac{\partial\log\widehat L}{\partial\alpha}=-n\,w_N, \qquad \frac{\partial\log\widehat L}{\partial\beta}=-d\,w_D.$$

For the raw exponent coordinate, the chain rule multiplies the first expression by $\operatorname{sigmoid}(u)$, and similarly for $v$. These derivatives explain three common identifiability failures:

1. If $w_E\approx0$ for every observed run, the floor has almost no gradient.
2. If the observed $n$ range is narrow, a change in $a$ can compensate for a change in $\alpha$.
3. If N and D move together along a thin diagonal, $w_N$ and $w_D$ can trade without enough off-frontier observations to identify each contribution.

The optimizer cannot manufacture curvature that the experiment did not sample. A tiny gradient may mean convergence, a saturated transform, or a parameter direction the data do not identify. The certificate must distinguish them.

def decode_theta(theta, eps=1e-6):
    a, b, e, raw_alpha, raw_beta = theta
    return {
        "log_A0": a,
        "log_B0": b,
        "log_E": e,
        "alpha": softplus(raw_alpha) + eps,
        "beta": softplus(raw_beta) + eps,
    }


def log_prediction(theta, log_N, log_D, log_N0, log_D0):
    p = decode_theta(theta)
    n = log_N - log_N0
    d = log_D - log_D0
    terms = stack([
        full_like(n, p["log_E"]),
        p["log_A0"] - p["alpha"] * n,
        p["log_B0"] - p["beta"] * d,
    ], axis=-1)
    return logsumexp(terms, axis=-1)

0x07B The Objective Defines Which Error Matters

For observation $i$, retain the 0x05 residual

$$r_i=\log\widehat L_i-\log L_i.$$

For small multiplicative errors, $r_i\approx(\widehat L_i-L_i)/L_i$, so equal squared log residuals roughly give equal importance to equal relative loss errors across scales. This differs from squared error in linear loss, which gives the same weight to an absolute error of 0.01 near loss 5.0 and near loss 1.8.

The derivative of the Huber loss from 0x05D is

$$\psi_\delta(r) = \frac{\partial h_\delta(r)}{\partial r} = \begin{cases} r,& |r|\le\delta,\\ \delta\operatorname{sign}(r),& |r|>\delta. \end{cases}$$

Equivalently, its local influence can be written as a residual-dependent weight

$$\psi_\delta(r)=w(r)r, \qquad w(r)=\min\left(1,\frac{\delta}{|r|}\right).$$

The quadratic region learns from ordinary residual magnitude; outside it, the gradient contribution saturates. With the Hoffmann et al. source value $\delta=10^{-3}$, the transition occurs near a 0.1% multiplicative loss error. That is an unusually small threshold and must be reported with the residual units. The source says larger values overfit its low-compute regime and predict held-out larger runs poorly, while smaller values did not materially change its predictions. That is evidence for that run corpus, not a universal Huber default.

Reasonable alternatives answer different questions:

Objective	Effective emphasis	Main risk
linear-loss least squares	large absolute loss deviations	low-quality high-loss runs dominate
log-loss least squares	relative predictive accuracy	one extreme residual has unbounded influence
log-loss Huber	relative accuracy with bounded outlier influence	threshold can silently discard a regime
per-configuration robust mean, then Huber fit	equal configuration influence after seed aggregation	hides the run-to-run distribution unless both levels are reported
heteroscedastic likelihood	scale- or configuration-dependent noise	variance model can absorb functional misspecification
Student-$t$ or hierarchical model	explicit heavy tails and group effects	more assumptions and harder convergence diagnostics

Sum versus mean: same exact optimum, different numerical problem

If there are $m$ rows,

$$J_{sum}(\theta)=\sum_{i=1}^m h_\delta(r_i), \qquad J_{mean}(\theta)=\frac{1}{m}J_{sum}(\theta).$$

In exact arithmetic they have the same minimizer. Their gradients and objective changes differ by a factor of $m$:

$$\nabla J_{mean}=\frac{1}{m}\nabla J_{sum}.$$

An optimizer that stops when the gradient, function decrease, or projected step falls below an absolute tolerance can therefore stop at different parameters. Multiplying an objective by a constant is harmless to the mathematics but not automatically harmless to a finite-precision stopping contract. Tolerances must scale with the reduction convention, or be checked through normalized diagnostics outside the optimizer.

Seed weighting introduces a second, statistical distinction. For configuration $g$ with $s_g$ valid seeds, one possible objective is

$$J(\theta) = \sum_g\omega_g \frac{1}{s_g} \sum_{j=1}^{s_g} h_\delta(r_{g,j}).$$

Setting all $\omega_g=1$ gives each configuration equal influence regardless of seed count. Giving every run row weight one estimates a run-weighted target and lets heavily replicated configurations dominate. Neither is inherently correct; the estimand and replication design decide. The policy must be frozen before residuals are inspected.

0x07C An Optimizer Certificate, Not A Boolean Flag

Hoffmann et al. report L-BFGS over a Cartesian grid of 4,500 initializations, selecting the best local minimum; 0x05D records the exact grid. Their appendix also says the selected initialization was not on the sweep boundary. That is a useful basin search, but success=True and an interior start still do not prove global optimality or parameter identification.

A fit certificate should retain for every start:

initial theta and decoded physical parameters
optimizer, version, precision, line-search policy
sum-or-mean reduction and every stopping tolerance
termination status and message
iterations and function/gradient evaluations
initial and final objective
final normalized gradient norm
minimum/maximum log prediction
decoded alpha, beta, E, A0, B0
distance from the best accepted objective

The best candidate then passes independent checks:

1. All predictions, parameters, gradients, and objective terms are finite.
2. The decoded exponents and amplitudes obey the declared domain by construction, not clipping.
3. The normalized gradient is below a published tolerance.
4. Re-running from the candidate with tighter tolerances does not materially lower the objective.
5. Several materially different starts reach the same objective basin.
6. Centering or an equivalent rescaling maps back to the same physical surface.
7. Small data perturbations do not send parameters to unrelated boundaries while leaving fit loss unchanged.
8. Residuals and component shares reveal adequate support for the estimated floor and slopes.

Near-equal objective values with very different parameters are not competing “answers” to average. They are evidence for a ridge. Inspect the Hessian or a profile objective, but treat either as a local diagnostic rather than proof:

def certify_candidate(result, objective, normalized_grad, theta_scale):
    checks = {
        "optimizer_status": result.converged,
        "finite": all_finite(result.theta, result.value, result.gradient),
        "gradient": normalized_grad(result) <= 1e-7,
    }

    tighter = lbfgs(
        objective,
        result.theta,
        gtol=1e-10,
        ftol=1e-14,
        max_iter=10_000,
    )
    checks["tighter_restart"] = (
        abs(tighter.value - result.value)
        <= 1e-9 * max(1.0, abs(result.value))
    )

    hessian = autodiff_hessian(objective, tighter.theta)
    checks["local_curvature"] = summarize_eigensystem(
        hessian,
        coordinate_scale=theta_scale,
    )
    return FitCertificate(checks, tighter)

The numerical constants above are teaching placeholders. Production thresholds must be calibrated to objective reduction, dtype, row count, and desired prediction accuracy. Publishing them is more important than copying these particular values.

L-BFGS is attractive because five parameters do not need a full stored Hessian, and automatic differentiation gives accurate gradients. Full BFGS, trust-region methods, or derivative-free basin checks can be useful independent implementations at this dimension. Agreement across algorithms is strongest when they share only the declared objective, not copied stopping behavior.

0x07D The Observation Table Is Part Of The Estimator

The fit table should have one declared statistical role per row. A final loss from an independent seed is not exchangeable with a smoothed checkpoint from the same trajectory, and an OOM does not disappear because it lacks a loss.

Row state	Keep in manifest?	Enter numeric surface fit?	Required treatment
completed, valid endpoint	yes	yes	retain exact run/configuration/seed IDs
intermediate checkpoint	yes	no for source-style Approach 3	use only in an explicitly trajectory-aware estimator
diverged with non-finite loss	yes	normally no	report failure rate and run a missingness sensitivity
OOM before training	yes	no	mark infeasible architecture/runtime region
preempted before endpoint	yes	no unless censoring model declared	do not substitute last checkpoint silently
protocol violation	yes	no	immutable exclusion reason and discovery time
statistically unusual but valid run	yes	yes by default	predeclared robust loss; exclusion sensitivity if investigated

Failures matter even when they cannot contribute a finite residual. If large models fail more often, fitting only completed runs conditions on a scale-dependent event. At minimum, publish completion probability by scale and repeat the decision analysis under plausible bounds for missing outcomes. Assigning an arbitrary giant loss to every OOM is also wrong unless the target decision explicitly combines quality and feasibility in one censored utility.

Seed aggregation is an estimand choice

Three common contracts are:

run-weighted:
    every valid seed is one objective row
    estimates performance of a randomly selected run from the sampled design

configuration-weighted:
    robustly aggregate seeds, then give every (N,D,recipe) point equal weight
    estimates the declared configuration grid rather than the replication budget

hierarchical:
    retain seed rows and model configuration/seed variance explicitly
    estimates both surface and run-to-run variation, at greater model complexity

The first two coincide only when each configuration has the same seed count and noise. Report both seed-level variation and the point estimate used by the surface. Never average checkpoints and call them seeds.

Outliers require a timeline, not only a threshold

An auditable policy distinguishes:

1. protocol errors, identified from logs or metadata independently of the fit;
2. valid heavy-tail outcomes, retained under the robust observation model;
3. candidate regime changes, retained and analyzed as structured residuals;
4. post-hoc exclusions, shown both removed and restored.

Every final fit should produce residual panels against at least log N, log D, actual compute, log(D/N), seed, data version, schedule horizon, run status, and experiment family. A random-looking aggregate histogram can hide a monotone residual drift within one data snapshot or at the largest scale.

def build_fit_records(run_manifest, policy):
    records = []
    for run in run_manifest:
        if run.status != "completed_valid_endpoint":
            record_nonfit_status(run)
            continue

        records.append(FitRecord(
            N=run.model.parameter_count,
            D=run.data.processed_tokens_actual,
            C=run.compute.analytic_training_flops,
            loss=run.evaluation.final_heldout_loss,
            configuration_id=run.config_hash,
            run_id=run.run_id,
            seed=run.optimization.seed,
            data_version=run.data.corpus_version,
            schedule_horizon=run.optimization.schedule_horizon_tokens,
            inclusion_reason=policy.reason_for(run),
        ))

    assert policy.was_frozen_before_residual_review
    return apply_declared_seed_weighting(records, policy)

0x07E Bootstrap The Independent Unit And The Entire Pipeline

The source uncertainty procedures establish useful historical facts but not a complete dependence-aware template:

• Hoffmann et al. report sampling 80% of the dataset 100 times to obtain the 10th and 90th percentiles in Table 2. The table and main description do not specify seed/configuration/checkpoint grouping.
• Besiroglu et al. sample 240 digitized points with replacement and refit 4,000 bootstrap datasets. They did not possess the original run hierarchy; their unit is necessarily the reconstructed figure point.

A new study with run metadata should do better. First decide what population the uncertainty should cover. If compute budgets are fixed experimental strata, resample configuration groups within each budget or design region. Then resample independent seeds within a selected configuration if seed variability is part of the target. Keep all checkpoints and derived rows attached to their parent run.

For bootstrap replicate $q$:

$$\mathcal R^{(q)} \xrightarrow{\text{preprocess}} \mathcal T^{(q)} \xrightarrow{\text{multistart fit}} \widehat\theta^{(q)} \xrightarrow{\text{allocation solve}} \left(N^{*(q)}(C),D^{*(q)}(C)\right).$$

Renderer-safe grouped-bootstrap pipeline:

resample complete configuration groups within frozen design strata
    -> retain all dependent seeds/checkpoints under each selected group
    -> rerun preprocessing and the certified multistart fit
    -> solve N_opt(C), D_opt(C) for every registered budget
    -> retain successful allocations and rejected replicate reasons

The arrows are important. Refitting only a linearized covariance around the original optimum misses basin changes, outlier-selection sensitivity, seed aggregation, and any estimator-selection step. A bootstrap must rerun all data-dependent operations that would have changed had those run groups been the observed corpus.

def grouped_bootstrap(studies, budgets, trials, rng):
    outputs = []

    for bootstrap_id in range(trials):
        sampled = []
        for stratum in studies.strata("compute_or_design_region"):
            groups = stratum.configuration_groups
            chosen = rng.choice(groups, size=len(groups), replace=True)

            for copy_id, group in enumerate(chosen):
                seeds = group.independent_seed_runs
                selected_seeds = rng.choice(
                    seeds,
                    size=declared_seed_sample_size(group),
                    replace=True,
                )
                sampled.extend(clone_group_with_ids(
                    selected_seeds,
                    bootstrap_id=bootstrap_id,
                    group_copy=copy_id,
                ))

        fit = fit_and_certify_complete_pipeline(sampled)
        if not fit.accepted:
            outputs.append(BootstrapFailure(
                bootstrap_id,
                fit.rejection_reasons,
            ))
            continue

        allocations = [
            solve_training_optimum(fit.theta, C, studies.compute_contract)
            for C in budgets
        ]
        outputs.append(BootstrapSuccess(
            bootstrap_id,
            fit.theta,
            fit.diagnostics,
            allocations,
        ))

    return outputs

Report the attempted, converged, certified, and rejected replicate counts. If bootstrap failures are silently removed, an unstable estimator can manufacture a deceptively clean interval from only its easiest resamples. Sensitivity axes should include the grouping unit, seed weighting, robust threshold, inclusion policy, and estimator initialization contract.

This is a recommended procedure, not a claim that either historical paper used these groups. A block, wild, parametric, or hierarchical bootstrap may be more appropriate for a different observation process. The invariant is that the resampling unit follows genuine independent variation rather than row count.

0x07F Propagate Uncertainty Into The Frontier

For each certified fit $q$, the allocation exponents from 0x04 are

$$a^{(q)} = \frac{\beta^{(q)}}{\alpha^{(q)}+\beta^{(q)}}, \qquad b^{(q)} = \frac{\alpha^{(q)}}{\alpha^{(q)}+\beta^{(q)}}.$$

The amplitudes also matter. Solve the complete expressions for $N^{*(q)}(C)$ and $D^{*(q)}(C)$ rather than inserting median coefficients into the formula. In general,

$$\operatorname{median}_q N^*(C;\theta^{(q)}) \ne N^*\left(C;\operatorname{median}_q\theta^{(q)}\right).$$

Nonlinear transforms do not commute with quantiles. The same rule applies to the token/parameter ratio and predicted optimal loss.

Slope uncertainty grows with extrapolation distance. Relative to a validated reference compute $C_0$,

$$\log\frac{N^*(C)}{N^*(C_0)} =a\log\frac{C}{C_0}.$$

A small perturbation gives

$$\delta\log N^*(C) \approx \log\frac{C}{C_0}\,\delta a.$$

At a million-fold compute extrapolation, $\log(C/C_0)\approx13.82$. An exponent uncertainty of only 0.02 therefore produces about 0.276 log units, or a multiplicative factor of $e^{0.276}\approx1.32$, before amplitude uncertainty, functional-form uncertainty, and rounding are added.

Report three different interval objects:

Interval	Construction	Interpretation
coefficient interval	quantiles of $\theta^{(q)}$ or a declared likelihood/posterior procedure	uncertainty in fitted coordinates under that procedure
mean-law interval	quantiles of $\widehat L(N,D;\theta^{(q)})$	uncertainty in the fitted mean surface
prediction interval	mean-law uncertainty plus seed/run noise and declared process variation	range for a new run under the same contract

A bootstrap band from one functional form is not protection against a future slope break. Pointwise 5th/95th percentiles at each compute are also not a 90% simultaneous band for the entire curve.

Finally map every fit to executable configurations. Let $\mathcal F(C)$ be the architectures and token budgets feasible under divisibility, memory, runtime, and data constraints. For candidate $x\in\mathcal F(C)$, define bootstrap regret

$$R^{(q)}(x;C) = \widehat L(x;\theta^{(q)}) - \min_{z\in\mathcal F(C)} \widehat L(z;\theta^{(q)}).$$

Choose with the regret distribution, not only distance from median $N^*$. A slightly non-median architecture can have lower tail regret, fit tensor-parallel shards, and leave a safer data budget.

0x07G Hold Out Scale, Not Random Rows

Randomly splitting a dense scaling grid leaks the geometry that the law is supposed to extrapolate. Neighboring seeds, checkpoints, and configurations can land on both sides; the test then measures interpolation through an already observed scale region.

Use a scale-ordered protocol:

fit set:
    complete configuration groups at lower and middle compute
    all preprocessing, objective, and candidate-law choices happen here

hidden large-scale set:
    one or more larger complete groups
    no seed, checkpoint, or schedule variant of a hidden configuration in fit

target-adjacent pilot:
    train only after estimator, intervals, and decision gate are frozen
    compare actual loss, cost, and feasible-choice regret with prediction

Alabdulmohsin et al. give a concrete warning outside language modeling. On one ViT/B/16 learning curve, several exponents fit the observed interval similarly; the source reports that an interpolation-selected exponent near -0.24 differs from an extrapolation-selected exponent near -0.40. The numbers are task-specific. The general lesson is that choosing a parameter by fit-region error and validating it by future-scale error are different procedures.

For hidden runs $j=1,\ldots,m$, report

$$\operatorname{RMSE}_{log} = \sqrt{ \frac{1}{m} \sum_{j=1}^{m} \left( \log\widehat L_j- \log L_j \right)^2 },$$

plus signed bias, median/maximum relative error, and prediction-interval coverage. Coverage from only two or three expensive runs is descriptive, not a well-estimated probability.

Fit quality and decision quality remain separate. If a held-out IsoFLOP slice contains feasible candidates $x_j$, define the observed decision regret of the predicted choice $\widehat x$ as

$$R_{heldout}(C) = L(\widehat x) - \min_{x_j\in\mathcal F_{observed}(C)}L(x_j).$$

Also report $\widehat N/N_{best}$, $\widehat D/D_{best}$, and whether the empirical best is bracketed by the uncertainty set. Two coefficient vectors can look different but choose the same rounded architecture; two nearly identical fit losses can choose materially different large-scale runs.

def scale_ordered_evaluation(configuration_groups, hidden_fraction):
    ordered = sorted(
        configuration_groups,
        key=lambda group: group.declared_scale_coordinate,
    )
    cut = floor((1 - hidden_fraction) * len(ordered))
    fit_groups, hidden_groups = ordered[:cut], ordered[cut:]

    assert disjoint_parent_runs(fit_groups, hidden_groups)
    assert disjoint_configurations(fit_groups, hidden_groups)

    estimator = choose_and_freeze_estimator(fit_groups)
    fit = fit_and_certify_complete_pipeline(fit_groups, estimator)
    intervals = grouped_bootstrap_fit(fit_groups, estimator)

    predictions = predict_hidden_groups(fit, intervals, hidden_groups)
    return {
        "log_rmse": log_rmse(predictions, hidden_groups),
        "relative_error": relative_errors(predictions, hidden_groups),
        "coverage": interval_coverage(predictions, hidden_groups),
        "frontier_regret": observed_frontier_regret(predictions, hidden_groups),
    }

The final pilot is not another chance to tune the law while calling the result validation. A failed gate can trigger a documented refit, but the previous prediction, miss, and decision must remain in the study record.

0x07H Case Study: The Approach 3 Replication

Besiroglu et al., Chinchilla Scaling: A Replication Attempt, is valuable here because it keeps the functional form and robust objective approximately fixed while auditing data reconstruction, printed precision, optimizer convergence, and interval width. Its evidence must be read in layers.

What data were actually available

The replication did not receive DeepMind’s original run table. It partially reconstructed Figure 4 from the arXiv PDF/SVG:

1. scatter-point SVG coordinates were mapped to model size and training FLOPs;
2. fill colors were mapped through the plotted logarithmic loss color scale;
3. token counts were inferred from the paper’s compute relation;
4. five high-loss points with very low tokens per parameter were excluded from the main fit, then restored in an appendix sensitivity.

The paper identifies two reconstruction limits: the model-size axis lacked ordinary ticks, and the 256-level color representation limited loss precision to about 0.01. Its main table uses 240 reconstructed points, compared with the more than 400 models reported by Hoffmann et al. These are not small details: the replication can audit whether published parameters plausibly fit the visible subset, but it cannot recreate missing runs, exact losses, seeds, or source grouping.

Rounding changes a power law multiplicatively

The body of Hoffmann et al. rounded the data exponent from a more precise TeX-comment value near 0.2849 to 0.28. The finite-data term contains $D^{-\beta}$, so the ratio induced by a rounding error $\Delta\beta$ is

$$\frac{D^{-(\beta-\Delta\beta)}}{D^{-\beta}} =D^{\Delta\beta}.$$

At $D=10^{11}$ and $\Delta\beta=0.0049$,

$$D^{\Delta\beta}-1 = (10^{11})^{0.0049}-1 \approx0.13.$$

Thus a visually small exponent rounding changes that component by about 13% at a typical large token count. Reporting rounded coefficients can be adequate for a prose summary while being inadequate for reconstructing predictions. Publish machine-readable parameters, units, covariance/bootstrap draws, and code revision separately from display rounding.

Same mathematical objective, incomplete numerical minimization

The replication reports the following fits on its reconstructed main subset:

Quantity	replication estimate	Hoffmann precise value recovered from source	Hoffmann body display
$A$	482.01	406.4	406.4
$B$	2085.43	410.7	410.7
$E$	1.8172	1.6934	1.69
$\alpha$	0.3478	0.3392	0.34
$\beta$	0.3658	0.2849	0.28
$a=\beta/(\alpha+\beta)$	0.5126	0.454	approximately 0.452 from displayed exponents

These are source-reported replication results on digitized data, not a refit performed for this article.

Using the precise source coefficients removes much of the visually obvious residual shift, but the replication reports that its converged fit still has a better objective on the reconstructed points. In the updated paper, the reported explanation is numerical: averaging Huber terms changed the scale seen by the L-BFGS-B stopping machinery, which terminated before reaching the better minimum. The key lesson is not that a mean objective has a different true argmin. It is that reduction, optimizer tolerances, and convergence checks form one algorithm.

The replication used BFGS and says it could reach the local minimum without the full start grid in its bootstrap fits. A new implementation should still run independent algorithms and starts; success on this five-dimensional reconstruction does not make single-start BFGS universally sufficient.

The narrow interval was also a stopping artifact

Besiroglu et al. bootstrap their 240 reconstructed points 4,000 times and report a standard error of 0.018 for the model-allocation exponent $a$. They contrast the resulting approximate 80% interval width of 0.05 with the Hoffmann et al. Approach 3 interval from 0.454 to 0.455. Under an independent-observation square-root sample-size heuristic, they estimate that an interval roughly 50 times narrower would require on the order of 600,000 runs. This is a source argument, not a newly verified sample-size theorem for a hierarchical training corpus.

The updated replication attributes the historical narrow bootstrap movement to the same premature optimizer termination: resampled fits stayed close to their initialization instead of fully responding to each sample. This creates a general audit test:

for every bootstrap replicate:
    record start -> final parameter displacement
    record start -> final objective reduction
    verify normalized gradient and tighter-restart stability
    compare accepted basin across independent optimizers

An interval cannot be more trustworthy than the optimization of each draw.

Source: Besiroglu et al., Chinchilla Scaling: A Replication Attempt, Figure 5, licensed under CC BY 4.0. The desktop image retains the attributed source figure; the mobile view is an attributed qualitative adaptation for legibility.

The source figure propagates parameter uncertainty into the tokens-per-parameter policy. Its broad blue band includes the Chinchilla model and the D/N=20 reference over the illustrated region, while the historical Approach 3 point estimate rises toward a much more data-heavy ratio. The replication reports a point estimate near 25.6 tokens per parameter and a range roughly 4 to 40 at 10^26 FLOPs or more. Those values inherit digitization, IID point-bootstrap, functional-form, and outlier assumptions.

The bounded conclusion is:

Supported by the case study	Not supported by the case study
printed coefficient precision can materially change extrapolation	every Chinchilla result is false
optimizer reduction and tolerances can corrupt point and bootstrap fits	the reconstructed 240 points equal the original corpus
uncertainty must be propagated into allocation policy	IID point resampling is ideal when run groups are available
Approach 3’s reported narrow interval deserved re-audit	one corrected fit establishes a universal 20-token ratio
converged Approach 3 can be compatible with Approaches 1 and 2	functional-form uncertainty has disappeared

The original Chinchilla result had two other estimators, and later controlled work in 0x06 finds near-half allocation under several modern contracts. The replication is therefore best read as an audit of one published estimation path, not a refutation of the broader empirical program.

0x07I End-To-End Fit Artifact

A publishable fit should be reproducible from a manifest and emit more than parameters:

@dataclass(frozen=True)
class FitSpec:
    formula_version: str
    coordinate_reference: tuple[float, float]
    residual: str
    robust_objective: str
    huber_delta: float
    reduction: str
    seed_weighting: str
    inclusion_policy_hash: str
    optimizer: str
    optimizer_version: str
    starts_hash: str
    gtol: float
    ftol: float
    max_iter: int


@dataclass
class FitArtifact:
    spec: FitSpec
    run_manifest_hash: str
    selected_theta: array
    decoded_parameters: dict
    all_start_results: list
    certificate: FitCertificate
    component_shares: table
    residual_table: table
    support_bounds: dict
    hidden_scale_metrics: dict
    bootstrap_artifact_uri: str
    code_commit: str


def fit_scaling_study(run_manifest, fit_spec, holdout_spec):
    validate_run_contract(run_manifest)
    fit_groups, hidden_groups = split_complete_groups(
        run_manifest,
        holdout_spec,
    )

    records = build_fit_records(fit_groups, fit_spec)
    objective = build_stable_objective(records, fit_spec)
    starts = materialize_initializations(fit_spec.starts_hash)

    candidates = parallel_map(
        lambda start: optimize_and_record(objective, start, fit_spec),
        starts,
    )
    accepted = [
        certify_candidate(candidate, objective)
        for candidate in candidates
    ]
    selected = select_best_certified(accepted)

    diagnostics = audit_residuals_and_support(selected, records)
    bootstrap = grouped_bootstrap(
        fit_groups,
        budgets=holdout_spec.decision_budgets,
        trials=holdout_spec.bootstrap_trials,
        rng=seeded_rng(holdout_spec.bootstrap_seed),
    )
    hidden_metrics = evaluate_frozen_fit(
        selected,
        bootstrap,
        hidden_groups,
    )

    return persist_fit_artifact(
        selected,
        candidates,
        diagnostics,
        bootstrap,
        hidden_metrics,
        fit_spec,
        run_manifest,
    )

For an expensive study, checkpoint the bootstrap itself, pin deterministic resample IDs, and save rejected replicates. Parallel completion order must not change random draws. Store full-precision machine-readable coefficients and predictions; round only in the article table.

0x07J Design Choices Must Remain Visible

Decision	Option A	Option B	Required comparison
exponent domain	direct bounded coordinates	softplus positive coordinates	physical predictions and convergence, not raw theta
coordinate scale	raw log N, log D	centered log ratios	condition number and mapped-back surface
robust fit	Huber in log loss	log least squares / heavy-tail likelihood	hidden-scale error and residual structure
row weighting	every run equally	every configuration equally	estimand plus seed-count sensitivity
uncertainty	local Hessian/delta method	grouped full-pipeline bootstrap	basin stability and nonlinear allocation bands
optimizer	multi-start L-BFGS	BFGS/trust-region independent check	objective, normalized gradient, and predictions
outliers	robustly retain valid runs	predeclared protocol exclusion	restored-run sensitivity
validation	random row split	scale-ordered group holdout	leakage audit and frontier regret

No single column is always correct. What is unacceptable is allowing a library default to choose silently and then reporting only the final exponent.

0x07K Fitting Failure Lab

Failure	Why the result looks plausible	Observable symptom	Required response
raw positive parameters clipped after optimization	displayed law has positive coefficients	optimizer visited invalid geometry; clipped point has nonzero gradient	transform parameters or solve a declared constrained problem
uncentered coordinates accepted blindly	log-sum-exp stays finite	amplitudes/slopes are highly correlated; condition number explodes	center coordinates and verify mapped predictions
objective reduction omitted	sum and mean have same mathematical argmin	one version stops near initialization	publish reduction and scale-aware stopping checks
one successful start selected	five dimensions feel easy	equal-fit basins or boundary solutions appear from other starts	retain all starts and independent optimizer check
optimizer success treated as convergence proof	library returned no exception	tighter restart materially lowers objective	normalized gradient and restart certificate
configurations with more seeds dominate	more rows seem like more evidence	frontier moves when seed count is equalized	declare run- or configuration-weighted estimand
OOM/diverged runs deleted	no finite loss is available	completion rate falls with scale	preserve statuses and run missingness/feasibility sensitivity
robust loss called an outlier policy	extreme residual influence is bounded	a whole low-compute regime is silently ignored	plot influence and restore/exclude regimes explicitly
checkpoints bootstrapped independently	thousands of rows yield tiny intervals	repeated trajectory points appear across both samples	resample parent runs/configurations
bootstrap fits not certified	many resamples appear to support stability	parameters barely move from initialization	save displacement, gradients, and rejection counts
coefficient medians transformed once	simple summary is easy to print	result differs from median transformed allocations	solve allocation for every draw
mean-law band called prediction interval	shaded area looks probabilistic	new-seed variation is absent	add run noise or rename the band
random-row test called extrapolation	held-out score is excellent	same scale/configuration families occur in train and test	hold out complete larger groups
best fit score used as decision score	RMSE is small	recommended rounded architecture loses on held-out IsoFLOP slice	report frontier regret and feasibility
rounded coefficients used in code	table has enough decimal places for reading	large-scale predictions drift materially	publish machine-readable full precision

0x07L Fitting And Uncertainty Review Gate

Before accepting a fitted scaling law, answer:

• Are N0 and D0 frozen from the fit set, with amplitudes mapped correctly between centered and original coordinates?
• Are amplitudes, floor, and exponents positive by parameterization rather than clipping?
• Can component shares and analytic gradients explain which observations identify each parameter?
• Is the residual in linear or log loss, and what practical relative error does the Huber threshold represent?
• Is the objective summed, averaged, configuration-weighted, or run-weighted?
• Are optimizer stopping tolerances interpreted under that reduction?
• Are all initializations, failed starts, final gradients, and tighter-restart checks retained?
• Do materially different optimizers recover the same physical predictions?
• Are seeds, configurations, independent runs, and checkpoints distinct in the manifest?
• Are failed, OOM, preempted, and excluded runs retained with immutable reasons?
• Was the outlier policy frozen before residual inspection, with excluded valid runs restored in a sensitivity?
• Are residuals inspected against model, data, compute, token ratio, seed, data version, schedule horizon, status, and experiment family?
• Does the bootstrap resample the genuine independent unit within declared design strata?
• Is the entire fit and selection pipeline rerun for every bootstrap sample?
• Are rejected bootstrap fits counted rather than silently dropped?
• Are coefficients transformed into full $N^*(C)$, $D^*(C)$, ratio, loss, and feasible-candidate regret distributions?
• Are mean-law, prediction, pointwise, and simultaneous intervals named correctly?
• Are complete larger-scale configuration groups hidden until the estimator is frozen?
• Is hidden-scale decision regret reported separately from fit RMSE?
• Are full-precision parameters, fit specification, run manifest hash, and code revision published?
• Is the Besiroglu result described as a reconstructed Approach 3 audit rather than a replacement for the original run corpus or the other Chinchilla estimators?

This fitting contract turns an opaque library call into a reproducible numerical experiment. A certified fit can still extrapolate badly, because convergence answers whether the declared objective was solved, not whether the declared law survives a new regime. M4.8 makes that distinction operational by comparing interpolation and extrapolation error under frozen scale-ordered tests.

0xFC Final Formula, Source, Render, And Boundary Gate

This terminal gate adds no new scaling law. It freezes the article’s complete measurement-to-fit argument and checks that equations, source evidence, figures, and ownership boundaries still describe one compatible experiment.

Formula contract

The final registry has six owner families:

Formula family	Semantic status	Publication guard
one-variable power law and floor	rewrite derivation plus empirical fit	normalized scale, loss unit, positive domain, and floor identifiability remain explicit
joint model-data surface	exact risk decomposition followed by a separability assumption	exact telescoping terms are not presented as proof of the empirical additive law
compute-optimal allocation	analytic identity under C = kND	dimensionless coordinates, cost convention, positivity, and continuous-versus-executable decision stay visible
three frontier estimators	source-local paper procedures and rewrite implementation contracts	checkpoints, final runs, and fitted observations retain distinct dependence structures
paper-contract reconciliation	source-reported results under pinned revisions	parameter, token, compute, schedule, optimizer, and data-version coordinates are not pooled silently
numerical fit and uncertainty	rewrite numerical-analysis contract plus source-reported replication evidence	objective reduction, starts, convergence, grouping, hidden scales, and transformed uncertainty remain auditable

The release audit must find 154 non-empty display equations, balanced math delimiters and code fences, and the renderer-safe formula families identified in Sections 0x02 through 0x07. Analytic equalities, fitted forms, approximations, source-reported coefficients, and engineering recommendations must remain labeled as different evidence classes. A code fallback is an accessibility representation of the same formula, not an independent claim.

Source and figure contract

• Every primary scaling source is tied to the revision in 0x00H.
• A source-reported count, coefficient, exponent, or benchmark remains inside the model, data, optimizer, schedule, and accounting regime that produced it.
• Licensed paper figures retain source, figure number, license, and scope; article-authored reconstructions remain labeled as explanations rather than copied evidence or new measurements.
• The rights manifest and the article must agree on owner, source revision, desktop asset, mobile asset, and semantic purpose.

Render and accessibility contract

• Every responsive figure has a meaningful alt value and a complete desktop/mobile pair, and every referenced local asset exists.
• Repository-authored SVGs expose intrinsic geometry, role="img", a linked title, and a linked description.
• Desktop and mobile, in light and dark themes, contain all 154 completed equations with no MathJax error nodes, broken images, duplicate heading IDs, or page-level horizontal overflow.
• Wide formulas, tables, and code may scroll inside their own containers; they may not widen the article or hide the renderer-safe decision rules.

Cross-article owner contract

Owner	M4.7 consumes or hands forward	M4.7 does not duplicate
M4.1	exact parameter, token, FLOP, and byte coordinates for the run manifest	Transformer tensor and training-cost derivation
M4.7	measured baseline laws, compute-optimal derivation, fit certification, and grouped uncertainty	hidden-scale validation or deployment economics
M4.8	receives frozen candidates, grouped observations, fit draws, and support coordinates	alternative-law selection, repetition, and data-constrained extrapolation
M4.9	eventually consumes M4.7 estimates only after M4.8 certifies support	downstream mapping, demand, lifecycle selection, pilot gate, and training authorization, with release and rollback requirements handed to M5 and, for a low-bit deployment, M8.9/M8.10
M6.1	supplies stored-versus-active parameter conventions when a scaling family is routed	router, expert-capacity, dispatch, and expert-parallel mechanisms

The final evidence state is source-audited methodology with synthetic formula checks, not a locally reproduced scaling experiment. No local training sweep, paper refit, hidden-scale run, or hardware measurement was created by this publication audit.

0xFD Figure Provenance And Rights

The article uses two different visual objects:

licensed source figure:
    attributed source evidence retained under an explicit reuse license

article-authored reconstruction:
    article-authored explanation of a mechanism or source-reported geometry;
    no paper artwork or digitized measurements; not new empirical evidence

An arXiv rights link records the terms selected for that submission; the non-exclusive distribution license is not treated here as a general permissive reuse license. Source artwork governed only by that record is not served locally. CC-licensed figures retain the named license and attribution, and responsive adaptations are identified as adaptations.

Source figures used	Desktop evidence	Mobile treatment	Rights record
Hoffmann et al., Figures 2, 3, 4	article-authored semantic reconstructions; source artwork not served	independent responsive reconstructions	paper link retained; submission uses arXiv non-exclusive distribution
DeepSeek LLM, Figure 6	article-authored semantic reconstruction; source artwork not served	independent responsive reconstruction	paper link retained; submission uses arXiv non-exclusive distribution
Porian et al., Figure 1	attributed CC BY source figure	attributed responsive adaptation	CC BY 4.0
Besiroglu et al., Figure 5	attributed CC BY source figure	attributed responsive adaptation	CC BY 4.0

0xFF References

• M4.1 - Transformer Cost Model: Parameters, Activations, FLOPs, and Memory Traffic
• M6.1 - Routed Capacity: MoE Experts, Routers, and Token Dispatch
• Hestness et al., “Deep Learning Scaling is Predictable, Empirically”, 2017.
• Rosenfeld et al., “A Constructive Prediction of the Generalization Error Across Scales”, 2019.
• Kaplan et al., “Scaling Laws for Neural Language Models”, 2020.
• Hoffmann et al., “Training Compute-Optimal Large Language Models”, 2022.
• DeepSeek-AI, “DeepSeek LLM: Scaling Open-Source Language Models with Longtermism”, 2024.
• Porian et al., “Resolving Discrepancies in Compute-Optimal Scaling of Language Models”, 2024.
• Huber, “Robust Estimation of a Location Parameter”, 1964.
• Nocedal, “Updating Quasi-Newton Matrices with Limited Storage”, 1980.
• Alabdulmohsin et al., “Revisiting Neural Scaling Laws in Language and Vision”, 2022.
• Besiroglu et al., “Chinchilla Scaling: A Replication Attempt”, 2024.