M4.8 - Extrapolation, Alternative Scaling Laws, and Data Constraints

0x00 Extrapolation Changes The Scientific Question

M4.7 ends with a fitted estimator and a certified optimization procedure. Those are necessary outputs, but they are not yet evidence that the estimator predicts a scale it did not observe.

A fitting routine can converge to the declared optimum while the declared functional form is wrong. A curve can interpolate every measured run and still choose the wrong target-scale model. A dataset can contain billions of processed tokens while exposing far less independent information because the same examples, sources, or near-duplicates recur. These failures do not belong to numerical optimization alone. They change the experiment being claimed.

This creates M4.8’s owner question:

How should a scaling estimator be selected and changed when the target lies beyond observed scale, the curve can change shape, or unique data is capped?

The thesis is that extrapolation requires its own validation geometry. Candidate forms must be compared by forward prediction and decision error, not only by fit residual. Data scarcity must change the coordinates and constraints of the model, not merely one fitted coefficient. The article therefore treats functional form, hidden-scale evidence, and effective data as one prediction contract.

0x00A A Good Fit Is Not Yet A Forecast

Four checks are often collapsed into the phrase “the law fits”:

Check	Question	Evidence it requires	What success does not prove
numerical convergence	did the optimizer solve the stated objective?	gradients, restarts, objective agreement, solver diagnostics	that the objective or form is appropriate
interpolation	does the form reconstruct withheld observations inside the measured scale range?	grouped in-range validation	prediction beyond the observed range
extrapolation	does a law fitted below a scale boundary predict larger hidden groups?	scale-ordered backtests and a final hidden-scale test	that its allocation decision has low regret
decision validity	does the selected law choose a good feasible action?	frontier regret, allocation displacement, pilot evidence	universal validity outside the registered run family

These checks form a sequence, not interchangeable scores. Better optimization cannot rescue a misspecified curve. Random row holdout cannot certify a forward-scale claim. Small prediction error can still matter if two candidate allocations lie close to a decision boundary, while a larger coefficient error may be harmless when every plausible fit chooses the same action.

The validation split must therefore preserve the direction in which the model will be used:

lower-scale groups
    -> fit candidate laws

next larger groups
    -> select form, break count, crop, objective, and lower cutoff

still-hidden larger groups
    -> estimate extrapolation, interval coverage, and decision error once

target-adjacent pilot
    -> proceed, revise the experiment family, or abort

Every seed, checkpoint, repeated treatment, or derived run sharing an experimental parent remains on one side of a boundary. Scale order without dependence control is still leakage.

0x00B Curve Shape Is A Model Choice

A straight line in log coordinates is evidence for a power-law region, not a license to force one power law through every regime. Floors, transitions, breaks, finite ceilings, and repeated-data effects can be weakly identified inside the observed interval yet dominate a distant prediction.

M4.8 compares forms by the behavior they are allowed to express:

Candidate family	Behavior it can represent	Main identification risk	Detailed owner
power law plus floor	one diminishing-return regime approaching a nonzero asymptote	floor and exponent trade off when the floor is not observed	Sections 0x01, 0x02A, and 0x02B
sigmoidal family	a transition between lower and upper regimes	apparent saturation can move outside measured support	Sections 0x02C-0x02D
broken neural scaling law	several smooth transitions or regimes	extra breaks can fit noise and extrapolate violently	Sections 0x02E-0x02H
repeated/effective-data law	diminishing value as finite data is revisited	repetition, overlap, and optimization time can be confounded	Section 0x03

No family receives priority because it is newer or more expressive. The candidate registry, parameter bounds, fitting objective, scale cutoffs, and tie-breaking rule must be frozen before the final hidden scales are opened. Complexity is justified only when it improves forward prediction, interval behavior, or the downstream allocation it is meant to support.

0x00C Data Has Four Noninterchangeable Coordinates

“Dataset size” is too ambiguous for data-constrained scaling. At least four quantities must remain distinct:

Coordinate	Meaning	Used for	Must not be substituted for
processed tokens, $D$	token instances that contribute to training updates, including repeats	training work and compute accounting	unique corpus support
unique support, $D_{unique}$	distinct admissible token instances or examples under a declared deduplication identity	finite-data constraint	independent information after dependence
repeat count, $R=D/D_{unique}$	average traversal of the declared unique support	repetition regime	a universal measure of overfitting
effective data, $D_{eff}$	fitted amount of independent fresh-data-equivalent signal under a stated response model	predictive saturation model	physical tokens processed or stored

The definitions are operational. Changing tokenization, document identity, deduplication thresholds, sampling weights, augmentation, or contamination policy can change the coordinates even when the raw corpus bytes are unchanged. Later sections therefore audit repetition and dependence at the same level as the scaling equation.

0x00D The Output Is A Prediction Contract

M4.8 does not emit one preferred exponent. Its reviewable artifact is a registered prediction contract:

run-family identity and immutable grouped split manifest
candidate functional forms, bounds, crops, objectives, and initialization policy
fit, selection-validation, hidden-test, and target-pilot scale boundaries
absolute and normalized extrapolation distances
hidden prediction error, signed bias, interval coverage, and decision regret
processed, unique, repeated, and effective-data definitions
dependence-aware resampling units and uncertainty draws
selected form or an explicit no-selection / abort result
validated support and prohibited extrapolation region

This artifact can be consumed by M4.9. It cannot silently grow its support when a later deployment candidate is inconvenient.

0x00E Owns And Defers

M4.8 owns	Deferred owner
scale-ordered fit, validation, hidden-test, and target-pilot roles	M4.7 owns run measurement and numerical fit certification
extrapolation distance, multistep backtests, prediction metrics, and decision regret	M4.9 owns the final lifecycle decision
alternative functional forms, break-count selection, and complexity control	M4.7 owns the baseline law and compute-optimal derivation
processed, unique, repeated, and effective-data coordinates	M4.9 owns over-training as a deployment tradeoff
repeated-data saturation, double-descent diagnostics, and dependence-aware resampling	M5 owns serving state and measured runtime behavior
support, uncertainty, and abort conditions for a forecast	M8 owns quantized artifact qualification, release, and rollback

The boundary is deliberate. This article can reject a forecast because the target is unsupported or data constraints were modeled incorrectly. It does not choose a production artifact by folding training, serving, quality, and rollback into one objective; that is M4.9’s job.

0x00F Claim-Status Contract

Claims in this article use the following evidence labels:

Label	Meaning
Source-local	the linked paper or project directly states the mechanism, equation, protocol, or result
Externally verified	an independent primary source or official implementation confirms the claim
Source-reported	a numerical result is reported for the source’s particular data, model, metric, and protocol; it is not universalized
Rewrite derivation	the algebra follows from declared assumptions shown in the article
Rewrite interpretation	a workflow, diagnostic, or engineering consequence synthesized from the evidence
Uncertain	plausible but not supported strongly enough for a production claim; it remains a hypothesis or abort trigger

Plots and paper examples are evidence about their registered experiment, not constants to copy into another model family.

0x00G Three-Article Reading Map

The empirical-scaling sequence has three owners:

Article	Input	Output	It must not substitute for
M4.7 - Empirical Training Scaling Laws and Compute-Optimal Allocation	controlled run records and analytic compute definitions	measured and numerically certified baseline laws plus a training-compute allocation	hidden-scale evidence or deployment economics
M4.8 - Extrapolation, Alternative Scaling Laws, and Data Constraints	M4.7 estimators plus grouped, scale-ordered observations and data-provenance records	validated form, predictive support, data-constraint surface, uncertainty, or abort	lifecycle artifact selection
M4.9 - Over-Training, Downstream Metrics, and Lifecycle Allocation	M4.8 certified laws, frozen estimator, and prediction draws, plus measured downstream, serving, and artifact evidence	discrete train-and-serve recommendation, pilot gate, and training authorization or abort, with release and rollback requirements handed to M5 and, for a low-bit deployment, M8.9/M8.10	M5 runtime mechanisms or M8 artifact qualification

Read the core in order:

M4.7: define, measure, fit, and certify the baseline relation
    -> M4.8: test extrapolation, select shape, and model constrained data
    -> M4.9: map predictions into downstream and lifecycle decisions

M4.8 hands M4.9 a validated surface with explicit support and uncertainty, not an unqualified best-fit curve.

0x00H Pinned Source-Version Contract

This article compares functional forms and data constraints whose details have changed across paper revisions. Its source-reported equations, experiment counts, fitted constants, and limitations are tied to these audited versions:

Source	Audited version	Contract consumed here
Alabdulmohsin et al.	arXiv:2209.06640v2	scale-ordered extrapolation loss and estimator-comparison protocol
Caballero et al.	arXiv:2210.14891v17	smoothly broken functional form, transitions, and predictability limits
Hernandez et al.	arXiv:2205.10487v1	repeated-subset double descent and mechanistic repetition warning
Muennighoff et al.	arXiv:2305.16264v5	unique/repeated coordinates, diminishing-return law, 182-run fit, and data-constrained frontier

The article does not combine these experiments into one universal repetition constant. Their corpus construction, model family, repetition treatment, metric, and support remain part of each result. Cross-paper equations are compared by contract; they are not pooled as exchangeable observations.

0x01 Interpolation Is Not Extrapolation

M4.7 Section 0x07 certified one estimator after its functional form had already been declared. It answered a numerical question:

Did the fitting algorithm solve the stated objective reproducibly?

It did not answer the predictive question:

Was that functional form a useful description of scales it did not see?

These questions can disagree. Two candidate floors, exponents, or curve families can be nearly indistinguishable over the observed interval while assigning different loss to the next scale decade. The candidate with the smallest training residual can then be the worse extrapolator. If its prediction drives an allocation, the error is not merely a coefficient discrepancy: it can select the wrong model, token count, training horizon, or release decision.

This section owns the validation geometry. Section 0x02 owns the candidate forms themselves.

0x01A Four Data Roles, Not One Convenient Split

A random train/test split asks whether the estimator can reconstruct missing points surrounded by scales it has already seen. That is interpolation. A scaling decision asks whether a law fitted below a boundary survives beyond that boundary. The split must preserve this direction.

Use four roles:

inner fit:
    lower and middle independent scale/configuration groups
    fit every candidate law and every allowed initialization

selection validation:
    the next contiguous larger groups
    choose form, break count, lower cutoff, objective, and other hyperparameters

final hidden-scale test:
    still larger groups, unopened until the full selection pipeline is frozen
    estimate one-time extrapolation, coverage, and decision error

target-adjacent pilot:
    a newly trained feasible configuration near the actual target decision
    proceed, revise the experimental family, or abort

The selection validation set may be reused by the predeclared selection algorithm. The final test may not. Once an analyst inspects final-test behavior and changes a floor, break count, crop point, metric, or cutoff, that set has become another validation set. A credible final estimate then requires a new hidden scale.

Groups, not rows, cross each boundary. All seeds and checkpoints from one configuration stay together. If a training family contains paired interventions or several runs derived from the same parent checkpoint, those related records also stay on one side. Otherwise scale ordering can coexist with dependence leakage.

For one scale variable $x$, record the extrapolation distance

$$d_{ext}=\log\frac{x_{target}}{x_{max,fit}},$$

and a fit-span-normalized distance

$$\rho_{ext} = \frac{\log(x_{target}/x_{max,fit})} {\log(x_{max,fit}/x_{min,fit})}.$$

Predicting one more factor of two and predicting one more complete observed log-span are not equivalent claims, even if both results are called a “holdout.” Report the boundaries and both distances, not only the held-out row count.

0x01B Alabdulmohsin’s Counterexample: The Best Exponent Moves

Alabdulmohsin et al. make the distinction concrete with a ViT/B/16 pretrained on JFT-300M and evaluated by ImageNet 10-shot error. Within their $M_2$ power-law-plus-floor family, several exponent choices overlap across the data used to fit the curve. Minimizing interpolation error favors approximately $c=-0.24$. Evaluating on the larger hidden interval instead favors approximately $c=-0.40$.

Article-authored reconstruction of Alabdulmohsin et al., Revisiting Neural Scaling Laws in Language and Vision, Figure 3. It preserves the fit-window-versus-hidden-window comparison and the source-reported minima near -0.24 and -0.40, but its curves and points are schematic rather than copied or digitized measurements. The exponents and errors remain scoped to the paper’s ViT/JFT/ImageNet experiment.

The figure’s green points extend through 500 million seen examples; the yellow points evaluate extrapolation in the source experiment. The right panel is the critical evidence: the minimum of interpolation RMSE and the minimum of extrapolation RMSE occur at different exponents. This is not an argument that -0.40 is universal. It is evidence that an exponent estimated by visual straightness or in-window residual can be the wrong exponent for the intended prediction.

Across the paper’s experiments, the authors set $\tau=x_{max}/2$, fit on $x\in[0,\tau]$, and evaluate extrapolation on $x\in(\tau,2\tau]$. That is a valid estimator-comparison test. A production pipeline that also uses these results to choose the estimator needs the nested roles above: an inner larger-scale segment for selection and a still-hidden outer segment for the final estimate.

0x01C Prediction Metrics Must Preserve Direction And Scale

For $m$ hidden observations with measured metric $L_j>0$ and prediction $\widehat L_j>0$, log RMSE is

$$\operatorname{RMSE}_{log} = \sqrt{ \frac{1}{m}\sum_{j=1}^{m} \left(\log\widehat L_j-\log L_j\right)^2 }.$$

It approximately measures relative error when errors are small, but it discards direction. Add signed log bias:

$$b_{log} = \frac{1}{m}\sum_{j=1}^{m} \left(\log\widehat L_j-\log L_j\right).$$

For a lower-is-better loss, $b_{log}<0$ means the estimator is optimistic. That direction matters to a launch decision even when RMSE is symmetric. Also report the pointwise relative error

$$r_j^{rel} = \frac{\widehat L_j-L_j}{L_j},$$

with median, upper quantiles, and maximum absolute magnitude. One excellent near-boundary prediction should not hide a catastrophic error at the farthest scale.

If the pipeline emits intervals $[\ell_j,u_j]$, empirical coverage is

$$\widehat{\operatorname{coverage}} = \frac{1}{m}\sum_{j=1}^{m} \mathbf 1\{L_j\in[\ell_j,u_j]\}.$$

Coverage without interval width is incomplete: intervals can cover by becoming uselessly broad. Report average log width and whether the interval is a mean-law, pointwise prediction, or simultaneous band under the M4.7 0x07 definitions.

RMSLE in Caballero et al. uses log1p because some evaluated metrics can reach zero:

$$\operatorname{RMSLE}_{+1} = \sqrt{ \frac{1}{m}\sum_j \left(\log(1+L_j)-\log(1+\widehat L_j)\right)^2 }.$$

That is not interchangeable with $\operatorname{RMSE}_{log}$. Near zero, log1p behaves approximately linearly, while log gives large relative weight. A benchmark must use one common selection score across candidate families or present the objectives as separate source reproductions.

0x01D Coefficient Error Is Not Decision Error

Suppose a frozen law recommends a feasible candidate $\widehat q(C)$ from a set $\mathcal F_C$ at compute budget $C$. After the hidden IsoFLOP slice has been measured, define frontier regret

$$R_{frontier}(C) = L_{obs}(\widehat q(C)) - \min_{q\in\mathcal F_C} L_{obs}(q).$$

A relative version divides by the observed minimum. If the recommended point is not one of the measured candidates, interpolate only inside a bracketed valley under the M4.7 Section 0x05 contract; otherwise report candidate regret rather than inventing a continuous oracle.

Also report allocation displacement:

$$\Delta_N = \log\frac{\widehat N}{N_{best,hidden}}, \qquad \Delta_D = \log\frac{\widehat D}{D_{best,hidden}}.$$

These metrics answer different questions:

Diagnostic	What it detects	What it can miss
coefficient difference	parameter instability	cancellation that preserves the decision
hidden log RMSE	predictive quality over held-out runs	whether the ranking of feasible candidates is right
signed bias	systematic optimism or pessimism	local ordering among candidates
allocation displacement	movement in model/token space	a broad valley with negligible loss cost
frontier regret	actual hidden decision cost	unsupported decisions beyond the hidden frontier
interval coverage	calibration under declared resampling/noise model	unknown future regime changes

Two fits can disagree strongly in $A,B,E,\alpha,\beta$ yet recommend nearly the same broad-valley allocation. Conversely, visually similar curves can cross between measured scales and reverse candidate ordering. Decision quality is therefore a first-class selection metric, not an appendix to coefficient recovery.

The values in this diagram are synthetic. The point is the protocol: law B slightly wins interpolation but loses the larger-scale and allocation tests. The final region remains unopened until law A and the entire fitting pipeline are frozen.

0x01E Residuals Predict Which Geometry Is Missing

Aggregate error can say that a form is wrong without saying why. Plot signed residuals against ordered $\log x$, not only predicted versus observed metric. Patterns have geometric interpretations:

Residual pattern	Plausible misspecification	Required check
random scatter with stable spread	form may be adequate in support	inspect group and seed structure before accepting
one long positive run then negative run	slope or floor wrong	local-slope plot and floor profile
two sign reversals	one transition cannot represent the curve	test a transition or broken family on larger validation groups
variance grows with scale	heteroscedasticity or unstable protocol	prediction intervals and scale-specific run replication
only one model/data family drifts	hidden intervention, not universal break	stratify the manifest and refit family-specific laws
residual reversal after metric saturation	bounded metric geometry	fit a loss/error transform with declared orientation
hidden points all optimistic	regime shift or target leakage in selection	stop deployment; do not average the miss away

A residual sign change is not automatically a “phase transition.” It can come from a wrong floor, mixed architectures, a schedule change, data contamination, metric quantization, or a broken compute convention. M4.7 Section 0x01 owns those contracts. Functional flexibility is considered only after those explanations have been tested.

0x01F Bootstrap The Selection Path, Not Only The Winner

M4.7 0x07E bootstrapped grouped runs for one declared form. Model-selection uncertainty requires more. In every bootstrap draw:

1. Resample the independent run/configuration groups inside the declared design strata.
2. Recreate the scale-ordered inner-fit and selection-validation roles.
3. Fit every allowed functional form from every required initialization.
4. Select the floor policy, crop point, transition family, and break count using the frozen rule.
5. Refit the selected specification on the development region if the protocol permits that refit.
6. Propagate it into hidden predictions and feasible allocations.

The result is a distribution over selected forms and decisions, not merely parameters conditional on the most convenient form. Report selection frequencies. If a floor law wins 48% of draws and a one-break law wins 46%, a single winner hides structural uncertainty.

Do not repeatedly inspect the final hidden set inside the bootstrap to choose the form. The outer test remains an evaluation set. Bootstrap resampling can quantify variability of its score, but changing the selection rule after seeing that score consumes the test.

0x01G Scale-Ordered Selection Trace

def ordered_group_split(groups, fit_end, validation_end):
    ordered = sorted(groups, key=lambda g: g.declared_scale)
    fit = ordered[:fit_end]
    validation = ordered[fit_end:validation_end]
    final = ordered[validation_end:]
    assert disjoint_parent_lineage(fit, validation, final)
    return fit, validation, final


def select_law(candidate_specs, fit_groups, validation_groups, policy):
    reports = []
    for spec in candidate_specs:
        fitted = fit_with_multistart_certificate(spec, fit_groups)
        reports.append({
            "spec": spec,
            "fit": fitted,
            "complexity": effective_parameter_count(spec),
            "fit_metrics": prediction_metrics(fitted, fit_groups),
            "validation_metrics": prediction_metrics(
                fitted, validation_groups
            ),
            "validation_decisions": decision_metrics(
                fitted, validation_groups
            ),
            "residual_audit": residual_by_scale_and_manifest(
                fitted, fit_groups + validation_groups
            ),
        })

    # Example policy: among candidates inside a predeclared validation
    # tolerance, choose the least complex form with acceptable decision regret.
    return policy.choose(reports)


fit, validation, final = ordered_group_split(
    independent_configuration_groups,
    fit_end=FIT_BOUNDARY,
    validation_end=VALIDATION_BOUNDARY,
)
winner = select_law(CANDIDATE_SPECS, fit, validation, FROZEN_POLICY)
freeze(winner.spec, winner.fit_contract, FROZEN_POLICY)
final_report = evaluate_once(winner, final)

“Simplest acceptable” needs a numerical definition. It can mean within one bootstrap standard error of the best validation score, within a predeclared relative-error tolerance, or Pareto-efficient across extrapolation error and frontier regret. The threshold is part of the study protocol. Choosing it after seeing which law wins is another model-selection degree of freedom.

0x01H Source Evidence: One Estimator Does Not Win Every Task

Alabdulmohsin et al. compare four estimators over image classification, neural machine translation, language modeling, and BIG-Bench tasks. Their Figure 1 uses black/gray points for fitting and amber points for larger unseen values; the lower panel summarizes source-reported average extrapolation ranking.

*Article-authored reconstruction of Alabdulmohsin et al., Revisiting Neural Scaling Laws in Language and Vision, Figure

1. It preserves the estimator-comparison argument and source-reported relative ordering, but the curves, points, and bar lengths are schematic rather than paper artwork or digitized values. Rankings belong to the paper’s declared domains and are not a universal model-selection order.*

The reusable result is methodological, not a universal leaderboard. The source’s own $M_4$ family, defined in Section 0x02A, performs strongly in the paper’s benchmark, but BIG-Bench shows a tie in the plotted average ranking and individual tasks vary. Candidate laws must be compared on the target metric, scale variable, family, and extrapolation distance rather than selected from a cross-domain winner count.

0x01I Extrapolation Failure Lab

Failure	Attractive but wrong conclusion	Detection	Repair
random row holdout	”test RMSE is low”	adjacent scales and sibling seeds occur on both sides	hold out complete larger groups
one outer set used for selection and reporting	”we have an unbiased final score”	analyst changed form after seeing outer residuals	acquire or reserve a new outer scale
smallest fit RMSE wins	”more precise fit means better law”	larger validation groups reverse the ranking	select by extrapolation and decision metrics
farthest point dominates score	”average extrapolation is poor everywhere”	error plotted by extrapolation distance	report pointwise errors and distance
RMSE without signed bias	”errors cancel safely”	all predictions are optimistic	publish signed bias and launch gate
coefficient mismatch called failure	”replication found different constants”	feasible allocation and regret agree	compare physical predictions and decisions
broad intervals called calibrated	”target is covered”	width makes decision non-actionable	report width and decision stability
one hidden break called discovered	”phase transition at $d_i$“	break moves across resamples or contracts	call it estimator uncertainty and gather runs
tuning on the target pilot	”pilot validated the frozen law”	pilot changed the estimator	label it development and run a new pilot

0x02 Alternative Functional Forms

The choice among a pure power law, a floor law, a transition law, and a broken law is not cosmetic. Each family permits a different set of slopes, asymptotes, inflections, and reversals. A flexible family can interpolate a simple regime; a simple family cannot express geometry outside its definition. Neither fact decides which one extrapolates.

This section compares one-dimensional learning-curve families $x\mapsto\epsilon_x$. It does not silently replace the joint $N,D\mapsto L$ surface derived in M4.7 Section 0x03. A one-dimensional winner cannot be used as a compute-optimal allocator until a corresponding multivariable law and constraint have been derived and validated.

0x02A The Source Families M1 Through M4

Alabdulmohsin et al. organize prior estimators as:

$$\begin{aligned} M_1:\quad &\epsilon_x=\beta x^c,\\ M_2:\quad &\epsilon_x-\epsilon_\infty=\beta x^c,\\ M_3:\quad &\epsilon_x=\beta(x^{-1}+\gamma)^{-c},\\ M_4:\quad & \frac{\epsilon_x-\epsilon_\infty} {(\epsilon_0-\epsilon_x)^\alpha} =\beta x^c, \end{aligned}$$

Renderer-safe source-family contract:

M1: epsilon_x = beta * x^c
M2: epsilon_x - epsilon_inf = beta * x^c
M3: epsilon_x = beta * (x^(-1) + gamma)^(-c)
M4: (epsilon_x - epsilon_inf) / (epsilon_0 - epsilon_x)^alpha
        = beta * x^c

source orientation: x > 0, beta > 0, c < 0, gamma >= 0, alpha >= 0
later in this article: g := the source M4 transition parameter alpha

with $x>0$, $\beta>0$, $c<0$, $\gamma\ge0$, and $\alpha\ge0$ under their decreasing-error orientation. To avoid confusing the source’s $M_4$ transition parameter $\alpha$ with the positive exponent used earlier in this article, later derivations call it $g$ while retaining the source equation here.

Family	Structural promise	Useful regime	Structural limitation
$M_1$	one straight line in log-log coordinates	regular power-law region with negligible floor	cannot saturate above zero
$M_2$	one power law in excess over a learned floor	asymptotic regular region	floor and exponent can trade before the floor is visible
$M_3$	shifted inverse-scale coordinate that saturates when $\gamma>0$	some gradual one-transition curves	monotone and more restrictive than a general multi-break law
$M_4$	bounded monotone transition into an asymptotic power law	observed data include a guess/transition region	cannot represent non-monotonic behavior
BNSL	$n+1$ approximate log-log slopes joined smoothly	repeated slope changes, delayed inflection, or reversal	$3+3n$ parameters and unsupported-break risk

The metric orientation must be frozen first. An error or loss usually decreases toward a lower floor. Accuracy, reward, or Elo may increase. Transforming accuracy to error is not harmless near a ceiling unless the scientific question is explicitly about error. The same observations can have different residual geometry under accuracy, error, log error, and logit accuracy.

0x02B Source Families M1 And M2: What The Floor Changes

Write a decreasing pure law with a positive exponent:

$$L(x)=Ax^{-p}, \qquad A>0,\ p>0.$$

Taking logs gives

$$\log L=\log A-p\log x,$$

so its log-log slope is exactly $-p$ at every scale. This is not an empirical consequence of fitting; it is hard-coded geometry.

Add a floor:

$$L(x)=E+Ax^{-p}.$$

The excess loss remains a straight power law,

$$\log(L-E)=\log A-p\log x,$$

but total loss does not. Its local log slope is

$$\frac{d\log L}{d\log x} = -p\frac{Ax^{-p}}{E+Ax^{-p}}.$$

The fraction is the reducible-loss share. When $Ax^{-p}\gg E$, the slope is approximately $-p$; when the floor dominates, it approaches zero. Thus $M_2$ can curve in a log-total-loss plot even though it has only one excess-loss exponent. M4.7 Section 0x02 showed why an unsupported $E$ lets the inferred exponent move.

$M_3$ also saturates. As $x\to\infty$, $x^{-1}\to0$, so for $\gamma>0$,

$$\epsilon_x\to\beta\gamma^{-c}.$$

Its shift occurs in inverse-scale coordinates, whereas $M_2$ adds a floor in metric coordinates. Similar-looking plateaus therefore encode different assumptions. They must be compared by held-out prediction, not described as equivalent ways to write the same law.

0x02C Deriving Source Family M4 From Bounded Performance

Alabdulmohsin et al. start with three requirements:

1. The estimator should fail gracefully when observations include the pre-power-law transition.
2. It should approach a power law above a limiting error $\epsilon_\infty$ at large $x$.
3. It should contain $M_2$ as a special case.

Let $\epsilon_0$ be the random-guess error and let $h(x)>0$ move from large to small as scale grows. A convex interpolation between the two endpoints is

$$\epsilon_x = \frac{h(x)}{1+h(x)}\epsilon_0 + \frac{1}{1+h(x)}\epsilon_\infty.$$

When $h\to\infty$, the first weight approaches one and $\epsilon_x\to\epsilon_0$. When $h\to0$, the second weight approaches one and $\epsilon_x\to\epsilon_\infty$. Choose $h(x)=\beta x^c$ with $c<0$. Rearranging gives

$$\frac{\epsilon_x-\epsilon_\infty} {\epsilon_0-\epsilon_x} = \beta x^c.$$

This particular convex mixture fixes the denominator exponent to one. The source generalizes it with a learned $g\ge0$:

$$\boxed{ \frac{\epsilon_x-\epsilon_\infty} {(\epsilon_0-\epsilon_x)^g} = \beta x^c }$$

When $g=0$, the denominator is one and the expression reduces exactly to $M_2$. That nesting matters: $M_4$ can recover the simpler family when the observed data already live in the asymptotic regime.

The monotonicity is derivable. Let $z=\log x$ and take logs:

$$\log(\epsilon-\epsilon_\infty) -g\log(\epsilon_0-\epsilon) = \log\beta+cz.$$

Differentiate implicitly:

$$\left[ \frac{1}{\epsilon-\epsilon_\infty} + \frac{g}{\epsilon_0-\epsilon} \right] \frac{d\epsilon}{dz} =c.$$

Inside $\epsilon_\infty<\epsilon<\epsilon_0$, the bracket is positive. Since $c<0$,

$$\frac{d\epsilon}{d\log x}<0.$$

$M_4$ may bend and possess a linear-coordinate inflection, but it cannot reverse direction. Its local excess-loss slope is

$$\frac{d\log(\epsilon-\epsilon_\infty)}{d\log x} = \frac{c} {1+g\dfrac{\epsilon-\epsilon_\infty} {\epsilon_0-\epsilon}}.$$

Near the upper endpoint, the denominator can be large and improvement is shallower. Near the lower endpoint, the ratio tends to zero and the slope tends to $c$. This is how the law transitions into its final power-law regime rather than merely fitting an arbitrary sigmoid.

The source gives the asymptotic expansion

$$\epsilon_x-\epsilon_\infty \sim (\epsilon_0-\epsilon_\infty)^g\beta x^c -g(\epsilon_0-\epsilon_\infty)^{2g-1}(\beta x^c)^2.$$

The first term is a power law with exponent $c$; the next correction decays as $x^{2c}$. The equation therefore satisfies the required large-scale limit rather than only resembling one over a plot window.

0x02D Source Family M4 Fitting Is An Implicit Root Problem

For general $g$, the equation does not provide a convenient explicit $\epsilon(x)$ in elementary operations. Prediction can use a bracketed root because the log-equation is strictly increasing in $\epsilon$ inside the two endpoints:

def m4_error(x, p, tiny=1e-10):
    # p: epsilon_inf < epsilon_0, g >= 0, beta > 0, c < 0
    log_rhs = log(p.beta) + p.c * log(x)

    # At g=0, M4 is exactly M2 and epsilon_0 leaves the equation.
    if p.g <= tiny:
        return p.epsilon_inf + exp(log_rhs)

    def residual(epsilon):
        return (
            log(epsilon - p.epsilon_inf)
            - p.g * log(p.epsilon_0 - epsilon)
            - log_rhs
        )

    return brentq(
        residual,
        p.epsilon_inf + tiny,
        p.epsilon_0 - tiny,
    )

The derivative used above,

$$\frac{\partial r}{\partial\epsilon} = \frac{1}{\epsilon-\epsilon_\infty} + \frac{g}{\epsilon_0-\epsilon}>0,$$

establishes uniqueness of the root in the bracket. A generic unconstrained Newton solve can leave the valid interval and take a logarithm of a negative number; bracketed solving makes the domain part of the implementation. The explicit g=0 branch is necessary because that nested $M_2$ case no longer uses $\epsilon_0$ as an upper endpoint in its equation.

The paper estimates $M_4$ by minimizing the square-log relation

$$\mathcal J_{M4} = \mathbb E_x\left[\left( \log(\epsilon_x-\epsilon_\infty) -g\log(\epsilon_0-\epsilon_x) -\log\beta-c\log x \right)^2\right].$$

For fixed nonlinear endpoint parameters, $g$, $\log\beta$, and $c$ enter as a linear least-squares problem. The source alternates closed-form least squares for that block with gradient updates for remaining parameters until convergence. This source procedure should be reproduced as such. A modern comparison can also use positive/bounded transforms, multiple starts, tighter certificates, and a common held-out score, but those changes must be labeled as an audit implementation rather than silently attributed to the paper.

Fixing $\epsilon_0$ to known random-guess performance removes one weakly identified degree of freedom. Optimizing it may be reasonable when no natural guess level exists, but then upper endpoint, transition parameter, and cutoff can trade. Profile that choice and propagate it through model selection.

0x02E BNSL: A Product Of Smooth Slope Changes

Caballero et al. define a broken neural scaling law with $n$ breaks:

$$\boxed{ y(x) = a + b x^{-c_0} \prod_{i=1}^{n} \left[ 1+ \left(\frac{x}{d_i}\right)^{1/f_i} \right]^{-c_i f_i} }$$

Renderer-safe BNSL contract:

y(x) = a + b * x^(-c_0)
             * product_i(
                   [1 + (x / d_i)^(1 / f_i)]^(-c_i * f_i)
               )

where:

a      limiting offset or asymptote
b      positive vertical scale of y-a
c_0    initial log-log slope magnitude
c_i    change in slope across break i
d_i    scale location of break i
f_i    transition width/sharpness; smaller positive f_i is sharper
n      number of breaks selected outside the continuous fit

For $n=0$, the empty product is one and the family reduces to $y=a+bx^{-c_0}$, the floor law. Each break adds three continuous parameters, so the unconstrained count is $3+3n$ before any fixed asymptote or shared parameter policy.

Source: Caballero et al., Broken Neural Scaling Laws, Figure 1, licensed by the authors under CC BY 4.0. The mobile figure is an attributed responsive reconstruction of the same geometric argument.

The source figure is deliberately non-monotonic: its second break changes a decreasing segment into an increasing one, and the third changes direction again. The figure proves representational capacity, not that every neural learning curve contains three breaks.

0x02F Why The Product Produces Piecewise Log-Log Slopes

Let $z=\log x$ and subtract the offset. For $y>a$,

$$q(z)=\log(y-a).$$

Because

$$\left(\frac{x}{d_i}\right)^{1/f_i} = \exp\left(\frac{z-\log d_i}{f_i}\right),$$

the BNSL becomes

$$q(z) = \log b-c_0z - \sum_{i=1}^{n} c_if_i\operatorname{softplus} \left(\frac{z-\log d_i}{f_i}\right).$$

Differentiate with respect to log scale:

$$\frac{dq}{dz} = -c_0 - \sum_{i=1}^{n} c_i\operatorname{sigmoid} \left(\frac{z-\log d_i}{f_i}\right).$$

Far before break $i$, its sigmoid is near zero and it contributes almost nothing to slope. Far after the break, it is near one and changes slope by $-c_i$. After the first $k$ well-separated breaks, the approximate slope is

$$-\left(c_0+\sum_{i=1}^{k}c_i\right).$$

The second derivative localizes the transition:

$$\frac{d^2q}{dz^2} = -\sum_{i=1}^{n} \frac{c_i}{f_i} \sigma(t_i)[1-\sigma(t_i)], \qquad t_i=\frac{z-\log d_i}{f_i}.$$

Its magnitude peaks near $z=\log d_i$. Smaller $f_i$ narrows and sharpens that curvature. In the limit $f_i\to0^+$,

$$f_i\operatorname{softplus} \left(\frac{z-\log d_i}{f_i}\right) \to \max(0,z-\log d_i),$$

so the smooth transition becomes a hinge in log-log coordinates.

If a cumulative slope crosses zero, direction reverses. This is why BNSL can represent double-descent-like or delayed-transition curves that the monotone $M_1$-$M_4$ families cannot. The same signed $c_i$ freedom also lets noise generate a spurious reversal when data are sparse.

0x02G Stable BNSL Parameterization And Code Trace

Evaluate the law in log space. Order break locations by construction so two nominal breaks cannot exchange labels between optimizer starts:

def decode_breaks(raw_first, raw_gaps):
    log_d = [raw_first]
    for gap in raw_gaps:
        log_d.append(log_d[-1] + softplus(gap))
    return array(log_d)


def bnsl_prediction(log_x, raw):
    a = softplus(raw.a)
    log_b = raw.log_b
    c0 = raw.c0                 # sign follows the metric contract
    c = raw.delta_slopes        # signed: direction may change
    log_d = decode_breaks(raw.first_break, raw.break_gaps)
    f = softplus(raw.raw_width) + 1e-6

    t = (log_x[..., None] - log_d) / f
    log_excess = (
        log_b
        - c0 * log_x
        - sum(c * f * softplus(t), axis=-1)
    )
    return a + exp(log_excess)

If the metric can have a negative asymptote, a=softplus(raw.a) is wrong; parameterize it under the metric’s valid domain instead. If a known chance level or zero floor is available, fixing a can remove a degree of freedom. The transform is part of the measurement contract, not reusable boilerplate.

Very small $f_i$ creates a nearly discontinuous derivative and a difficult objective. Very close $d_i$ values let adjacent slope changes cancel. Record minimum break separation, width bounds, optimizer starts, and whether estimates sit on those bounds. A numerically converged zero-width break is not necessarily identified.

0x02H How The BNSL Paper Fits And Chooses Break Count

Caballero et al. report this source pipeline:

1. scipy.optimize.brute grid-searches constants for an initialization.
2. scipy.optimize.curve_fit performs nonlinear least squares.
3. The optimized score is a numerically stable MSLE using $\log(1+y)$.
4. To choose $n$, the last few largest black fit points are held out as an inner validation set.
5. The $n$ with lowest inner validation error is selected.
6. The green larger points remain separate extrapolation-evaluation points.

That distinction prevents the source’s reported green extrapolation set from also choosing model complexity. The paper notes that the same strategy can choose a crop point when earlier, already-completed sharp breaks are unhelpful for predicting a later smooth regime.

A strict audit should preserve the source result and add the M4.7 fitting checks:

source reproduction:
    source grid, curve_fit, log1p-MSLE, source break-count procedure

certified sensitivity:
    ordered-break parameterization, independent optimizer,
    multi-start certificate, grouped bootstrap, common selection score

decision audit:
    hidden-scale bias, interval coverage, feasible-candidate regret,
    and sensitivity to n, crop point, asymptote, and width bounds

If a richer audit changes the selected breaks, report both. Do not rewrite the source method retrospectively.

0x02I Fair Comparison Requires A Common Target

Functional forms are often compared after each paper’s preferred objective is used. That can be useful for source reproduction but confounds curve family with loss function. Use two ledgers:

Ledger	Fitting contract	What it answers
source-faithful	original transform, optimizer, cutoff, parameter constraints	can the published result be reproduced?
common-comparison	same fit groups, metric orientation, residual score, and selection protocol where mathematically valid	which geometry predicts this target best?

For example, $M_4$‘s source objective fits a transformed implicit relation, while BNSL’s source pipeline minimizes log1p MSLE. A lower BNSL source objective and a lower $M_4$ source objective do not share units. Compare both on the same held-out log RMSE, relative error, and decision score.

Complexity also needs a declared policy. A practical rule is:

1. Begin with the least expressive family compatible with the expected regime.
2. Reject it if residuals show geometry it cannot represent.
3. Fit the next family under the same scale-ordered development split.
4. Prefer the simpler family when validation performance is within a
   predeclared uncertainty/tolerance region.
5. Open the final hidden scale only after form and complexity are frozen.
6. Bootstrap the entire path, including the selected number of breaks.

Information criteria computed on fit rows can supplement this protocol but do not replace extrapolation. AIC or BIC penalizes in-sample likelihood complexity under assumptions about independent observations and a correctly specified likelihood. It does not test an unseen scale regime or feasible allocation.

0x02J Alternatives Are Design Choices, Not A Ladder Of Progress

The families answer different scientific questions:

Observed situation	First candidate	Alternative	Why not jump directly to BNSL?
stable straight excess-loss region	$M_1$ or $M_2$	profile floor and cutoff	breaks can fit ordinary run noise
plausible nonzero floor but weak saturation	$M_2$ with floor profile	$M_4$ with fixed $\epsilon_0$	transition and floor may both be unidentified
chance-level region enters observed support	$M_4$	crop to regular region and fit $M_2$	keeping versus cropping answers different targets
one reproducible change in local slope	$M_4$ if monotone transition is plausible	one-break BNSL	BNSL allows sharper and signed slope change but costs parameters
multiple validated slope changes	BNSL with selected $n$	segmented or spline sensitivity	unconstrained splines can extrapolate pathologically
non-monotonic metric	BNSL or mechanistic piecewise model	model the causal intervention explicitly	$M_1$-$M_4$ cannot reverse direction
architecture/data policy changes at the bend	stratified laws or intervention model	hierarchical multi-family surface	a smooth break hides a known discrete cause

Cropping is not always data deletion. If the intended claim concerns the final regular regime, excluding a known guess or obsolete pre-break regime can reduce bias. But the crop threshold is then a selected hyperparameter. It belongs inside the scale-ordered validation and bootstrap path.

Splines and Gaussian processes are useful interpolation and residual tools, but unconstrained versions have no automatic power-law asymptote. A spline can trace every fit point and become arbitrary beyond its boundary knots. Use them as flexible baselines, impose scientifically meaningful tail behavior, and score the actual extrapolation.

0x02K A Unified Candidate Pipeline

def candidate_specs():
    yield Spec("pure", parameters=2, breaks=0)
    yield Spec("floor", parameters=3, breaks=0)
    yield Spec("m3_shift", parameters=3, breaks=0)
    yield Spec("m4_transition", parameters=4, epsilon0="fixed")
    for n in range(0, MAX_BREAKS + 1):
        yield Spec("bnsl", parameters=3 + 3 * n, breaks=n)


def compare_functional_forms(groups, split, policy):
    fit_groups, selection_groups, final_groups = split
    reports = []

    for spec in candidate_specs():
        fit = certified_multistart_fit(
            spec,
            fit_groups,
            source_objective=SOURCE_REPRODUCTION,
            common_objective=COMMON_COMPARISON,
        )
        reports.append(CandidateReport(
            spec=spec,
            fit=fit,
            validation=score_prediction_and_decision(
                fit, selection_groups
            ),
            residuals=residual_audit(
                fit, fit_groups + selection_groups
            ),
            bound_hits=fit.bound_hits,
        ))

    winner = policy.choose_simplest_acceptable(reports)
    frozen = refit_if_predeclared(winner, fit_groups + selection_groups)
    assert no_selection_change_after_this_line()
    final = score_prediction_and_decision(frozen, final_groups)
    return reports, frozen, final

If development groups are refit together after selection, the final test is essential: the selected specification has not previously been scored after seeing all development rows. The code should enforce that final results cannot flow backward into policy.choose_simplest_acceptable.

Store a machine-readable record:

candidate family and exact equation
metric orientation and transform
fit/selection/final group IDs and scale boundaries
parameter domain and fixed values
optimizer starts, convergence certificates, and rejected fits
source-faithful and common-objective scores
selected break count, break ordering, widths, and bound hits
pointwise residuals and extrapolation distances
selection policy and tolerance
hidden prediction, coverage, allocation, and regret
bootstrap selection frequency and decision distribution

0x02L The Limit Of Predicting An Unseen Break

The most important BNSL result is also its strongest limitation. Caballero et al. study a sharp break in 4-digit addition and a noiseless simulation of the fitted law. Accurate post-break extrapolation requires fit points near the break. Even removing seed noise does not let a pre-break curve reveal an arbitrary sharp transition far beyond observed support.

In the paper’s formulation:

1. A sufficiently sharp break imposes a lower bound on how far before the break the largest fit point can be for accurate extrapolation.
2. If another sharp break lies sufficiently beyond the largest fit scale, current observations do not identify the behavior after it.
3. Once a break is well inside observed support, much earlier points can become unhelpful for a later-regime extrapolation, though the paper documents exceptions.

This prevents a dangerous interpretation. Fitting a BNSL with two observed breaks does not license the model to invent the location and direction of a third unseen break. The flexible law describes supported transitions; it does not forecast arbitrary future emergence from no evidence.

Treat a break beyond support as scenario uncertainty:

validated baseline:
    extrapolate the selected supported law only to the tested distance

stress scenarios:
    posit an additional bend at declared distances and slope changes
    quantify decision sensitivity without assigning data-derived probability

evidence acquisition:
    place new runs where candidate scenarios disagree most

release gate:
    do not proceed if plausible unsupported breaks reverse the decision and
    no target-adjacent pilot can resolve them

Calling a fitted $d_i$ a phase transition also overstates the model. It is a smooth-curve parameter whose value depends on metric, transform, data family, crop policy, and break count. A mechanistic phase claim requires independent evidence.

0x02M One-Dimensional Forms Do Not Allocate N And D

Every law in this section maps one chosen $x$ to one metric. The choice of $x$ matters:

x = parameter count:
    valid only under a declared data/training policy

x = processed tokens:
    valid only under a declared model family and size policy

x = training compute:
    collapses model/data choices into the policy used to generate the frontier

x = upstream loss:
    predicts downstream transfer, not training-resource allocation directly

A one-dimensional BNSL over Pareto-optimal compute can predict a frontier metric, but it cannot recover which $(N,D)$ pair produced the frontier unless that allocation policy is separately modeled. Replacing each term in $E+AN^{-\alpha}+BD^{-\beta}$ with arbitrary broken factors also changes the constraint derivatives and may introduce several local optima. That is a new joint model requiring its own experiment grid and derivation, not a harmless upgrade.

Use one-dimensional alternative forms to audit the shape of fixed-policy learning curves and frontier metrics. Keep M4.7 Sections 0x03-0x05 as the owner of joint model-data allocation until multivariate evidence supports a richer surface.

0x02N Functional-Form Failure Lab

Failure	Symptom	Why it happens	Response
pure law spans saturation	slope weakens systematically	zero-floor geometry is wrong	profile $M_2$ floor or restrict regime
$M_2$ floor floats below evidence	excellent fit, divergent exponent	floor and slope are weakly identified	scale holdout and floor profile
$M_4$ fits non-monotonic data	residual sign changes survive	$M_4$ is monotone by derivation	BNSL or explicit intervention model
BNSL adds one break per noise cluster	fit error collapses, validation worsens	$3+3n$ capacity interpolates noise	inner scale validation and simplest-acceptable rule
two breaks exchange labels	unstable $d_i,c_i$ across starts	unordered parameter symmetry	ordered log-break parameterization
width tends to zero	sharp transition appears certain	boundary solution or isolated point	width sensitivity and more runs near break
break lies beyond fit support	dramatic future change is forecast	unconstrained extrapolation or prior assumption	mark scenario, not discovered break
source objectives compared directly	one method “wins” incomparable score	log, log1p, and implicit residuals differ	common held-out scoring ledger
monotone fit hides reversal	average RMSE appears acceptable	opposite signed regimes cancel	local slopes and signed residuals
BNSL winner used as N/D law	compute allocation lacks derivation	one-dimensional form lost policy variables	retain or derive a joint surface
final hidden set tunes $n$	published extrapolation is optimistic	test set was consumed by selection	acquire a new final scale
fitted break called causal	paper narrative outruns curve evidence	$d_i$ is descriptive	audit interventions and mechanisms

0x02O Functional-Form Selection Review Gate

Before accepting an alternative scaling-law form, answer:

• Are fit, selection-validation, final-test, and target-pilot roles distinct?
• Are complete dependent groups kept on one side of every boundary?
• Is extrapolation distance reported relative to both the fit maximum and the observed log-span?
• Are log error, signed bias, pointwise relative error, interval coverage, and frontier regret reported separately?
• Is the metric orientation and transform identical across the common candidate comparison?
• Are source-faithful objectives distinguished from the common held-out score?
• Can residual geometry explain why a more expressive family is needed?
• Is the simplest acceptable rule numerical and frozen before validation?
• Is form, floor, cutoff, break count, and crop selection repeated inside the grouped bootstrap?
• Are $M_4$‘s endpoints, monotonicity, implicit root, and asymptotic slope treated consistently?
• Are BNSL break locations ordered and widths positive by parameterization?
• Are bound hits, near-zero widths, and unstable break identities reported?
• Is non-monotonic behavior preserved rather than averaged under a monotone fit?
• Is an unseen sharp break represented as uncertainty rather than a predicted phase transition?
• Is a one-dimensional curve prevented from silently becoming a joint N/D allocation law?
• Is the final hidden set evaluated once, after the complete pipeline is frozen?

The functional-form analysis establishes a model-selection discipline: added parameters are useful only when larger-scale evidence needs the geometry they provide. It also marks the boundary of that discipline. No univariate functional form can recover information about a sharp regime change that leaves no trace in observed support. Section 0x03 addresses a different source of curvature: finite unique data, repeated tokens, and the distinction between processed and effective data.

0x03 Data Constraints, Repetition, And Effective Sample Size

The abundant-data law in M4.7 Section 0x03 assigns the data term to processed tokens:

$$L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}.$$

That coordinate silently bundles two claims:

1. another processed token is drawn from the same target distribution under the same data policy;
2. its expected return is described by the same data power law as prior tokens.

Neither claim survives automatically when the finite corpus is traversed again. Compute still pays for every occurrence. Corpus coverage does not grow when the same occurrence returns. Statistical concentration depends on which items return. Validation performance may improve, plateau, or even worsen.

The owner question for data-constrained scaling is therefore:

When unique data is capped, how should the scaling record distinguish work, corpus support, exposure concentration, and performance-equivalent data?

This section owns:

processed tokens versus unique token occurrences
full-corpus epochs versus concentrated repeated subsets
Muennighoff-style diminishing-return effective data
the symmetric excess-parameter construction used by that source
data-constrained compute allocation
exposure concentration and estimator-specific effective sample size
dependence-aware splitting, fitting, bootstrap, and runtime instrumentation

It does not turn one source’s repetition constant into a universal law. M4.9 Section 0x01 owns deliberate over-training, downstream metrics, and apparent emergence. M5 owns serving demand, and M8 owns compression. Data mixture optimization and data-quality intervention remain separate experiments unless their variables are explicitly included in the fit.

0x03A One Letter D Is No Longer Enough

Start with a canonicalized training corpus under one immutable tokenizer. Let $i=1,\ldots,M$ index canonical units such as documents or deduplicated spans, $\ell_i$ be their token lengths, $u_i\le\ell_i$ be the number of distinct canonical token positions actually covered, and $w_i\ge u_i$ be the total number of emitted token occurrences attributed to that unit. Then the first inventory ledgers are

$$D_{unique} = \sum_{i=1}^{M}u_i,$$

and

$$D_{total} = \sum_{i=1}^{M}w_i.$$

D_unique is a count of covered token occurrences in canonical source units. It is not the number of distinct vocabulary IDs. D_total is optimizer work: every token occurrence actually processed, including repeats. Both depend on the tokenizer, canonicalization level, inclusion policy, and actual loader trace. If every emitted unit is complete, $u_i=\ell_i$ for covered units and $w_i=m_i\ell_i$. With truncation or span sampling, unique canonical token ranges must be unioned instead of crediting an entire document after one fragment.

For a perfectly uniform traversal of one fixed corpus, define

$$R_D = \frac{D_{total}}{D_{unique}}-1, \qquad E_D=1+R_D,$$

where $E_D$ is the epoch-equivalent exposure. The first pass has $R_D=0$ and $E_D=1$. Four complete passes have $R_D=3$.

This ratio is not automatically an epoch counter. In weighted mixture sampling, curriculum learning, partial epochs, streaming replacement, or localized duplication, it is only a mean exposure ratio. Two runs can have the same D_total, D_unique, and $R_D$ while allocating repeat mass to very different units.

Ledger	Operational question	How it is obtained	What it cannot answer alone
D_total	how much token work did optimization process?	loader/runtime counters	how much new information was available
D_unique	how much canonical token support was covered?	hashes, lineage, tokenizer, inclusion manifest	how evenly support was sampled
exposure distribution $p_i$	where did processed token mass go?	per-unit or per-shard counts	equivalent validation performance
$D_{eff}$	what fresh-data amount gives comparable fitted loss?	controlled performance model	statistical independence or corpus diversity
run-level $n_{eff}$	how much independent information supports an estimator?	covariance/group model	effective training data for the network

Shuffling can reduce short-range order effects in SGD. It cannot turn duplicate content into new content. Deduplication can increase the number of canonical units represented per processed token, but it can also remove useful recurring templates. These are measured interventions, not identities implied by the token count.

0x03B Repetition Has A Topology

At least four repetition designs occur in language-model training:

Repetition design	Exposure pattern	Meaning of an epoch	Main risk
complete corpus passes	most units receive nearly equal multiplicity	meaningful after declaring traversal and remainder	diminishing marginal return and eventual overfitting
repeated subset inside mostly unique data	a small set receives very high multiplicity	separate epoch count for the repeated subset	concentrated memorization and non-monotonic damage
mixture upweighting	one source/domain receives higher sampling probability	source-specific exposure, not one global epoch	target distribution changes with the weighting
accidental duplicates	repeated spans/documents survive collection or packing	generally unknown until canonicalized	hidden concentration, contamination, train/eval leakage

Hernandez et al. isolate the second design. They hold 10% of training-token mass in a repeated subset while varying the subset’s size and repeat count; the remaining 90% is non-repeated data. A tiny subset repeated many times and a larger subset repeated fewer times can therefore consume the same compute share while presenting different memorization pressure.

Source: Hernandez et al., Scaling Laws and Interpretability of Learning from Repeated Data, Figure 1, licensed under CC BY 4.0. The desktop image retains the attributed source figure; the mobile image is an attributed responsive adaptation of its data split and training composition. The design repeats a subset, not the complete corpus.

Muennighoff et al. target a different contract: a bounded corpus is traversed for one or more epochs, and runs using fewer unique tokens repeat that complete available subset. Their data-constrained law is appropriate evidence for that controlled topology. It is not evidence that an arbitrary duplicate distribution can be reduced to one global epoch count.

The run manifest should therefore retain more than epochs:

data_exposure:
  canonical_unit: "deduplicated document"
  unique_tokens_seen: 100000000000
  processed_tokens: 400000000000
  epoch_equivalent: 4.0
  traversal_policy: "full shuffled passes with final partial pass"
  sampler: "without replacement within pass"
  source_mixture_hash: "..."
  canonicalization_hash: "..."
  multiplicity_histogram_uri: "artifacts/exposure-counts.parquet"
  max_unit_token_share: 0.000004
  inverse_simpson_support: 23800000

If canonical IDs are unavailable, D_unique and exposure concentration are not observed. Estimating them from a document count, byte count, or nominal dataset name must be marked as an estimate with its collision and granularity policy.

0x03C Deriving A Diminishing-Return Effective-Data Law

Muennighoff et al. begin with $U_D$ unique token occurrences and assume that the value of the same corpus decays geometrically on later passes. If each repeat retains a fraction $q=1-\delta$ of the prior pass’s value, then after $R_D$ complete repeats the exact discrete construction is

$$D'_{geo} = U_D\sum_{k=0}^{R_D}q^k.$$

For $q\ne1$, the finite geometric sum gives

$$D'_{geo} = U_D\frac{1-q^{R_D+1}}{1-q} = U_D + U_Dq\frac{1-q^{R_D}}{1-q}.$$

The first $U_D$ is the unique pass. The remaining term is the accumulated value of repeated passes. As $R_D\rightarrow\infty$,

$$D'_{geo} \rightarrow \frac{U_D}{1-q} = \frac{U_D}{\delta}.$$

The source then defines

$$R_D^* = \frac{1-\delta}{\delta} = \frac{q}{1-q},$$

so the repeated contribution saturates at $U_DR_D^*$. For small $\delta$, $q=1-\delta\approx e^{-\delta}\approx e^{-1/R_D^*}$. Substituting this continuous approximation yields the source’s interpretable form:

$$\boxed{ D_{eff} = U_D + U_DR_D^* \left(1-e^{-R_D/R_D^*}\right) }$$

or equivalently

$$D_{eff} = U_D + \left(D_{total}-U_D\right) \underbrace{ \frac{R_D^*}{R_D} \left(1-e^{-R_D/R_D^*}\right) }_{\text{average value of one repeated token}}.$$

Renderer-safe effective-data contract:

D_total = U_D * (1 + R_D)
D_eff = U_D + U_D * R_D_star * (1 - exp(-R_D / R_D_star))

That second form exposes the conversion being claimed. It is not saying a repeated occurrence disappears. It says the performance return of the repeated portion is equivalent, under the fitted contract, to a smaller fresh-data increment.

For $R_D\ll R_D^*$, expand the exponential:

$$1-e^{-R_D/R_D^*} = \frac{R_D}{R_D^*} - \frac{R_D^2}{2R_D^{*2}} + O(R_D^3).$$

Therefore

$$D_{eff} = U_D(1+R_D) - U_D\frac{R_D^2}{2R_D^*} + O\!\left(\frac{U_DR_D^3}{R_D^{*2}}\right) = D_{total} - U_D\frac{R_D^2}{2R_D^*} + O\!\left(\frac{U_DR_D^3}{R_D^{*2}}\right).$$

Early repetitions are almost fresh in the model. Their penalty enters at second order in $R_D/R_D^*$.

For $R_D\rightarrow\infty$,

$$D_{eff} \rightarrow U_D(1+R_D^*),$$

while $D_{total}=U_D(1+R_D)$ continues to grow without bound. Compute and effective data have separated.

The derivative gives an even clearer contract. Holding $U_D$ fixed, $D_{total}=U_D(1+R_D)$, so

$$\frac{dD_{eff}}{dD_{total}} = \frac{dD_{eff}/dR_D}{dD_{total}/dR_D} = e^{-R_D/R_D^*}.$$

The marginal value of another repeated token is one at $R_D=0$ and decays exponentially. At $R_D=R_D^*$ it is $e^{-1}\approx0.368$ of a fresh token. The average value of the first $R_D^*$ repeats is $1-e^{-1}\approx0.632$. Calling $R_D^*$ a literal half-life is therefore imprecise: the marginal half-value occurs at $R_D=R_D^*\log2$.

This derivation is a source-backed phenomenological assumption. The geometric series makes the decay interpretation explicit; it does not prove that SGD extracts an independent constant fraction of information on every pass.

0x03D A Numeric Return Ledger

Muennighoff et al. fit their complete model on 182 samples and report $R_D^*\approx15.3878$ for their setup. Keep that number source-scoped. With $U_D=100$ billion tokens, the law gives:

Epochs $E_D$	Repeats $R_D$	Processed $D_{total}/U_D$	Effective $D_{eff}/U_D$	Marginal value of next repeated token	Average value of repeated portion
1	0	1.000	1.000	1.000	not applicable
4	3	4.000	about 3.73	about 0.823	about 0.909
about 16.39	15.39	about 16.39	about 10.73	0.368	0.632
41	40	41.000	about 15.24	about 0.074	about 0.356
unbounded	unbounded	unbounded	approaches 16.39	approaches 0	approaches 0

The four-epoch row explains the paper’s source-reported observation that a few complete repeats can be close to fresh data under its experimental contract. It does not make four epochs universally free. The conversion depends on model family, data distribution, optimizer, schedule, deduplication, regularization, and whether repetition is uniformly distributed.

Numerically stable code should avoid subtracting two nearly equal numbers when $R_D/R_D^*$ is small:

from dataclasses import dataclass
from math import exp, expm1, inf


@dataclass(frozen=True)
class EffectiveAxis:
    base: float
    repeat_count: float
    repeat_scale: float
    effective: float
    marginal_value: float
    elasticity: float


def effective_repeated_axis(base, repeat_count, repeat_scale):
    if base <= 0:
        raise ValueError("base support must be positive")
    if repeat_count < 0:
        raise ValueError("repeat count cannot be negative")
    if repeat_scale <= 0:
        raise ValueError("repeat scale must be positive")

    if repeat_scale == inf:
        effective = base * (1.0 + repeat_count)
        marginal = 1.0
    else:
        # -expm1(-x) computes 1 - exp(-x) stably near x = 0.
        gained_repeats = repeat_scale * -expm1(
            -repeat_count / repeat_scale
        )
        effective = base * (1.0 + gained_repeats)
        marginal = exp(-repeat_count / repeat_scale)

    processed = base * (1.0 + repeat_count)
    elasticity = (processed / effective) * marginal

    return EffectiveAxis(
        base=base,
        repeat_count=repeat_count,
        repeat_scale=repeat_scale,
        effective=effective,
        marginal_value=marginal,
        elasticity=elasticity,
    )

The returned elasticity will matter for allocation. A data axis with tiny marginal value should not be treated as if doubling processed tokens doubles the useful data coordinate.

0x03E The Source Also Decays Excess Parameters

Replacing only $D$ by $D_{eff}$ leaves another implausible extrapolation. In the separable Chinchilla surface, increasing $N$ from one billion to ten billion produces the same parameter-term reduction whether the model sees one token or one trillion. Muennighoff et al. therefore introduce a symmetric effective parameter construction.

First, use the abundant-data allocation law to find the one-epoch compute-optimal parameter count for $U_D$. Call the usable base

$$U_N = \min\left(N_{opt}(D_{opt}=U_D),N\right).$$

Then define the excess-parameter ratio

$$R_N = \max\left(\frac{N}{U_N}-1,0\right),$$

and its diminishing-return coordinate

$$N_{eff} = U_N + U_NR_N^* \left(1-e^{-R_N/R_N^*}\right).$$

The final source family is

$$\boxed{ L = E + \frac{A}{N_{eff}^{\alpha}} + \frac{B}{D_{eff}^{\beta}} }.$$

Renderer-safe two-axis contract:

U_N = min(N_opt_at_one_epoch(U_D), N)
R_N = max(N / U_N - 1, 0)
N_eff = U_N + U_N * R_N_star * (1 - exp(-R_N / R_N_star))

L = E + A / N_eff^alpha + B / D_eff^beta

When $R_D=R_N=0$, it reduces to the abundant-data law. When $R_D^*,R_N^*\rightarrow\infty$, repeated tokens and excess parameters retain full value and the original law is recovered.

The source fit reports

$$R_N^*\approx5.3097, \qquad R_D^*\approx15.3878.$$

Because its fitted excess-parameter return decays sooner than repeated-data return, its data-constrained frontier allocates additional compute to epochs faster than to parameters. This conclusion belongs to the source experiment. The phrase “parameter repetition” is a mathematical analogy, not a claim that network parameters are duplicate samples or independent observations.

The empirical and fitted iso-loss contours make the change in allocation visible:

Article-authored reconstruction of Muennighoff et al., Scaling Data-Constrained Language Models, Figure 3. It preserves the empirical-versus-fitted comparison, the source scope of 93 models and 100M unique tokens, and the distinction between fresh-data and data-constrained frontiers. Contours and points are schematic rather than copied or digitized measurements.

Three limitations are structural:

1. $U_N$ inherits the abundant-data law used to define it. An error in that baseline propagates into both repeat coordinates.
2. The symmetric parameter saturation is phenomenological. It is not derived from tensor rank, optimization dynamics, or an information-capacity theorem.
3. Both effective axes are monotone. They can plateau but cannot predict that more epochs or more parameters make held-out loss worse.

The source acknowledges the third limitation and excludes harmful/outlying regions from its main fit. The formula must not be used to erase those regions when a new experiment observes them.

0x03F The Compute-Optimal Tangent Changes

Under the abundant-data approximation $C=kND$, moving along fixed compute gives $d\log D/d\log N=-1$. In M4.7 Section 0x04, equalizing the two marginal loss terms produced

$$\alpha F_N=\beta F_D,$$

where $F_N=AN^{-\alpha}$ and $F_D=BD^{-\beta}$.

With effective axes, define their local elasticities with respect to the physical coordinates:

$$\varepsilon_N = \frac{d\log N_{eff}}{d\log N}, \qquad \varepsilon_D = \frac{d\log D_{eff}}{d\log D_{total}}.$$

The parameter term changes along the frontier as

$$\frac{dF_N}{d\log N} = -\alpha F_N\varepsilon_N,$$

while the data term changes as

$$\frac{dF_D}{d\log N} = \beta F_D\varepsilon_D,$$

because increasing $N$ at fixed $C$ decreases physical $D$. The new tangent is

$$\boxed{ \alpha F_N\varepsilon_N = \beta F_D\varepsilon_D }.$$

For fixed $U_D$ and the exponential repeat law,

$$\varepsilon_D = \frac{D_{total}}{D_{eff}} \frac{dD_{eff}}{dD_{total}} = \frac{(1+R_D)e^{-R_D/R_D^*}} {1+R_D^*(1-e^{-R_D/R_D^*})}.$$

Renderer-safe effective-axis tangent:

epsilon_N = d log(N_eff) / d log(N)
epsilon_D = d log(D_eff) / d log(D_total)

fixed-compute balance:
    alpha * F_N * epsilon_N = beta * F_D * epsilon_D

epsilon_D = ((1 + R_D) * exp(-R_D / R_D_star))
            / (1 + R_D_star * (1 - exp(-R_D / R_D_star)))

At $R_D=0$, $\varepsilon_D=1$. At large repetition it tends to zero. The same expression applies to $\varepsilon_N$ after replacing the data variables with the source’s parameter variables. The allocation is determined by both loss gaps and both elasticities. Looking only at $D_{eff}$ saturation and declaring “always buy more parameters” misses the source’s faster saturation of excess parameters.

The constrained problem is

$$\min_{N,D_{total}} L\left(N_{eff}(N,U_D),D_{eff}(D_{total},U_D)\right)$$

subject to

$$C_{exact}(N,D_{total})\le C_{budget}, \qquad U_D=\min(D_{total},D_C),$$

plus architecture, memory, batch, schedule, and data-loader feasibility. The closed-form abundant-data exponents no longer solve this piecewise nonlinear problem. Use the exact M4.1 cost ledger and enumerate executable neighbors.

from math import exp, inf

from scipy.optimize import minimize_scalar


def data_constrained_candidate(log_n, budget, data_cap, law, system):
    config = system.architecture_from_log_size(log_n)
    if config is None:
        return None

    flops_per_token = system.training_flops_per_token(config)
    d_total = system.round_tokens_down(budget / flops_per_token)
    if d_total <= 0:
        return None

    u_d = min(d_total, data_cap)
    r_d = max(d_total / u_d - 1.0, 0.0)
    d_axis = effective_repeated_axis(u_d, r_d, law.r_d_star)

    u_n = min(law.one_epoch_optimal_parameters(u_d),
              config.parameter_count)
    r_n = max(config.parameter_count / u_n - 1.0, 0.0)
    n_axis = effective_repeated_axis(u_n, r_n, law.r_n_star)

    predicted = (
        law.floor
        + law.a / n_axis.effective**law.alpha
        + law.b / d_axis.effective**law.beta
    )
    return {
        "config": config,
        "D_total": d_total,
        "D_unique": u_d,
        "epochs": 1.0 + r_d,
        "D_effective": d_axis.effective,
        "data_marginal": d_axis.marginal_value,
        "N_effective": n_axis.effective,
        "parameter_marginal": n_axis.marginal_value,
        "predicted_loss": predicted,
    }


def solve_data_constrained(budget, data_cap, law, system, bounds):
    def objective(log_n):
        report = data_constrained_candidate(
            log_n, budget, data_cap, law, system
        )
        return inf if report is None else report["predicted_loss"]

    result = minimize_scalar(
        objective,
        bounds=bounds,
        method="bounded",
    )

    # The continuous search only locates a region. The release candidate is
    # selected from exact executable neighbors under all runtime constraints.
    neighbors = system.executable_neighbors(exp(result.x))
    reports = [
        data_constrained_candidate(
            system.log_size(c), budget, data_cap, law, system
        )
        for c in neighbors
    ]
    reports = [r for r in reports if r is not None]
    return min(reports, key=lambda r: r["predicted_loss"]), reports

Production code must also reject candidates outside the validated N, D_unique, repetition, and compute region. A fitted saturation is not a license for thousand-epoch extrapolation.

0x03G Return And Allocation Are Different Experiments

Muennighoff et al. separate three protocols:

Protocol	Held fixed	Varied	Owner question
fixed unique data	unique-token cap $D_C$	parameters and epochs	how should extra compute be allocated under a finite corpus?
fixed FLOPs	compute and model size within each comparison	unique-data amount and therefore repeat count	how much return is lost when fresh data is replaced by repetition?
parametric fit	combined controlled records	$N$, unique data, epochs, compute	can one law interpolate and extrapolate both return and allocation?

These protocols cannot substitute for one another. At fixed FLOPs, a fresh token can dominate a repeated token while repeated training remains better than stopping when fresh data is unavailable. At fixed unique data, increasing both parameters and epochs can lower loss even though neither has the same return as in the abundant-data regime.

The source reports more than 400 training runs, models from roughly 10M to 9B parameters, up to 900B processed tokens, and extreme repetition sweeps. Its main parametric fit uses 182 selected samples over roughly 7M to 9B parameters and 1 to 500 epochs. These are source-reported experimental coordinates, not recommended production ranges.

Article-authored reconstruction of Muennighoff et al., Scaling Data-Constrained Language Models, Figure 1. It preserves the separation between return and allocation experiments and the source-scoped finite-data frontier, but its curves and markers are schematic rather than copied or digitized measurements.

The source’s compact “up to about four epochs” message needs its full qualifier:

under the paper's whole-corpus repetition setup,
at the studied fixed-compute comparisons,
validation-loss differences through a few epochs were small,
and downstream differences were not significant in its evaluated suite.

It does not imply that four passes over a contaminated benchmark, a tiny memorized subset, a narrow domain upsample, or a different optimizer are equivalent to four passes of fresh data.

The target metric must be held-out. The source’s appendix shows that training loss favors repeated-data runs because the model increasingly fits the reused corpus. Training loss therefore answers optimization fit, not return on unseen data. A data-constrained run should record both, but allocation must use the predeclared held-out distribution or downstream target.

0x03H A Monotone Return Law Cannot Represent Harm

The effective-axis model satisfies

$$\frac{dD_{eff}}{dR_D} = U_De^{-R_D/R_D^*}>0.$$

Therefore its data contribution to loss is non-increasing for every repeat count. It can express diminishing return and a plateau. It cannot express a validation-loss increase, epoch-wise double descent, or a memorization-induced reversal.

Hernandez et al. observe such a reversal under concentrated repetition. In one source design, 10% of training tokens come from a repeated subset. Test-loss degradation peaks in an intermediate region of subset repeat counts and coincides with the repeated-subset training loss approaching zero.

Source: Hernandez et al., Scaling Laws and Interpretability of Learning from Repeated Data, Figure 2, licensed under CC BY 4.0. The desktop image retains the attributed source figure; the mobile image is an attributed responsive adaptation of the double-descent and repeated-subset-overfitting panels. Model sizes, repeat frequencies, and effect magnitudes belong to the source experiment.

These papers are not contradictory replications of one experiment:

Axis	Muennighoff et al.	Hernandez et al.
repeated object	complete available training subset across epochs	a much smaller subset inside mostly unique training data
main control	unique-data cap, epochs, model size, compute	repeated-subset size, repeat count, model size, repeated-token fraction
modeled outcome	diminishing return and data-constrained allocation	non-monotonic degradation and mechanistic correlates
main law shape	monotone saturation	poor-performance region / double descent
validation contract	held-out test loss emphasized	held-out test loss plus repeated-subset train loss and copying analyses
transfer warning	repeat scales are setup-specific	concentrated duplicate damage is setup-specific

Muennighoff et al. explicitly note that their main formula cannot model epochs or parameters making performance worse; harmful/outlying regions are removed from the main fit. This is an estimator boundary, not evidence that the observations are invalid. If a new residual plot shows a reversal, preserve it, audit the exposure topology, and compare a harm-capable form. Do not let a monotone prior relabel the regime as noise automatically.

Possible responses include:

stop before the harmful region if the run policy permits early stopping
fit only the validated monotone regime and report its boundary
add a repetition-topology variable instead of one global R_D
use a transition/broken law selected by scale-ordered validation
model train and held-out losses jointly when memorization is the mechanism

The last three are new models and need new experiments. BNSL from 0x02 can describe a reversal in one dimension, but its break is descriptive until a controlled repetition intervention supports the mechanism.

0x03I Exposure Concentration Before Performance Equivalence

The epoch-equivalent ratio ignores how repeat mass is distributed. Define

$$w_i=m_i\ell_i, \qquad p_i=\frac{w_i}{\sum_jw_j},$$

where $w_i$ is processed token mass attributed to canonical unit $i$. Three useful exposure-support summaries are

$$n_{Kish} = \frac{\left(\sum_iw_i\right)^2}{\sum_iw_i^2},$$ $$n_{Simpson} = \frac{1}{\sum_ip_i^2},$$

and

$$n_{entropy} = \exp\left(-\sum_ip_i\log p_i\right).$$

Because $p_i=w_i/\sum_jw_j$, $n_{Kish}=n_{Simpson}$. The two names emphasize different interpretations: variance inflation under unequal weights versus inverse concentration. Entropy effective support is more sensitive to the full distribution and generally differs.

For $M$ equally weighted units, all three equal $M$. Repeating every unit four times leaves $p_i=1/M$ and the concentration support remains $M$, even though D_total quadruples. The diagnostic correctly says exposure stayed uniform; it does not say the fourth pass is fresh.

Now consider 100 equal-length units. Expose 90 units once and 10 units ten times. Every unit is still covered, so D_unique represents all 100 units. But

$$\sum_iw_i=90+10\cdot10=190,$$ $$\sum_iw_i^2=90+10\cdot10^2=1090,$$

and

$$n_{Kish} = \frac{190^2}{1090} \approx33.1.$$

The exposure mass is as concentrated, under this index, as roughly 33 equally weighted units. This still does not mean the network has exactly 33 fresh examples. It is an inventory alarm that one global repeat count hides a heavy tail.

Granularity matters. Document-level hashes miss repeated paragraphs across different documents; span-level hashes can overcount boilerplate fragments as independent units; token-level identity confuses common vocabulary with duplicate examples. Publish the canonical unit and run sensitivity at more than one granularity when duplication is central to the decision.

from collections import defaultdict
from math import exp, log


def exposure_ledger(loader_trace):
    emitted_tokens = defaultdict(float)
    covered_intervals = defaultdict(list)

    for event in loader_trace:
        # event must retain canonical lineage through tokenization and packing.
        for unit_id, start, end in event.canonical_token_ranges:
            emitted_tokens[unit_id] += end - start
            covered_intervals[unit_id].append((start, end))

    weights = list(emitted_tokens.values())
    d_total = sum(weights)
    d_unique = sum(
        interval_union_length(intervals)
        for intervals in covered_intervals.values()
    )

    probabilities = [w / d_total for w in weights]
    inverse_simpson = 1.0 / sum(p * p for p in probabilities)
    entropy_support = exp(
        -sum(p * log(p) for p in probabilities if p > 0)
    )

    return {
        "D_total": d_total,
        "D_unique": d_unique,
        "epoch_equivalent": d_total / d_unique,
        "inverse_simpson_units": inverse_simpson,
        "entropy_units": entropy_support,
        "maximum_token_share": max(probabilities),
        "multiplicity_quantiles": quantiles(weights),
    }

The code assumes each packed sequence retains source-span lineage. If packing drops that mapping, post-hoc token totals cannot reconstruct the exposure distribution exactly.

0x03J Three Objects Called “Effective Size”

D_effective and effective sample size are often discussed as if they were one interchangeable quantity. They are not.

Object	Equality it tries to preserve	Inputs	Typical unit	Valid transfer
exposure effective support	concentration of sampling mass	canonical multiplicities or probabilities	documents/spans	only under same canonicalization
variance-equivalent $n_{eff}$	variance of a named estimator	covariance or cluster model	residuals/runs/groups	only for that estimator and dependence model
performance-equivalent $D_{eff}$	fitted held-out loss under fresh versus repeated data	controlled training runs and scaling law	tokens	only inside validated training/data contract

One run can therefore have:

D_total = 400B processed tokens
D_unique = 100B corpus tokens
exposure support = 24M canonical documents
D_effective = 373B token-equivalents under one fitted law
fit n_eff = 12 independent configuration groups for one coefficient estimate

No arithmetic inconsistency exists because the numbers answer different questions. Trouble begins when an exposure index is inserted into a loss law, or when a fitted $D_{eff}$ is used as the number of independent residuals.

0x03K Deriving Variance-Equivalent Effective Sample Size

Suppose $X_1,\ldots,X_n$ are equal-variance observations for a mean estimator, with

$$\operatorname{Var}(X_i)=\sigma^2$$

and lag correlation

$$\operatorname{Corr}(X_i,X_{i+k})=\rho_k.$$

The mean has variance

$$\operatorname{Var}(\bar X) = \frac{1}{n^2} \sum_{i=1}^{n}\sum_{j=1}^{n} \operatorname{Cov}(X_i,X_j).$$

There are $n$ diagonal variance terms and $2(n-k)$ ordered off-diagonal pairs at lag $k$. Therefore

$$\operatorname{Var}(\bar X) = \frac{\sigma^2}{n} \left[ 1 + 2\sum_{k=1}^{n-1} \left(1-\frac{k}{n}\right)\rho_k \right].$$

Define $n_{eff}$ by equating this variance to the independent-sample variance $\sigma^2/n_{eff}$. Then

$$\boxed{ n_{eff} = \frac{n} {1+2\sum_{k=1}^{n-1}(1-k/n)\rho_k} }.$$

The familiar large-$n$ approximation removes the finite weighting:

$$n_{eff} \approx \frac{n}{1+2\sum_{k\ge1}\rho_k}.$$

This formula is appropriate only when an ordered dependence model makes sense for the named estimator. Treating arbitrary checkpoints sorted by model size as a stationary time series does not make it so.

For equal-size clusters of $m$ observations with within-cluster intraclass correlation $\rho$ and independent clusters, one cluster sum has variance

$$\operatorname{Var}\left(\sum_{j=1}^{m}X_j\right) = m\sigma^2 + m(m-1)\rho\sigma^2.$$

Relative to $m$ independent rows, the design effect is

$$DEFF = 1+(m-1)\rho,$$

so

$$n_{eff} = \frac{n}{1+(m-1)\rho}.$$

If 10 runs each contribute 10 highly correlated checkpoints and $\rho=0.8$, row count says n=100, while

$$DEFF=1+9(0.8)=8.2, \qquad n_{eff}\approx12.2.$$

This does not manufacture two tenths of a run. It says the mean’s variance under this simplified covariance model matches about 12.2 independent rows. For a scaling-law surface, using one predeclared endpoint per run is often cleaner than estimating checkpoint covariance from ten runs.

Effective sample size depends on:

the estimator (mean, slope, nonlinear allocation, quantile)
the weights and heteroskedasticity
which hierarchy level is considered random
the covariance model
the target population

Positive correlation generally reduces $n_{eff}$. Negative correlation can raise variance efficiency above the nominal row count for a particular mean, but that does not create more independently generalizable configurations. Do not use one scalar ESS to replace the complete group and covariance report.

0x03L Dependence-Aware Surface Fitting

A scaling corpus has several dependence layers:

canonical spans share documents and source domains
packed sequences share spans and training order
optimizer steps share model state
checkpoints share one trajectory
seeds share one configuration and data snapshot
configurations share architecture and training policy
studies may share pilots used to choose the grid

The fit does not need to model every token-to-token covariance. It does need to avoid counting deterministic descendants as independent scaling evidence.

Use the highest practical independent unit for each operation:

Operation	Minimum grouping rule	Why
train/validation/test scale split	complete configuration lineage	no seed or checkpoint twin crosses the boundary
coefficient bootstrap	configuration groups within design strata, then seeds if targeted	preserves experimental design and seed variation
trajectory model	complete run as one correlated sequence	checkpoint residuals share all prior optimizer state
data-repetition comparison	data snapshot and exposure policy as factors	repeat count is confounded otherwise
target pilot	new configuration group after pipeline freeze	prevents reuse of selection evidence

This extends the grouped bootstrap in M4.7 0x07E: the data-constrained pipeline adds data snapshot, canonicalization, and exposure-policy lineage to each group.

def make_scaling_groups(run_manifest):
    groups = {}

    for run in run_manifest:
        key = (
            run.architecture.config_hash,
            run.data.corpus_snapshot_hash,
            run.data.canonicalization_hash,
            run.data.tokenizer_hash,
            run.data.mixture_hash,
            run.data.exposure_policy_hash,
            run.optimization.recipe_family_hash,
            run.target_tokens,
        )
        groups.setdefault(key, []).append(run)

    for key, runs in groups.items():
        assert all_same_except(runs, allowed={"seed", "run_id"})
        yield ConfigurationGroup(key=key, independent_seed_runs=runs)


def scale_ordered_data_split(groups, compute_cutoffs):
    fit, selection, final = [], [], []

    for group in groups:
        # The complete group moves as one object. No checkpoint or seed is
        # allowed to leak across a scale boundary.
        if group.compute <= compute_cutoffs.fit_max:
            fit.append(group)
        elif group.compute <= compute_cutoffs.selection_max:
            selection.append(group)
        else:
            final.append(group)

    assert disjoint_lineage(fit, selection, final)
    return fit, selection, final

If several configurations share one fixed set of unique data, their losses are not independent evidence about arbitrary corpora. They are controlled evidence about allocation conditional on that corpus. Generalization across corpora requires independent data snapshots or a hierarchical data effect.

0x03M Fit The Complete Repetition Pipeline

The effective-axis law introduces parameters only identifiable through the right grid. One-epoch runs cannot identify $R_D^*$; runs with no excess model capacity cannot identify $R_N^*$; one fixed model size can confound both.

A minimum controlled matrix contains:

multiple immutable unique-data caps
multiple model sizes around and beyond each one-epoch optimum
multiple repeat counts spanning near-fresh and diminishing-return regions
independent seeds at selected configurations
held-out loss from a distribution excluded from all repeat pools
fresh-data fixed-FLOP controls where feasible

The three source protocols should remain visible in the records:

def build_data_constrained_records(runs, data_cap):
    records = []

    for run in runs:
        exposure = exposure_ledger(
            run.loader_trace,
        )
        u_d = min(exposure["D_unique"], data_cap)
        r_d = exposure["D_total"] / u_d - 1.0

        records.append({
            "protocol": run.experiment.protocol,
            "N": run.model.parameter_count,
            "C": run.compute.actual_training_flops,
            "D_total": exposure["D_total"],
            "D_unique": u_d,
            "R_D": r_d,
            "exposure_support": exposure["inverse_simpson_units"],
            "exposure_max_share": exposure["maximum_token_share"],
            "heldout_loss": run.evaluation.predeclared_heldout_loss,
            "training_loss": run.training.final_loss,
            "seed": run.optimization.seed,
            "configuration_group": run.configuration_group_id,
            "data_snapshot": run.data.corpus_snapshot_hash,
        })

    return records

Fit and compare at least these baselines:

Candidate	Effective data	Claim
fresh-token law	$D_{eff}=D_{total}$	every repeat is worth a fresh token
hard cap	$D_{eff}=\min(D_{total},D_C)$	repeats have zero return
exact geometric decay	$U_D\sum_{k=0}^{R_D}q^k$	each complete pass retains constant fraction $q$
source exponential approximation	$U_D[1+R_D^*(1-e^{-R_D/R_D^*})]$	smooth diminishing return with a finite plateau
harm-capable alternative	transition/broken/joint train-test form	return can reverse under evidenced topology

Use the same held-out objective, weighting, group splits, and decision score for comparison. Source-faithful replication can preserve the original Huber log-sum-exp objective separately, but the selection ledger must not rank one form on training loss and another on held-out loss.

The Muennighoff appendix fits $R_N^*$ and $R_D^*$ by multi-start L-BFGS while holding the abundant-data coefficients to its C4 estimates. It removes regions the monotone form cannot represent. A new implementation should report:

which baseline coefficients were fixed or refit
initialization grid and parameter bounds
all excluded configurations and immutable reasons
fit/selection/final group counts
whether harmful regions were excluded, censored, or modeled
optimizer convergence and bound hits
bootstrap failures as well as successes
allocation regret under each accepted form

Run the entire candidate-law and allocation pipeline inside each grouped bootstrap. Holding $R_D^*$ fixed while bootstrapping only $\alpha,\beta$ omits the uncertainty specific to the data-constrained decision.

0x03N Training And Runtime Integration

The law cannot be audited if the loader exposes only a scalar global step. Instrumentation should connect data identity to optimizer work:

offline corpus stage:
    immutable source snapshot
    canonical document/span IDs
    dedup and filter decisions
    tokenizer hash and token lengths

loader stage:
    canonical contributions to every packed sequence
    sampler probability and source mixture
    pass/epoch index where meaningful
    dropped, truncated, and padding tokens

training stage:
    tokens that actually contributed to loss
    optimizer step, learning rate, batch, and sequence count
    checkpoints tied to one run lineage

evaluation stage:
    held-out snapshot never entering the repeat pool
    training and held-out losses kept separate
    downstream tasks with contamination audit

Do not infer processed tokens from global_step * nominal_batch * sequence when gradient accumulation, variable sequence lengths, dropped batches, packing efficiency, or restart replay can change actual token work.

def train_step(batch, state, ledger):
    loss_mask = batch.loss_mask
    contributing_tokens = int(loss_mask.sum())

    ledger.processed_loss_tokens += contributing_tokens
    ledger.padding_tokens += int(batch.padding_mask.sum())

    for unit_id, start, end in batch.canonical_token_ranges:
        ledger.canonical_exposure_tokens[unit_id] += end - start
        ledger.canonical_covered_ranges[unit_id].add(start, end)

    loss = model(batch.tokens, loss_mask=loss_mask)
    loss.backward()

    if state.should_optimizer_step():
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

    return loss

Schedule semantics must follow actual horizon. A four-epoch run with a cosine schedule decaying over four epochs is not the same intervention as taking the four-epoch checkpoint from a forty-epoch schedule. If early stopping is used to avoid repetition harm, the stopping rule becomes part of the estimator and must be repeated inside validation/bootstrap rather than selected after seeing the final test curve.

At runtime, data scarcity can also change throughput: more epochs may improve cache locality and preprocessing reuse, while a larger model can reduce hardware utilization or exceed memory. Those effects belong in the exact cost ledger, not in $D_{eff}$. Performance-equivalent data predicts loss; it does not price wall time, storage, or pipeline engineering.

0x03O Design Choices And Their Boundaries

Choice	Useful when	Hidden assumption	Required sensitivity
count complete epochs	uniform full-corpus traversal	all units have comparable multiplicity	exposure histogram and partial-pass treatment
use token mass for $p_i$	variable document lengths matter to compute	canonical token contribution is traceable	document versus span granularity
use document count for support	document-level duplication is the target	long and short documents receive equal unit weight	token-mass comparison
fit monotone saturation	held-out return decays without reversal	harmful regimes lie outside policy/support	signed residuals and harm-capable candidate
stop at best held-out epoch	deployment permits early stopping	validation was not reused for law selection	nested stopping/selection protocol
aggregate one endpoint per run	terminal scaling law is target	trajectory shape is nuisance	separate trajectory audit
include every checkpoint	learning-curve law is target	within-run covariance is modeled	run-level clusters or trajectory likelihood
share $R_D^*$ across corpora	evidence supports invariant repeat return	data quality/topology do not change decay	corpus-by-repeat interaction
equate concentration ESS with $D_{eff}$	never justified by identity	exposure inequality predicts loss universally	keep ledgers separate

No scalar data-quality score repairs a missing exposure contract. If filtering changes both unique volume and average quality, fit a controlled quality/data intervention or treat the new corpus as a new study stratum. Do not attribute a changed repeat scale to “quality” after simultaneously changing deduplication, source mixture, tokenizer, and validation match.

0x03P Data-Constraint Failure Lab

Failure	Symptom	Why the conclusion fails	Response
D_total called dataset size	repeated run appears to have four times more data	optimizer work and corpus support are conflated	report D_total, D_unique, and exposure policy
D_unique means unique vocabulary IDs	corpus seems tiny and model-independent	unit definition is wrong	count canonical token occurrences in source units
one epoch ratio used for subset upsampling	same $R_D$ predicts unlike runs	exposure topology is hidden	multiplicity distribution and source-specific repeats
four epochs declared free	source result transferred as a constant	repeat scale is contract-specific	local fixed-FLOP controls and intervals
training loss selects allocation	repeated data looks superior	memorization lowers train loss	predeclared held-out or target metric
$D_{eff}$ inserted into compute	repeated tokens vanish from FLOPs	performance equivalence is not work accounting	compute with D_total
exposure ESS inserted into loss law	concentrated corpus gets arbitrary token conversion	concentration is not learned return	fit performance-equivalent law separately
checkpoints treated as independent runs	confidence bands collapse	shared trajectory is counted repeatedly	endpoint estimator or run-cluster model
seeds cross scale split	extrapolation appears easy	configuration twins leak across roles	split complete lineage groups
monotone law fits double descent	harmful region labeled outlier	family cannot express reversal	preserve regime and compare harm-capable form
harmful points removed silently	smooth plateau appears universal	exclusion selects the desired geometry	immutable exclusion ledger and restored sensitivity
repeat scale held fixed in bootstrap	frontier band is too narrow	data-constraint uncertainty is omitted	refit complete pipeline per grouped sample
nominal steps estimate tokens	restarts/packing distort D_total	schedule counter is not token work	runtime loss-token counter
dedup changes validation too	loss improvement attributed to train diversity	target distribution moved	immutable held-out snapshot
full-corpus law applied to mixture weighting	upsampled domain treated as redundant noise	target and exposure distribution both changed	explicit mixture variable and domain metrics

0x03Q Data-Constraint And Dependence Review Gate

Before accepting a data-constrained recommendation, answer:

• Are D_total, D_unique, repeat count, epoch-equivalent exposure, and D_effective separate fields?
• Is the canonical unit defined at document, span, or another auditable level?
• Does the loader preserve source lineage through tokenization and packing?
• Is repetition uniform, subset-concentrated, source-weighted, or accidental?
• Are exposure concentration and performance-equivalent data kept distinct?
• Is the exact geometric series distinguished from the source’s exponential approximation?
• Are the small-repeat expansion, large-repeat limit, marginal value, and elasticity derived and tested?
• Is $R_D^*$ source-scoped rather than treated as a universal epoch constant?
• Is the symmetric $N_{eff}$ construction labeled phenomenological?
• Does allocation use D_total for compute and the effective axes only for fitted performance?
• Are fixed-data, fixed-FLOP, and joint-fit protocols interpreted separately?
• Is held-out loss used where training loss would reward memorization?
• Can the candidate family represent observed degradation or double descent?
• Are excluded harmful regions restored in a sensitivity analysis?
• Are checkpoints, seeds, configurations, data snapshots, and recipe lineages grouped correctly?
• Is variance-equivalent $n_{eff}$ tied to a named estimator and covariance model?
• Are all data-dependent fitting, model-selection, stopping, and allocation steps repeated inside the grouped bootstrap?
• Is the final recommendation inside validated model, unique-data, epoch, and compute support?

The data-constraint analysis changes the feasible surface. Once fresh data is capped, processed tokens remain a compute coordinate, unique tokens become a resource constraint, exposure concentration becomes a diagnostic, and effective data becomes a fitted performance coordinate. None can replace the others.

M4.9 Section 0x01 asks a related but distinct question: when training deliberately continues beyond a compute-optimal token ratio, how should the article model over-training, average downstream quality, individual-task failures, and metric-created thresholds without turning every nonlinearity into “emergence”?

0xFC Final Formula, Source, Render, And Boundary Gate

The final audit freezes the prediction-and-data-constraint contract. It does not choose another functional form after seeing the final scale and does not turn effective data into a new physical accounting coordinate.

Formula contract

Formula family	Semantic status	Publication guard
scale-ordered extrapolation metrics	rewrite evaluation contract	fit, selection, final test, and target pilot remain complete disjoint groups
$M_1$-$M_4$ source families	source-local comparison	source sign convention, domain, floor, and extrapolation orientation remain explicit
monotone transition and BNSL alternatives	source-backed candidate forms	transition count, initialization, bounds, complexity penalty, and hidden-scale selection are frozen
unique, processed, repeated, and effective data	operational definitions plus a source-backed phenomenological law	physical work never uses D_eff; data identity and exposure topology remain registered
effective parameter and data axes	source-local fitted construction	abundant-data baseline, monotonicity, and harmful-region exclusions remain visible
dependence and effective sample size	exact covariance example plus rewrite audit contract	variance-equivalent sample size is not renamed as unique support or performance-equivalent data

The publication surface must retain 79 non-empty display equations, balanced delimiters and fences, and renderer-safe forms for the source families, BNSL, the effective-data construction, the two-axis law, and the modified tangent. Every symbol must be defined locally or inherited through an explicit M4.7 handoff. Source equations, article derivations, fitted constants, and synthetic examples remain distinguishable.

Source and visual contract

• The four primary source revisions in 0x00H are immutable inputs to this article’s final audit.
• Run counts, repeat ranges, fitted constants, break behavior, and harmful regimes remain source-reported rather than universal defaults.
• Every retained source figure carries figure number, license, experimental scope, and adaptation status; every reconstruction is identified as an article-authored semantic explanation.
• The visual-rights manifest must cover every paper-derived semantic figure and agree with the responsive assets used by the article.

Render and accessibility contract

• All 11 picture elements have meaningful alternative text and resolvable desktop/mobile assets.
• Every repository-authored SVG has intrinsic dimensions, a viewBox, an accessible image role, title, and description.
• Desktop/mobile light and dark renders contain all 79 completed equations, no error markup, no broken image, no duplicate heading ID, and no article- or document-level horizontal overflow.
• Wide equations, tables, and code blocks remain keyboard-scrollable inside bounded containers, with text fallbacks for the operational formula families.

Cross-article owner contract

Owner	M4.8 consumes or hands forward	M4.8 does not duplicate
M4.7	frozen run manifest, fitted candidates, fit draws, and numerical certificates	baseline-law derivation, compute-optimal solution, or optimizer certification
M4.8	scale-ordered prediction tests, form selection, finite-data coordinates, dependence, and explicit abort regions	lifecycle deployment choice
M4.9	receives validated prediction draws, support bounds, and failure/abort status	downstream utility mapping, demand scenarios, lifecycle cost, and pilot selection
M5.1	later supplies measured serving state to M4.9	K/V lifecycle, batching, scheduling, or cache mechanisms
M8.10	later qualifies the exact deployment artifact consumed by M4.9	compression, artifact release, and rollback workflow

The final evidence state is source-audited extrapolation methodology with synthetic invariants, not a locally reproduced repeated-data study. The article validates what a study must measure and withhold; it does not claim a new universal data-return constant.

0xFD Figure Provenance And Rights

The article uses two different visual objects:

licensed source figure:
    attributed source evidence retained under an explicit reuse license

article-authored reconstruction:
    article-authored explanation of a mechanism or source-reported geometry;
    no paper artwork or digitized measurements; not new empirical evidence

An arXiv rights link records the terms selected for that submission; the non-exclusive distribution license is not treated here as a general permissive reuse license. Source artwork governed only by that record is not served locally. CC-licensed figures retain the named license and attribution, and responsive adaptations are identified as adaptations.

Source figures used	Desktop evidence	Mobile treatment	Rights record
Alabdulmohsin et al., Figures 1, 3	article-authored semantic reconstructions; source artwork not served	independent responsive reconstructions	paper link retained; submission uses arXiv non-exclusive distribution
Caballero et al., Figure 1	attributed CC BY source figure	attributed responsive adaptation	CC BY 4.0
Hernandez et al., Figures 1, 2	attributed CC BY source figures	attributed responsive adaptations	CC BY 4.0
Muennighoff et al., Figures 1, 3	article-authored semantic reconstructions; source artwork not served	independent responsive reconstructions	paper link retained; submission uses arXiv non-exclusive distribution

0xFF References

• Alabdulmohsin et al., “Revisiting Neural Scaling Laws in Language and Vision”, 2022.
• Caballero et al., “Broken Neural Scaling Laws”, 2023.
• Hernandez et al., “Scaling Laws and Interpretability of Learning from Repeated Data”, 2022.
• Muennighoff et al., “Scaling Data-Constrained Language Models”, 2023.