Skip to main content

Neural Scaling Laws Explained

Explore neural scaling laws in deep learning: power law relationships between model size, data, and compute that predict AI performance.

Predicting loss within an experimental regime

Neural scaling laws are empirical relationships between training resources and model loss. A fitted curve can help allocate a larger run when the model family, data distribution, tokenizer, and training procedure remain comparable. Its coefficients must come from experiments; knowing an exponent alone does not determine the final loss.

The distinction between resources matters. Parameters describe model size, tokens describe data processed, and compute pays for processing those tokens with that model. The figures distinguish authored toy curves, calculations from published fits, and estimates based on editable hardware assumptions.

Capacity, data, and budget

A larger kitchen cannot compensate for missing ingredients, and more ingredients still require cooking time. The analogy is limited, but it helps separate model capacity, available data, and the compute needed to use them together.

Try changing one resource while holding the others fixed. The figure makes its toy loss and budget constraint explicit.

Capacity, ingredients, and a cooking budget

An authored resource analogy: model capacity and data can each limit a recipe, while compute pays for processing that data. The toy loss below is defined explicitly; it is not a language-model measurement.

1×
1×
1×
Model-limited loss term: 1.000
Data-limited loss term: 1.000
Calculated toy loss: 3.690
Required / available compute: 1× / 1×

The allocation fits the selected toy budget.

Normalized toy model: L = 1.69 + N⁻⁰·³ + D⁻⁰·³. Compute relative to the N=D=1 baseline is ND, following C≈6ND. N and D are dimensionless multipliers. The illustrative coefficients do not predict a real training run.

Power-law relationships

A common single-resource fit has the form:

L(x) = a x^{-\alpha} + L_{\infty}

Here x is a resource measured in stated units, a is a fitted scale, and α controls how quickly the reducible part of the loss falls. The offset can represent a limiting loss within the chosen model; a fitted offset is not a proof of the dataset's entropy.

Kaplan et al. (2020) reported parameter- and data-limited exponents near 0.076 and 0.095 in their experimental regime. With exponent 0.076, a 10× increase multiplies a pure power-law term by about 0.84. That is not a universal reduction in total loss when an offset or another bottleneck is present. Comparing exponents alone also does not compare the cost of adding parameters versus data.

A frequently used joint fitting family is:

L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}

This is a model to fit, not a statement that data and capacity act independently in every training regime. Hoffmann et al. (2022) used this family as one of their approaches to training allocation.

Change the exponent below to see its effect on explicitly normalized toy curves. The x-axis is a multiplier of a separate reference resource for each line, not interchangeable parameter, token, and FLOP units.

Inspect a power-law shape

Authored normalized curves L(x) = 1.69 + A x⁻ᵅ. x is a multiplier of a separate reference resource for each curve, so the lines illustrate shape rather than equal-cost alternatives.

0.076

Scroll the chart horizontally if it does not fit.

Authored power-law curves with selectable resource series

A 10× increase multiplies the reducible loss term by 0.839. The irreducible offset remains 1.69.

A is 1, 1.2, and 0.8 for the parameter, data, and compute illustrations. The exponent is shared here only to expose its effect. Fitted exponents and offsets depend on the data, tokenizer, model family, and training setup.

Training-compute optimal allocation

For a dense transformer approximation:

C \approx 6ND

N is the parameter count used by the approximation, and D is the number of training tokens processed. This operation count omits some architecture-dependent costs and does not measure wall-clock time.

Kaplan's historical compute-efficient fit favors N growing approximately as Cmin^0.73, with a slower increase in data. Cmin is an idealized minimum-compute quantity tied to their batch-size analysis, not simply a hardware invoice. The figure uses the published parameter fit in PF-days and reconstructs a token budget with C≈6ND; this reconstructed D is not the paper's independently fitted D coefficient.

Hoffmann and colleagues found approximately equal scaling of parameters and tokens. Their Chinchilla model used 70B parameters and 1.4T tokens, with the same training compute budget as Gopher, which had 280B parameters. Chinchilla outperformed Gopher and several other models on the evaluated tasks. Four times fewer parameters is not four times less training compute.

For a simple 20:1 token-to-parameter rule, combining D=20N with C≈6ND gives N=√(C/120). Equal scaling exponents alone do not determine the ratio 20; it is a reference approximation, and fitted coefficients depend on the setup.

Two training allocation prescriptions

Compare a calibrated historical Kaplan fit with the simplified 20-token-per-parameter Chinchilla rule. These are calculated allocations under different assumptions, not a new empirical fit or a hardware budget forecast.

10^23.0

Scroll the chart horizontally if it does not fit.

Calibrated Kaplan and simplified Chinchilla parameter allocations

Kaplan historical fit (amber)

Parameters: 224.02B
Tokens: 74.40B
Tokens / parameter: 0.33

20:1 approximation (teal)

Parameters: 28.87B
Tokens: 577.35B
Tokens / parameter: 20.00

Both allocations satisfy 6ND = 1.00e+23 FLOPs.

Kaplan et al., Appendix A, Table 6: N = 1.3×10⁹(Cmin / 8.64×10¹⁹ FLOPs)^0.73, with non-embedding parameters and idealized minimum compute. Here D = C/(6N) reconstructs an approximate token budget, not the paper’s separately fitted D coefficient. Hoffmann et al. motivate roughly equal scaling; the simplified D = 20N rule gives N = √(C/120). Treat extrapolation beyond each study as uncertain.

From operations to an elapsed-time estimate

Convert FLOPs to time only after stating throughput and utilization assumptions. GPU count, communication, sequence length, precision, data loading, and checkpointing all affect the realized result.

The planner exposes those assumptions and calculates an idealized time and cost. It performs no benchmark and reports no predicted model accuracy.

A transparent compute estimate

Calculate 6ND FLOPs and convert that operation count using explicit throughput, utilization, and price assumptions. These results are estimates, not measured training time or a cloud quote.

1.00e+9
3.16e+11
256
312
40
$1.00
Training operations: 1.897e+21 FLOPs
Estimated elapsed time: 0.69 days
Estimated GPU-hours: 4.223e+3
Estimated cost: $4,223
Tokens per parameter: 316.23
Assumes dense training, perfect scaling across GPUs, constant utilization, and no extra overhead. Attention, embeddings, communication, checkpointing, data loading, validation, and failures can change the total. Hourly rate and GPU throughput are editable assumptions, not current product specifications.

Training and serving are different objectives

Training-optimal allocation seeks the lowest loss for a training compute budget. It does not by itself optimize the total cost of training and serving a model. If deployment requires many predictions, spending extra training compute on a smaller model can reduce later serving cost. LLaMA (2023) explicitly discusses this distinction.

Compare objectives before comparing allocations. Sparse mixture-of-experts models also require distinguishing total parameters from parameters activated per token.

Different scaling objectives

Kaplan (2020)

Objective
Minimize language-model loss for an idealized training compute budget.
Allocation
Historical fit: N ∝ Cmin^0.73, with data growing more slowly. The token-to-parameter ratio is not constant.
Limit
Fits are tied to the original model family, data, and compute assumptions.

Chinchilla (2022)

Objective
Minimize loss within a fixed training compute budget.
Allocation
Approximately equal scaling of parameters and training tokens. Roughly 20 tokens per parameter is a useful simplified reference, not a universal constant.
Limit
The same training budget can favor a smaller, longer-trained model than earlier prescriptions.

Inference-aware training

Objective
Account for expected deployment usage as well as training cost.
Allocation
Training a smaller model on more tokens can reduce serving cost for a target quality.
Limit
Extra training cost must be compared with actual serving savings, hardware, and workload.

Sparse mixture of experts

Objective
Increase available parameters while activating a subset for each token.
Allocation
Separate total parameters, active parameters, routing, and shared layers when counting compute.
Limit
Communication, expert balance, and memory complicate dense-model approximations.

Training-optimal and lifecycle-optimal allocations answer different questions. None of these approaches supplies a universal cost or accuracy ranking.

Common pitfalls

Extrapolating a fitted curve too far

A power law summarizes the observed regime. Validate intermediate scales and report uncertainty before using it far beyond the fitted range. A loss curve also does not establish a universal threshold for a specific downstream capability.

Treating tokens as interchangeable

Data distribution, repetition, contamination, filtering, and tokenization affect what is learned. A token count alone does not certify data quality. Repeated-data training can have a different scaling regime from fresh-data training.

Confusing FLOPs, time, and money

A theoretical operation count is not measured latency. Report hardware, precision, workload, utilization, and the measurement boundary for a timing claim. State whether an optimization concerns training cost, serving cost, or both.

If you found this explanation helpful, consider sharing it with others.

Mastodon