Predicting loss within an experimental regime
Neural scaling laws are empirical relationships between training resources and model loss. A fitted curve can help allocate a larger run when the model family, data distribution, tokenizer, and training procedure remain comparable. Its coefficients must come from experiments; knowing an exponent alone does not determine the final loss.
The distinction between resources matters. Parameters describe model size, tokens describe data processed, and compute pays for processing those tokens with that model. The figures distinguish authored toy curves, calculations from published fits, and estimates based on editable hardware assumptions.
Capacity, data, and budget
A larger kitchen cannot compensate for missing ingredients, and more ingredients still require cooking time. The analogy is limited, but it helps separate model capacity, available data, and the compute needed to use them together.
Try changing one resource while holding the others fixed. The figure makes its toy loss and budget constraint explicit.
Capacity, ingredients, and a cooking budget
An authored resource analogy: model capacity and data can each limit a recipe, while compute pays for processing that data. The toy loss below is defined explicitly; it is not a language-model measurement.
The allocation fits the selected toy budget.
Power-law relationships
A common single-resource fit has the form:
Here x is a resource measured in stated units, a is a fitted scale, and α controls how quickly the reducible part of the loss falls. The offset can represent a limiting loss within the chosen model; a fitted offset is not a proof of the dataset's entropy.
Kaplan et al. (2020) reported parameter- and data-limited exponents near 0.076 and 0.095 in their experimental regime. With exponent 0.076, a 10× increase multiplies a pure power-law term by about 0.84. That is not a universal reduction in total loss when an offset or another bottleneck is present. Comparing exponents alone also does not compare the cost of adding parameters versus data.
A frequently used joint fitting family is:
This is a model to fit, not a statement that data and capacity act independently in every training regime. Hoffmann et al. (2022) used this family as one of their approaches to training allocation.
Change the exponent below to see its effect on explicitly normalized toy curves. The x-axis is a multiplier of a separate reference resource for each line, not interchangeable parameter, token, and FLOP units.
Inspect a power-law shape
Authored normalized curves L(x) = 1.69 + A x⁻ᵅ. x is a multiplier of a separate reference resource for each curve, so the lines illustrate shape rather than equal-cost alternatives.
Scroll the chart horizontally if it does not fit.
A 10× increase multiplies the reducible loss term by 0.839. The irreducible offset remains 1.69.
Training-compute optimal allocation
For a dense transformer approximation:
N is the parameter count used by the approximation, and D is the number of training tokens processed. This operation count omits some architecture-dependent costs and does not measure wall-clock time.
Kaplan's historical compute-efficient fit favors N growing approximately as Cmin^0.73, with a slower increase in data. Cmin is an idealized minimum-compute quantity tied to their batch-size analysis, not simply a hardware invoice. The figure uses the published parameter fit in PF-days and reconstructs a token budget with C≈6ND; this reconstructed D is not the paper's independently fitted D coefficient.
Hoffmann and colleagues found approximately equal scaling of parameters and tokens. Their Chinchilla model used 70B parameters and 1.4T tokens, with the same training compute budget as Gopher, which had 280B parameters. Chinchilla outperformed Gopher and several other models on the evaluated tasks. Four times fewer parameters is not four times less training compute.
For a simple 20:1 token-to-parameter rule, combining D=20N with C≈6ND gives N=√(C/120). Equal scaling exponents alone do not determine the ratio 20; it is a reference approximation, and fitted coefficients depend on the setup.
Two training allocation prescriptions
Compare a calibrated historical Kaplan fit with the simplified 20-token-per-parameter Chinchilla rule. These are calculated allocations under different assumptions, not a new empirical fit or a hardware budget forecast.
Scroll the chart horizontally if it does not fit.
Kaplan historical fit (amber)
20:1 approximation (teal)
Both allocations satisfy 6ND = 1.00e+23 FLOPs.
From operations to an elapsed-time estimate
Convert FLOPs to time only after stating throughput and utilization assumptions. GPU count, communication, sequence length, precision, data loading, and checkpointing all affect the realized result.
The planner exposes those assumptions and calculates an idealized time and cost. It performs no benchmark and reports no predicted model accuracy.
A transparent compute estimate
Calculate 6ND FLOPs and convert that operation count using explicit throughput, utilization, and price assumptions. These results are estimates, not measured training time or a cloud quote.
Training and serving are different objectives
Training-optimal allocation seeks the lowest loss for a training compute budget. It does not by itself optimize the total cost of training and serving a model. If deployment requires many predictions, spending extra training compute on a smaller model can reduce later serving cost. LLaMA (2023) explicitly discusses this distinction.
Compare objectives before comparing allocations. Sparse mixture-of-experts models also require distinguishing total parameters from parameters activated per token.
Kaplan (2020)
- Objective
- Minimize language-model loss for an idealized training compute budget.
- Allocation
- Historical fit: N ∝ Cmin^0.73, with data growing more slowly. The token-to-parameter ratio is not constant.
- Limit
- Fits are tied to the original model family, data, and compute assumptions.
Chinchilla (2022)
- Objective
- Minimize loss within a fixed training compute budget.
- Allocation
- Approximately equal scaling of parameters and training tokens. Roughly 20 tokens per parameter is a useful simplified reference, not a universal constant.
- Limit
- The same training budget can favor a smaller, longer-trained model than earlier prescriptions.
Inference-aware training
- Objective
- Account for expected deployment usage as well as training cost.
- Allocation
- Training a smaller model on more tokens can reduce serving cost for a target quality.
- Limit
- Extra training cost must be compared with actual serving savings, hardware, and workload.
Sparse mixture of experts
- Objective
- Increase available parameters while activating a subset for each token.
- Allocation
- Separate total parameters, active parameters, routing, and shared layers when counting compute.
- Limit
- Communication, expert balance, and memory complicate dense-model approximations.
Training-optimal and lifecycle-optimal allocations answer different questions. None of these approaches supplies a universal cost or accuracy ranking.
Common pitfalls
Extrapolating a fitted curve too far
A power law summarizes the observed regime. Validate intermediate scales and report uncertainty before using it far beyond the fitted range. A loss curve also does not establish a universal threshold for a specific downstream capability.
Treating tokens as interchangeable
Data distribution, repetition, contamination, filtering, and tokenization affect what is learned. A token count alone does not certify data quality. Repeated-data training can have a different scaling regime from fresh-data training.
Confusing FLOPs, time, and money
A theoretical operation count is not measured latency. Report hardware, precision, workload, utilization, and the measurement boundary for a timing claim. State whether an optimization concerns training cost, serving cost, or both.
Related concepts
Explore emergent abilities in large language models: sudden capabilities at scale thresholds, phase transitions, and the mirage debate.
Master prompt engineering for large language models: from basic composition to Chain-of-Thought, few-shot, and advanced techniques.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Understand cross-entropy loss for classification: interactive demos of binary and multi-class CE, the -log(p) curve, softmax gradients, and focal loss.
Understand dilated (atrous) convolutions: how dilation rates expand receptive fields without extra parameters and how to avoid gridding artifacts.
Learn how gradients propagate through deep neural networks during backpropagation. Understand vanishing and exploding gradient problems.
