Start with the task and the measurement
An “emergent ability” is often used to describe a capability whose measured performance appears absent at smaller scales and present at larger scales. Wei et al. (2022) document this pattern in particular model families and tasks. The observation alone does not identify a universal parameter threshold or prove a physical phase transition inside the model.
The metric matters from the outset. Exact-match accuracy can change sharply when a final answer crosses from one wrong token to no wrong tokens. A continuous measure of the same predictions can tell a different story. Training data, architecture, prompting, decoding, and the amount of evaluation data also affect the result.
A phase-transition analogy, with limits
Water changes phase near its melting and boiling points under specified pressure. This provides an analogy for an abrupt observable change, not a mechanism for language-model abilities. Model parameters are not temperature, and no physical phase diagram establishes an order in which translation, coding, or reasoning must appear.
The authored particle schematic assumes water near one atmosphere and omits mixed-phase behavior and latent heat. Use it only to visualize the idea of changing an observable state.
Ice (Solid) · Authored schematic at about one atmosphere; no model-size mapping.
A threshold can belong to the evaluation rule
Suppose a benchmark declares a task successful only when at least six of eight answers exactly match a reference. That criterion is a choice made by the evaluator. Moving the criterion can change the success label without changing any model prediction.
The next figure uses eight explicit toy predictions, each with six binary token choices. Change the prediction progress and the required exact-match fraction separately. The progress control is not a real model-size axis.
Example 1
1/6 tokens correct
Not exact
Example 2
1/6 tokens correct
Not exact
Example 3
2/6 tokens correct
Not exact
Example 4
3/6 tokens correct
Not exact
Example 5
3/6 tokens correct
Not exact
Example 6
4/6 tokens correct
Not exact
Example 7
5/6 tokens correct
Not exact
Example 8
6/6 tokens correct
Exact match
Eight authored binary-token predictions, six tokens each. Greedy token correctness uses p(correct) ≥ 0.5. The criterion slider changes only the success label; it does not change any prediction. Progress is a toy input, not a parameter count or a fitted scaling variable.
The length of an answer changes the score
Exact match requires every token in the answer to be correct. A longer answer therefore offers more ways to fail, even when its token-level probabilities come from the same smooth rule. Token decisions are discrete too, but token accuracy gives partial credit that exact match withholds.
Compare prefixes of the same authored probability sequence at several answer lengths. These are measurement scenarios, not real abilities unlocking in a fixed order.
1-token answer
2-token answer
4-token answer
8-token answer
All four scenarios use prefixes of the same authored token-probability rule. A longer answer requires more correct token decisions to achieve exact match. This does not assign real abilities an unlock order or a universal model-size threshold.
Rescoring the same predictions
Schaeffer et al. (2023) argue that some reported emergence can arise from nonlinear or discontinuous evaluation metrics. Their experiments rescore fixed outputs; changing the metric need not change the underlying model.
Here, each toy token has a probability assigned to its correct alternative and a complementary probability assigned to an incorrect alternative. Greedy decoding selects the more likely one. Exact match counts fully correct answers, token accuracy counts correct individual decisions, and negative log likelihood uses the probabilities directly.
Toggle among those metrics, then inspect the probabilities. All curves are calculated from the same explicit rule and the same eight examples at each progress value. Smooth probability changes can yield abrupt exact-match changes.
Rescore the same predictions
Switch the metric while keeping all authored probabilities and greedy answers fixed. Inspect the token probabilities to see exactly what each score measures.
Scroll the chart horizontally if it does not fit.
Inspect all probabilities and token decisions
| Example | Token 1 | Token 2 | Token 3 | Token 4 | Token 5 | Token 6 |
|---|---|---|---|---|---|---|
| 1 | 0.54 ✓ | 0.51 ✓ | 0.47 × | 0.44 × | 0.40 × | 0.36 × |
| 2 | 0.57 ✓ | 0.53 ✓ | 0.50 × | 0.46 × | 0.43 × | 0.39 × |
| 3 | 0.59 ✓ | 0.56 ✓ | 0.52 ✓ | 0.49 × | 0.45 × | 0.42 × |
| 4 | 0.61 ✓ | 0.58 ✓ | 0.54 ✓ | 0.51 ✓ | 0.47 × | 0.44 × |
| 5 | 0.64 ✓ | 0.60 ✓ | 0.57 ✓ | 0.54 ✓ | 0.50 ✓ | 0.46 × |
| 6 | 0.67 ✓ | 0.63 ✓ | 0.59 ✓ | 0.56 ✓ | 0.53 ✓ | 0.49 × |
| 7 | 0.69 ✓ | 0.66 ✓ | 0.62 ✓ | 0.59 ✓ | 0.55 ✓ | 0.52 ✓ |
| 8 | 0.72 ✓ | 0.68 ✓ | 0.65 ✓ | 0.61 ✓ | 0.58 ✓ | 0.54 ✓ |
This construction demonstrates a possible measurement mechanism. It does not prove that every empirical jump is a metric artifact or that every future capability is predictable. Assess those claims with task-specific evidence, uncertainty estimates, and additional metrics.
What the cited experiments establish
A useful claim names the model family, training setup, task, metric, and comparison. Code generation, for example, was evaluated in code-trained models up to 12B parameters, with nonzero HumanEval pass@1 already reported at 42M. Toolformer studies tool-augmented training rather than a generic parameter-count switch. GPT-4's technical report does not disclose its parameter count.
Program synthesis
- Model and setup
- Chen et al. (2021), code-finetuned GPT models from 12M to 12B parameters
- Task and metric
- HumanEval, executable unit tests, pass@k
- What follows
- The reported 42M model already has nonzero pass@1. The result depends on the code training and executable-test protocol.
Reasoning with demonstrations
- Model and setup
- Wei et al. (2022), multiple model families including PaLM 540B
- Task and metric
- Arithmetic, commonsense, and symbolic tasks; compare direct and chain-of-thought demonstrations
- What follows
- Benefits vary by task and scale in the studied models. The paper does not establish a universal parameter threshold for reasoning.
Tool-assisted prediction
- Model and setup
- Schick et al. (2023), Toolformer with GPT-J 6.7B plus smaller-model scaling experiments
- Task and metric
- API-augmented language-model training and downstream tasks
- What follows
- Tool behavior depends on tool-augmented training and evaluation. Parameter count alone does not specify whether tool behavior will appear.
Metric-dependent emergence
- Model and setup
- Schaeffer et al. (2023), fixed predictions rescored across tasks
- Task and metric
- Discontinuous versus continuous evaluation metrics
- What follows
- Apparent jumps can change when the metric changes. This is evidence about the evaluated setups, not a proof that all future capabilities are predictable.
GPT-4 reporting boundary
- Model and setup
- OpenAI (2023), GPT-4 technical report
- Task and metric
- Reported benchmark evaluations
- What follows
- The report withholds model size and architecture details. No parameter coordinate or universal theory-of-mind threshold can be inferred from it.
Compare observations only with their training setup, prompting protocol, task, and metric. These studies do not imply a consistent ordering of ability thresholds across model families.
Common pitfalls
Turning observations into universal thresholds
An observed improvement in one family is not a budget rule for another. Parameters, training tokens, data composition, and prompting can change together. Avoid attributing the result to parameter count alone.
Reporting only a binary score
Alongside exact match, examine partial credit or token probabilities when meaningful, sample counts, confidence intervals, and failure cases. A score of zero in a finite sample does not prove a success probability of exactly zero.
Treating an analogy as a theory
A sigmoid can fit a curve without identifying the cause of that curve. A threshold diagram or an information-capacity metaphor does not establish which internal computations exist. Claims about mechanisms need interventions and evidence beyond task scores.
Related concepts
Explore neural scaling laws in deep learning: power law relationships between model size, data, and compute that predict AI performance.
Master prompt engineering for large language models: from basic composition to Chain-of-Thought, few-shot, and advanced techniques.
Explore how prompt token positions connect through causal self-attention, and why connectivity and attention weights do not establish causal influence.
Explore an illustrative region-tiling policy: resize tiles, calculate visual tokens, and distinguish analytical savings from measured model quality.
Learn batch normalization in deep learning: how normalizing layer inputs accelerates training, improves gradient flow, and acts as regularization.
BatchNorm normalizes over the batch and spatial axes; LayerNorm normalizes over the channel and spatial axes for each sample. The choice changes whether your model trains stably with batch=1, depends on batch composition at inference, and behaves consistently across train and eval.
