Skip to main content

Emergent Abilities in Large Language Models

Explore emergent abilities in large language models: sudden capabilities at scale thresholds, phase transitions, and the mirage debate.

Start with the task and the measurement

An “emergent ability” is often used to describe a capability whose measured performance appears absent at smaller scales and present at larger scales. Wei et al. (2022) document this pattern in particular model families and tasks. The observation alone does not identify a universal parameter threshold or prove a physical phase transition inside the model.

The metric matters from the outset. Exact-match accuracy can change sharply when a final answer crosses from one wrong token to no wrong tokens. A continuous measure of the same predictions can tell a different story. Training data, architecture, prompting, decoding, and the amount of evaluation data also affect the result.

A phase-transition analogy, with limits

Water changes phase near its melting and boiling points under specified pressure. This provides an analogy for an abrupt observable change, not a mechanism for language-model abilities. Model parameters are not temperature, and no physical phase diagram establishes an order in which translation, coding, or reasoning must appear.

The authored particle schematic assumes water near one atmosphere and omits mixed-phase behavior and latent heat. Use it only to visualize the idea of changing an observable state.

Phase transitions
-20°C
Rigid structureNo flow
Ice (Solid)Water (Liquid)Steam (Gas)
-30°C0°C (melting)100°C (boiling)150°C

Ice (Solid) · Authored schematic at about one atmosphere; no model-size mapping.

A threshold can belong to the evaluation rule

Suppose a benchmark declares a task successful only when at least six of eight answers exactly match a reference. That criterion is a choice made by the evaluator. Moving the criterion can change the success label without changing any model prediction.

The next figure uses eight explicit toy predictions, each with six binary token choices. Change the prediction progress and the required exact-match fraction separately. The progress control is not a real model-size axis.

A threshold in the evaluation rule
0.50
75%

Example 1

1/6 tokens correct

Not exact

Example 2

1/6 tokens correct

Not exact

Example 3

2/6 tokens correct

Not exact

Example 4

3/6 tokens correct

Not exact

Example 5

3/6 tokens correct

Not exact

Example 6

4/6 tokens correct

Not exact

Example 7

5/6 tokens correct

Not exact

Example 8

6/6 tokens correct

Exact match

Eight authored binary-token predictions, six tokens each. Greedy token correctness uses p(correct) ≥ 0.5. The criterion slider changes only the success label; it does not change any prediction. Progress is a toy input, not a parameter count or a fitted scaling variable.

The length of an answer changes the score

Exact match requires every token in the answer to be correct. A longer answer therefore offers more ways to fail, even when its token-level probabilities come from the same smooth rule. Token decisions are discrete too, but token accuracy gives partial credit that exact match withholds.

Compare prefixes of the same authored probability sequence at several answer lengths. These are measurement scenarios, not real abilities unlocking in a fixed order.

Answer length changes exact-match difficulty
0.55

1-token answer

Exact matches: 8/8
Correct tokens: 8/8
Mean correct-token probability: 0.628

2-token answer

Exact matches: 8/8
Correct tokens: 16/16
Mean correct-token probability: 0.610

4-token answer

Exact matches: 5/8
Correct tokens: 27/32
Mean correct-token probability: 0.575

8-token answer

Exact matches: 0/8
Correct tokens: 34/64
Mean correct-token probability: 0.505

All four scenarios use prefixes of the same authored token-probability rule. A longer answer requires more correct token decisions to achieve exact match. This does not assign real abilities an unlock order or a universal model-size threshold.

Rescoring the same predictions

Schaeffer et al. (2023) argue that some reported emergence can arise from nonlinear or discontinuous evaluation metrics. Their experiments rescore fixed outputs; changing the metric need not change the underlying model.

Here, each toy token has a probability assigned to its correct alternative and a complementary probability assigned to an incorrect alternative. Greedy decoding selects the more likely one. Exact match counts fully correct answers, token accuracy counts correct individual decisions, and negative log likelihood uses the probabilities directly.

Toggle among those metrics, then inspect the probabilities. All curves are calculated from the same explicit rule and the same eight examples at each progress value. Smooth probability changes can yield abrupt exact-match changes.

Rescore the same predictions

Switch the metric while keeping all authored probabilities and greedy answers fixed. Inspect the token probabilities to see exactly what each score measures.

0.55

Scroll the chart horizontally if it does not fit.

Metrics calculated from the same authored token probabilities
Exact match: 2/8
Token accuracy: 68.8%
NLL: 0.628 nats
Inspect all probabilities and token decisions
p(correct), with ✓ when greedy decoding is correct and × otherwise.
ExampleToken 1Token 2Token 3Token 4Token 5Token 6
10.54 ✓0.51 ✓0.47 ×0.44 ×0.40 ×0.36 ×
20.57 ✓0.53 ✓0.50 ×0.46 ×0.43 ×0.39 ×
30.59 ✓0.56 ✓0.52 ✓0.49 ×0.45 ×0.42 ×
40.61 ✓0.58 ✓0.54 ✓0.51 ✓0.47 ×0.44 ×
50.64 ✓0.60 ✓0.57 ✓0.54 ✓0.50 ✓0.46 ×
60.67 ✓0.63 ✓0.59 ✓0.56 ✓0.53 ✓0.49 ×
70.69 ✓0.66 ✓0.62 ✓0.59 ✓0.55 ✓0.52 ✓
80.72 ✓0.68 ✓0.65 ✓0.61 ✓0.58 ✓0.54 ✓
For example e=0…7 and token t=0…5: p(correct)=clip(0.1+0.8s+0.025e−0.035t, 0.01, 0.99). Each token has two alternatives; greedy decoding chooses the correct one when p≥0.5. Exact match requires every token correct. NLL averages −log p(correct). No real checkpoint or empirical scaling curve is represented.

This construction demonstrates a possible measurement mechanism. It does not prove that every empirical jump is a metric artifact or that every future capability is predictable. Assess those claims with task-specific evidence, uncertainty estimates, and additional metrics.

What the cited experiments establish

A useful claim names the model family, training setup, task, metric, and comparison. Code generation, for example, was evaluated in code-trained models up to 12B parameters, with nonzero HumanEval pass@1 already reported at 42M. Toolformer studies tool-augmented training rather than a generic parameter-count switch. GPT-4's technical report does not disclose its parameter count.

Named observations, not universal thresholds

Program synthesis

Model and setup
Chen et al. (2021), code-finetuned GPT models from 12M to 12B parameters
Task and metric
HumanEval, executable unit tests, pass@k
What follows
The reported 42M model already has nonzero pass@1. The result depends on the code training and executable-test protocol.

Reasoning with demonstrations

Model and setup
Wei et al. (2022), multiple model families including PaLM 540B
Task and metric
Arithmetic, commonsense, and symbolic tasks; compare direct and chain-of-thought demonstrations
What follows
Benefits vary by task and scale in the studied models. The paper does not establish a universal parameter threshold for reasoning.

Tool-assisted prediction

Model and setup
Schick et al. (2023), Toolformer with GPT-J 6.7B plus smaller-model scaling experiments
Task and metric
API-augmented language-model training and downstream tasks
What follows
Tool behavior depends on tool-augmented training and evaluation. Parameter count alone does not specify whether tool behavior will appear.

Metric-dependent emergence

Model and setup
Schaeffer et al. (2023), fixed predictions rescored across tasks
Task and metric
Discontinuous versus continuous evaluation metrics
What follows
Apparent jumps can change when the metric changes. This is evidence about the evaluated setups, not a proof that all future capabilities are predictable.

GPT-4 reporting boundary

Model and setup
OpenAI (2023), GPT-4 technical report
Task and metric
Reported benchmark evaluations
What follows
The report withholds model size and architecture details. No parameter coordinate or universal theory-of-mind threshold can be inferred from it.

Compare observations only with their training setup, prompting protocol, task, and metric. These studies do not imply a consistent ordering of ability thresholds across model families.

Common pitfalls

Turning observations into universal thresholds

An observed improvement in one family is not a budget rule for another. Parameters, training tokens, data composition, and prompting can change together. Avoid attributing the result to parameter count alone.

Reporting only a binary score

Alongside exact match, examine partial credit or token probabilities when meaningful, sample counts, confidence intervals, and failure cases. A score of zero in a finite sample does not prove a success probability of exactly zero.

Treating an analogy as a theory

A sigmoid can fit a curve without identifying the cause of that curve. A threshold diagram or an information-capacity metaphor does not establish which internal computations exist. Claims about mechanisms need interventions and evidence beyond task scores.

If you found this explanation helpful, consider sharing it with others.

Mastodon