Multimodal Scaling Laws

Explore a fixed compute budget with Chinchilla’s text-only loss fit, then see why multimodal scaling needs its own data, objectives, and evaluation metrics.

Multimodal Scaling Laws

At a fixed training budget, a larger model leaves less compute for training data. A scaling law estimates the resulting trade-off from measured runs. To use it, first identify the loss, data distribution, architecture, and range of sizes it describes.

For a vision-language model, “better” might mean lower contrastive loss, lower generative token loss, or better zero-shot classification. Those quantities need separate fits. Start with one text-only example, then consider what changes when images enter the training objective.

Spend one compute budget

Choose a FLOPs budget and move model size. The token count adjusts to keep approximate training compute fixed. Compare the chosen loss with the minimum of this particular fit; it is a prediction, not a measured training run.

What the explorer calculates

Hoffmann et al. (2022) fit text language-model loss to an irreducible term plus parameter and data terms:

L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}

Here N is the absolute parameter count and D counts training tokens. In Approach 3, Appendix D.2, Equation 10, the published rounded constants are:

E=1.69,\quad A=406.4,\quad B=410.7,\quad \alpha=0.34,\quad \beta=0.28

Approximate dense-transformer training compute connects the two counts:

C\approx 6ND,\qquad D=\frac{C}{6N}

Increasing N reduces the parameter term. At fixed C, it also reduces D, increasing the data term. That is why the curve has a minimum. These coefficients use absolute counts, so inserting “1” for one billion parameters gives the wrong result.

The study spans 70M–16B parameters and 5B–500B training tokens. Predictions outside those bounds are extrapolations. Inside the bounds, a prediction still needs validation: the rectangle does not imply every allocation was tested.

Find the minimum of the fit

Substitute D=C/(6N) and differentiate with respect to N. At the minimum, the marginal changes balance:

\alpha\frac{A}{N_*^{\alpha}}=\beta\frac{B}{D_*^{\beta}}

The two loss terms need not be equal. Solving gives:

N_*(C)=\left[\frac{\alpha A}{\beta B}\left(\frac{C}{6}\right)^{\beta}\right]^{\frac{1}{\alpha+\beta}},\qquad D_*(C)=\frac{C}{6N_*(C)}

With the rounded coefficients used here, N_*\propto C^{0.452} and D_*\propto C^{0.548}. The paper reports approximately 0.46 and 0.54 for its parametric fit, while its other two approaches give exponents near 0.5 for both quantities. Those are related empirical estimates, not identical rules.

“About 20 tokens per parameter” summarizes allocations from another Chinchilla approach and its 70B/1.4T training run. It is not a universal constant, and forcing that ratio onto this rounded fit would change its calculated minimum. Neither allocation rule includes the cost of serving the resulting model.

What changes for multimodal models?

The text-only constants above do not predict CLIP, ALIGN, Flamingo, or LLaVA performance. A multimodal fit needs its own objective and measurements:

Setting	Quantity to model	Evidence and scope
Contrastive image–text training	A named downstream metric, such as zero-shot classification error	Cherti et al. find different scaling behavior for OpenCLIP/LAION and OpenAI CLIP/WIT, including task-dependent results. Contrastive training loss and zero-shot error are different quantities.
Generative mixed-modal tokens	Per-modality token loss and interactions between modalities	Aghajanyan et al. fit modality-specific terms and competition/synergy terms. Their speech–text experiment does not establish a universal vision:language ratio.
Frozen encoders with an adapted language model	The chosen task metric under that training recipe	A vision-language adapter changes which parameters train and how inputs are processed. Total stored parameters alone do not specify training FLOPs.

Image–text pairs, image patches, and language tokens are different units. Record how each contributes to compute and loss before fitting a curve. Changes in resolution, text length, frozen components, or data mixture can invalidate an estimate based only on total parameter count.

The modality gap is a geometric property of representations. Its size is not determined by a vision:language parameter ratio, and a scaling fit alone does not establish whether a model will hallucinate. Likewise, lower token loss does not automatically imply the same improvement on every downstream task.

Use a fit for the decision it measures

A useful allocation study holds the data distribution and training recipe consistent, varies parameters and training data, and checks predictions on additional runs. Fit the metric your decision depends on, with uncertainty and a stated range.

For deployment, training compute is only one part of the objective. Memory limits, inference latency, serving volume, and access to suitable data can favor a smaller model trained longer. FLOPs also do not determine elapsed time or a dollar price without a hardware and utilization model. The explorer answers the narrower question: which allocation minimizes this fitted text loss under approximate training compute?

References

Kaplan et al. "Scaling Laws for Neural Language Models"
Hoffmann et al. "Training Compute-Optimal Large Language Models" (Chinchilla)
Cherti et al. "Reproducible Scaling Laws for Contrastive Language-Image Learning"
Aghajanyan et al. "Scaling Laws for Generative Mixed-Modal Language Models"
Radford et al. "Learning Transferable Visual Models From Natural Language Supervision" (CLIP)
Jia et al. "Scaling Up Visual and Vision-Language Representation Learning" (ALIGN)
Alayrac et al. "Flamingo: a Visual Language Model for Few-Shot Learning"
Liu et al. "Visual Instruction Tuning" (LLaVA)

Transformers & LLMs

The Vision-Language Alignment Problem

Trace one image-caption pair through cosine scores, contrastive positives and negatives, a trainable projection, and retrieval evaluation.

Transformers & LLMs

The Modality Gap in Multimodal AI

Measure the centroid gap between normalized image and text embeddings, then check why a smaller gap alone does not guarantee correct retrieval.

Transformers & LLMs

Vision-Language Adapters: Efficient Fine-tuning

Trace frozen weights and trainable LoRA matrices, calculate adapter parameter counts, and compare bottleneck adapters, prefix tuning and BitFit for vision-language models.

Transformers & LLMs

Vision-Language Alignment: CLIP, BLIP-2, LLaVA, Flamingo

CLIP, BLIP-2, LLaVA and Flamingo align images with text by different routes. Compare their objectives, bridges, data, and which parts train or stay frozen.

Transformers & LLMs

Flash Attention vs MHA vs GQA vs MQA: Comparing Attention Mechanisms

Compare head sharing with explicit projection dimensions and KV-cache bytes. Separate exact memory ratios from measured quality and latency, and see why FlashAttention is an independent choice.

Transformers & LLMs

Cross-Attention: Bridging Different Modalities

Trace a target query through source keys, attention weights, and a weighted value mix. Understand cross-attention origins, shapes, masks, and applications.