TL;DR
LLaVA (Large Language and Vision Assistant) connects a frozen CLIP ViT-L/14 vision encoder to a Vicuna large language model through a single linear projection layer, then fine-tunes the system on GPT-4-generated multimodal instruction-following data. The architecture is deliberately minimal — no Q-Former, no cross-attention modules, just a learned linear map from visual tokens to the LLM’s input space. Despite this simplicity, LLaVA achieves 85.1% relative performance compared to GPT-4 on a synthetic multimodal benchmark and reaches 92.53% on ScienceQA when the ScienceQA-tuned LLaVA is ensembled with a text-only GPT-4 judge. The paper demonstrated that instruction tuning, not architectural complexity, is the key ingredient for multimodal LLMs.
The Core Idea: Projecting Vision into Language Space
Prior multimodal models like Flamingo and BLIP-2 used heavyweight bridging modules (Perceiver Resamplers, Q-Formers) to translate between vision and language representations. LLaVA takes a radically simpler approach: use a single trainable linear projection adapter W to map CLIP visual features directly into the word embedding space of a language model.
Given an image, the CLIP ViT-L/14 encoder produces a grid of visual feature tokens Z_v \in \mathbb{R}^{N \times d_v}, where N is the number of patch tokens and d_v is the CLIP feature dimension. The projection maps these to language tokens:
This is the row-major form for the stated token shape, corresponding to the paper’s Equation 1. Historical nn.Linear also adds a learned bias; the implementation below makes that convention explicit.
where d_l is the LLM’s hidden dimension. These projected visual tokens H_v are then concatenated with the text token embeddings and fed into the LLM as a unified sequence. The language model processes visual and text tokens with the same self-attention mechanism — no modality-specific architectural changes needed.
This design bets that the LLM’s existing language understanding can be repurposed for multimodal reasoning, provided the visual features are placed in the right embedding space. The experimental results validate this bet.
The contrast with BLIP-2 is instructive. BLIP-2’s Q-Former uses 32 learnable query tokens and a cross-attention transformer with roughly 188M parameters to bridge modalities. LLaVA’s biased linear projection has about 4.20M parameters for the 7B model or 5.25M for the 13B model and no attention mechanism at all. Despite this roughly 36–45x parameter gap in the connector, LLaVA achieves competitive or superior performance on instruction-following tasks, suggesting that the heavy lifting is done by the pretrained components on either side of the bridge.
Architecture: Three Components
Vision encoder. CLIP ViT-L/14, frozen throughout training. It processes the input image at 224x224 resolution and produces a sequence of patch-level feature vectors. It keeps grid tokens rather than the pooled [CLS] token. The paper compares features before and after the last vision transformer layer; its best ScienceQA variant uses the preceding layer, while the pinned historical code defaults to the last layer.
Linear projection. A single trainable matrix W that maps from CLIP’s 1024-dimensional feature space to Vicuna’s input space: 4096 dimensions for 7B or 5120 for 13B. Including the historical layer’s bias, this connector has 4,198,400 parameters for 7B or 5,248,000 for 13B.
Language model. Vicuna-13B (or 7B), a fine-tuned variant of LLaMA. Vicuna is itself instruction-tuned on ShareGPT conversations, so it already possesses strong instruction-following capabilities. LLaVA extends these capabilities to the multimodal domain.
The total architecture is CLIP ViT-L/14 (304M parameters, frozen) + linear projection (~5.25M parameters) + Vicuna-13B (13B parameters, selectively tuned). The simplicity is the point: the authors argue that a minimal connector is sufficient when both the vision encoder and the language model are already well-trained.
GPT-4 Generated Instruction-Following Data
The paper’s second major contribution is a pipeline for generating multimodal instruction-following data using GPT-4 (text-only, at the time). Since GPT-4 could not process images directly when this work was done, the authors encoded visual information as text: they fed GPT-4 the ground-truth captions and bounding box coordinates from COCO images, then prompted it to generate instruction-response pairs as if it were looking at the image.
This process produced 158K instruction-following samples across three categories:
Conversation (58K). Multi-turn dialogues about the image, mimicking natural user interactions. Example: "What is the person in the image doing?" followed by "What equipment are they using?" These train the model to handle sequential, contextual queries about visual content.
Detailed description (23K). Extended, paragraph-length descriptions of image content, spatial relationships, and scene attributes. These teach the model to produce comprehensive visual descriptions on demand.
Complex reasoning (77K). Questions requiring multi-step inference over visual content — counting objects, understanding spatial relationships, interpreting actions, or combining visual evidence with world knowledge. These develop the model’s ability to reason beyond surface-level recognition.
The use of GPT-4 as a data generator was prescient. Manual annotation of instruction-following data at this quality would be prohibitively expensive, but GPT-4 can produce diverse, high-quality responses from structured image metadata at scale. This approach has since become standard practice in the field.
A subtle but important design choice: the authors seeded GPT-4 with a small number of hand-written examples for each data type, establishing the desired format and depth of response. This few-shot prompting strategy controlled the quality distribution of the generated data without requiring extensive manual curation of the full 158K dataset.
Two-Stage Training
LLaVA’s training proceeds in two stages that progressively unlock the model’s capabilities.
Stage 1: Feature alignment (pre-training). Only the linear projection W is trained. Both the vision encoder and the LLM remain frozen. The training data consists of 595K image-caption pairs filtered from CC3M (Conceptual Captions). Each caption is reformulated as a simple instruction: "Describe this image briefly." The objective is the standard autoregressive language modeling loss, applied only to the caption tokens. This stage teaches the projection to provide visual conditioning from which the frozen LLM can predict caption tokens. Training takes roughly 4 hours on 8 A100 GPUs.
Stage 2: End-to-end fine-tuning. The projection layer and the LLM are both updated; the vision encoder stays frozen. The training data is the 158K GPT-4-generated instruction-following samples. The model learns to follow diverse visual instructions — answering questions, describing scenes in detail, and performing multi-step reasoning. The autoregressive loss is applied only to the assistant’s response tokens, not to the user’s instructions or the image tokens. This stage takes roughly 10 hours on 8 A100 GPUs.
The two-stage approach is computationally efficient. Stage 1 aligns modalities with a lightweight, frozen-LLM setup. Stage 2 unlocks the full model for instruction following. The reported 13B runs use about 4 hours for alignment and 10 hours for instruction tuning on eight A100s. That is approximately 14 elapsed hours and 8 × (4 + 10) = 112 A100 GPU-hours, not 14 GPU-hours (Appendix C). These are the authors’ reported runs, not timings reproduced here.
The decision to freeze the LLM in stage 1 is important. Updating 13B parameters on noisy caption data risks catastrophic forgetting of the language model’s instruction-following capabilities. By training only the roughly 5.25M-parameter 13B connector first, the model learns a stable cross-modal mapping without disturbing the LLM’s weights. Stage 2 then fine-tunes the full LLM on high-quality instruction data, where the gradient signal is clean enough to improve multimodal capabilities without degrading language performance.
Implementation
The stage boundary is a parameter-training rule, not a break in the computation graph. The reference below follows features into the text sequence, masks assistant targets, and verifies which parameters receive updates before returning to the paper’s measured results.
Authored reduced original-LLaVA reference: seeded offline CPU mechanics with synthetic selected vision inputs, three patch tokens, a single linear connector and a random width-eight causal language model. No pretrained CLIP/Vicuna weights, semantic visual answers, native GPU performance or benchmark reproduction. The code conventions are pinned to the pre-LLaVA-1.5 July 23, 2023 revision 5e89b7ed725f. Equations below are authored explanations of the cited operations.
1. Keep patch tokens from frozen vision features
Original paper
The paper considers grid features before and after the last layer.
Math and tensor shapes
The pinned July 2023 source selects a configurable CLIP hidden state, then drops its leading CLS slot for patch-only features. Its train.py default selects the last layer (-1); the ScienceQA best variant uses the preceding layer. This reference models only that selected-state interface with a frozen random linear module, not a CLIP image encoder. Only this frozen extraction uses no_grad. Three patches replace the original 256 at 224 × 224.
Selected-state stand-in [2, 4, 3] → patch features [2, 3, 6]
Python implementation
def vision_features(vision, selected_inputs):
"""Extract patch features from a frozen selected-state stand-in.
Args:
vision: Frozen linear module mapping three channels to d_v.
selected_inputs: Float [B, N+1, 3], including a leading CLS slot.
Returns:
Detached patch features [B, N, d_v], with no pooled/CLS token.
"""
if selected_inputs.ndim != 3 or selected_inputs.shape[1] < 2:
raise ValueError("Expected [B, N+1, 3] with at least one patch")
if selected_inputs.shape[-1] != vision.in_features:
raise ValueError("Vision input channels do not match")
if not torch.isfinite(selected_inputs).all():
raise ValueError("Vision inputs must be finite")
if any(p.requires_grad for p in vision.parameters()):
raise ValueError("Vision must remain frozen")
with torch.no_grad():
selected_hidden_state = vision(selected_inputs)
return selected_hidden_state[:, 1:, :]2. Project row-major features with one linear layer
Original paper
The paper abstracts the connector as a projection matrix W.
Math and tensor shapes
Equation 1 writes W · Z_v in the paper’s feature convention. In row-major PyTorch tensors, multiply features by W.T. Historical nn.Linear includes a bias, so the displayed affine equation explains the implementation rather than quoting the paper equation literally. This is one linear layer, not LLaVA-1.5’s two-layer MLP. At original width 1024, the biased connector has 4,198,400 parameters for Vicuna-7B (4096) or 5,248,000 for Vicuna-13B (5120).
W [8, 6], b [8]; [2, 3, 6] → [2, 3, 8]
Python implementation
def linear_projector(features, projector):
"""Map row-major patches through one biased linear layer, not an MLP."""
if features.ndim != 3 or features.shape[1] < 1:
raise ValueError("Expected nonempty [B, N, d_v] patch features")
if features.shape[-1] != projector.in_features:
raise ValueError("Feature width must match projector")
if not torch.isfinite(features).all():
raise ValueError("Patch features must be finite")
# W: [d_l, d_v]; b: [d_l]; output: [B, N, d_l].
return features @ projector.weight.T + projector.bias3. Replace the image placeholder and rebuild sequence axes
Original paper
The first instruction may put the image before or after its question.
Math and tensor shapes
One IMAGE sentinel is replaced by N projected embeddings, never passed to the vocabulary lookup. This follows the pre-1.5 July integrated source; earlier releases instead expanded repeated image-patch placeholders in the tokenizer. The authored assembly strips suffix padding, inserts at the actual placeholder, then right-pads embeddings with zero and labels with -100. It explicitly rebuilds attention and positions; padded positions are zero. Tests vary image placement, patch counts, text lengths and text-only samples. This single-image reference rejects duplicate or missing placeholders for supplied images.
Two smoke samples: valid text lengths 6 / 5 → lengths 8 / 7
Python implementation
def splice_image(ids, text_embeddings, visual_sequences, labels, attention):
"""Replace one IMAGE sentinel per visual sample and right-pad the batch.
visual_sequences is a B-long list of [N_b, d_l] tensors or None for
text-only samples. Labels are already masked by assistant_labels.
Returns embeddings, attention, positions, labels with identical B/L axes.
"""
if ids.ndim != 2 or ids.dtype != torch.long:
raise ValueError("Token IDs must be long [B, T]")
if text_embeddings.shape[:2] != ids.shape or text_embeddings.ndim != 3:
raise ValueError("Text embeddings must be [B, T, d_l]")
if labels.shape != ids.shape or labels.dtype != torch.long:
raise ValueError("Labels must be long [B, T]")
if attention.shape != ids.shape or attention.dtype != torch.bool:
raise ValueError("Attention must be boolean [B, T]")
if len(visual_sequences) != ids.shape[0] or ids.shape[0] < 1:
raise ValueError("Supply one optional visual tensor per sample")
if (attention[:, 1:] & ~attention[:, :-1]).any():
raise ValueError(
"Only contiguous valid prefix / right padding is supported"
)
if ((ids == PAD) & attention).any() or (ids[~attention] != PAD).any():
raise ValueError("PAD IDs must coincide with masked suffix positions")
sequences, targets = [], []
for b in range(ids.shape[0]):
count = int(attention[b].sum())
if count < 1:
raise ValueError("Every sample needs a valid context token")
where = (ids[b, :count] == IMAGE).nonzero().flatten()
visual = visual_sequences[b]
if len(where) != (0 if visual is None else 1):
raise ValueError(
"Exactly one IMAGE is required for each supplied image"
)
x, y = text_embeddings[b, :count], labels[b, :count]
if visual is not None:
if (
visual.ndim != 2
or visual.shape[0] < 1
or visual.shape[1] != x.shape[1]
):
raise ValueError(
"Visual embeddings must be nonempty [N_b, d_l]"
)
if not torch.isfinite(visual).all():
raise ValueError("Visual embeddings must be finite")
k = int(where[0])
x = torch.cat((x[:k], visual, x[k + 1 :]), dim=0)
image_labels = torch.full((visual.shape[0],), IGNORE, dtype=y.dtype)
y = torch.cat((y[:k], image_labels, y[k + 1 :]))
sequences.append(x)
targets.append(y)
length = max(x.shape[0] for x in sequences)
embeddings = torch.stack(
[F.pad(x, (0, 0, 0, length - len(x))) for x in sequences]
)
new_labels = torch.stack(
[F.pad(y, (0, length - len(y)), value=IGNORE) for y in targets]
)
lengths = torch.tensor([len(x) for x in sequences])
new_attention = torch.arange(length)[None, :] < lengths[:, None]
positions = (new_attention.long().cumsum(1) - 1).clamp_min(0)
positions = positions.masked_fill(~new_attention, 0)
return embeddings, new_attention, positions, new_labels4. Supervise assistant responses only
Original paper
Instructions and visual context condition the answer without target loss.
Math and tensor shapes
Explicit boolean role masks replace tokenizer-specific prompt parsing. Assistant response tokens and their stop token are targets. System/user tokens, image embeddings and padding receive the ignore index. Ignoring a target means zero direct loss at its prediction logit, not zero gradient on its context embedding: image and user context can influence later assistant predictions through causal attention. The source shows image-label insertion; the paper specifies answer-only supervision across conversation turns.
Role, attention and labels share [B, T]; insertion expands them to [B, L]
Python implementation
def assistant_labels(ids, assistant, attention, vocabulary):
"""Keep assistant response/stop-token targets, ignore all context/padding.
Role masks are supplied explicitly rather than inferred from a tokenizer.
Ignored targets have zero direct loss, but context embeddings can get
gradients through causal attention to supervised assistant predictions.
"""
if ids.ndim != 2 or ids.dtype != torch.long:
raise ValueError("Token IDs must be long [B, T]")
if assistant.shape != ids.shape or assistant.dtype != torch.bool:
raise ValueError("Assistant role mask must be boolean [B, T]")
if attention.shape != ids.shape or attention.dtype != torch.bool:
raise ValueError("Attention mask must be boolean [B, T]")
if (
vocabulary < 2
or ((ids != IMAGE) & ((ids < 0) | (ids >= vocabulary))).any()
):
raise ValueError("IDs must be IMAGE or valid vocabulary tokens")
valid = assistant & attention & (ids != IMAGE) & (ids != PAD)
return ids.masked_fill(~valid, IGNORE)5. Shift once and average valid next-token losses
Original paper
The objective predicts answer tokens from image and preceding conversation.
Math and tensor shapes
Pair logits[:, :-1] with labels[:, 1:] exactly once, as the historical language-model code does. Mean cross entropy weights each valid target equally across the batch, not each sequence equally. The paper states the answer probability factorization; the formula here explains its code loss. An independent log-sum-exp and analytic gradient oracle checks the shift and reduction. Empty valid-target sets return differentiable zero as an explicit reference extension; ordinary mean cross entropy would return NaN.
Logits [2, 8, 24]; five valid shifted assistant targets
Python implementation
def causal_loss(logits, labels):
"""Shift exactly once, averaging NLL over valid assistant targets."""
if (
logits.ndim != 3
or labels.shape != logits.shape[:2]
or labels.dtype != torch.long
):
raise ValueError("Expected logits [B, L, V] and long labels [B, L]")
if not torch.isfinite(logits).all():
raise ValueError("Logits must be finite")
if (
(labels != IGNORE) & ((labels < 0) | (labels >= logits.shape[-1]))
).any():
raise ValueError("Target is outside vocabulary")
shifted_logits = logits[:, :-1, :]
shifted_labels = labels[:, 1:]
valid = shifted_labels != IGNORE
if not valid.any():
# Defined finite empty-target extension, unlike mean CE's NaN.
return logits.sum() * 0.0
return F.cross_entropy(shifted_logits[valid], shifted_labels[valid])6. Freeze parameters without cutting the projector gradient
Original paper
Vision stays frozen in both stages; the LM is unlocked in stage 2.
Math and tensor shapes
Set requires_grad=False on LM parameters in stage 1 while keeping the LM forward inside autograd. The loss must still differentiate through that frozen computation into projected visual embeddings. Stage 2 enables both projector and LM parameters; vision remains frozen. Clear stale gradients when switching stages and build the optimizer from the newly trainable parameters. The last gradient expression describes the observed seeded run, not a guarantee for every input. A finite-difference oracle verifies a projector gradient through the frozen LM. Tiny random causal attention, learned absolute positions and an FFN replace Vicuna; SGD demonstrates stage control, not the original Adam schedule.
One SGD step per stage; frozen vision parameters and gradients checked
Python implementation
def configure_stage(model, stage):
"""Stage 1 trains only projector; stage 2 trains projector plus LM."""
if stage not in (1, 2):
raise ValueError("Stage must be 1 or 2")
model.zero_grad(set_to_none=True)
model.vision.requires_grad_(False)
model.projector.requires_grad_(True)
model.lm.requires_grad_(stage == 2)
def train_step(model, batch, stage, learning_rate=0.05):
"""Run one illustrative SGD update and expose input/parameter gradients."""
if not math.isfinite(learning_rate) or learning_rate <= 0:
raise ValueError("Learning rate must be finite and positive")
configure_stage(model, stage)
optimizer = torch.optim.SGD(
(p for p in model.parameters() if p.requires_grad), lr=learning_rate
)
embeddings, attention, positions, labels = model.assemble(*batch)
embeddings.retain_grad()
# Do not put the frozen LM in no_grad: stage 1 needs dL/d(image embeddings).
logits = model.lm(embeddings, attention, positions)
loss = causal_loss(logits, labels)
loss.backward()
observations = {
"loss": float(loss.detach()),
"projector_gradient_norm": float(
model.projector.weight.grad.norm().detach()
),
"input_gradient_norm": float(embeddings.grad.norm().detach()),
"vision_gradients_absent": all(
p.grad is None for p in model.vision.parameters()
),
"lm_gradients_absent": all(
p.grad is None for p in model.lm.parameters()
),
}
optimizer.step()
return observationsRun the source behind these highlights. Actual smoke JSON records the source SHA-256, historical revision, seed, reduced shapes, sequence lengths, attention-derived positions, masked labels, finite losses, direct ignored-target gradients, input and projector gradients, and measured parameter changes for both stages. Synthetic token IDs have no visual or linguistic meaning.
llava.pyrequirements-llava.txtOriginal training protocol
python -m pip install -r requirements-llava.txt python llava.py --smoke --seed 7
Key Results
ScienceQA. The ScienceQA-tuned LLaVA-13B predicts reasons before answers and achieves 90.92% accuracy. The reported 92.53% belongs to an ensemble in which text-only GPT-4 judges disagreements between its answer and LLaVA’s answer. Text-only GPT-4 alone scores 82.69%; the previous MM-CoT Large baseline scores 91.68% (§5.2, Table 7). The 92.53% result is therefore not an effect of CoT prompting alone on the instruction-tuned chatbot.
LLaVA-Bench. The authors introduce a GPT-4-based evaluation protocol: GPT-4 scores the quality of model responses against reference answers on a scale of 1-10. On this benchmark, LLaVA-13B achieves 85.1% relative to the text-only GPT-4 reference that receives ground-truth image descriptions. This is a ratio of GPT-4-assigned response-quality scores, not 85.1% absolute human accuracy (§5.1, Table 4). This evaluation approach, while imperfect, was one of the first systematic attempts to measure open-ended multimodal instruction following.
Qualitative capabilities. The paper demonstrates LLaVA handling tasks that require genuine visual understanding: reading text in images, interpreting memes, explaining visual humor, describing spatial relationships, and answering knowledge-grounded questions about image content. These qualitative examples were influential in demonstrating what a simple architecture could achieve.
Ablation insights. The paper includes ablations showing that both stages of training are necessary. Skipping stage 1 (training everything end-to-end from scratch) produces worse results, confirming that pre-aligning the projection layer provides a better initialization for full fine-tuning. Additionally, the diversity of instruction types matters: models trained on only conversation data underperform on reasoning tasks, and vice versa. The mixture of all three data types (conversation, description, reasoning) produces the best overall performance.
Critical Analysis
Strengths.
- Architectural simplicity. A single linear layer connecting a frozen vision encoder to an LLM is about as minimal as a multimodal architecture can get. This makes the approach easy to implement, reproduce, and extend. The simplicity also means fewer hyperparameters and failure modes compared to Q-Former or Perceiver-based designs.
- Training efficiency. The original 13B alignment and instruction-tuning runs total approximately 112 A100 GPU-hours, using the authors’ reported eight-GPU 4-hour and 10-hour timings. Reusing pretrained backbones reduces the work needed for multimodal adaptation; this total excludes upstream CLIP and Vicuna training.
- Data generation pipeline. Using GPT-4 to convert existing image annotations into instruction-following data is a scalable, reusable methodology. The 158K samples produced this way proved sufficient for strong multimodal instruction following.
- Open source. The model weights, training data, and code were all released publicly, enabling rapid community iteration. This openness was a deliberate strategic choice that shaped the field’s trajectory.
Limitations.
- Single image input. LLaVA processes one image per conversation. It cannot compare two images, process video frames, or handle multi-image reasoning — a significant constraint for real-world applications.
- Hallucination. Like all multimodal LLMs, LLaVA generates plausible-sounding descriptions of content not present in the image. The linear projection provides no mechanism for the model to verify its claims against the visual evidence.
- Linear projection bottleneck. A single linear layer is a limited function class for cross-modal alignment. It can translate and rotate the embedding space but cannot learn nonlinear feature interactions. LLaVA-1.5 later confirmed this by showing that a two-layer MLP projection produces measurable gains.
- Resolution constraint. The CLIP ViT-L/14 encoder operates at 224x224, which limits fine-grained visual understanding. Small text, distant objects, and detailed textures are poorly represented at this resolution.
- Evaluation limitations. The GPT-4-based evaluation benchmark, while creative, introduces its own biases. GPT-4’s preferences may not align with human judgments, and the relative scoring makes cross-paper comparison difficult.
LLaVA-1.5: Addressing the Bottlenecks
LLaVA-1.5 (Liu et al., 2023) addressed several of the original paper’s limitations with targeted improvements that maintained the core architectural philosophy:
MLP projection. The single linear layer was replaced with a two-layer MLP (linear → GELU → linear), allowing nonlinear feature interactions during cross-modal alignment. This change alone improved performance across multiple benchmarks.
Higher resolution. Upgrading from CLIP ViT-L/14 at 224x224 to 336x336 resolution improved fine-grained visual understanding, particularly for tasks involving text recognition and small object detection.
Academic task data. Adding VQA, OCR, and region-level grounding datasets to the instruction tuning mix improved performance on structured benchmarks without degrading conversational ability.
Scaling the data. The instruction-following dataset was expanded from 158K to 665K samples, with broader coverage of visual tasks.
These changes pushed LLaVA-1.5-13B to competitive performance with models trained on orders of magnitude more data, reinforcing the original paper’s thesis that architecture simplicity plus good instruction data is a strong formula. Notably, LLaVA-1.5 outperformed InstructBLIP (which uses a 188M-parameter Q-Former) on 11 out of 12 benchmarks, despite using a connector with two orders of magnitude fewer parameters.
Impact and Legacy
LLaVA catalyzed the open-source multimodal LLM movement. Before LLaVA, multimodal instruction-following models were either proprietary (GPT-4V) or required specialized architectures and large-scale pre-training (Flamingo, BLIP-2). LLaVA showed that a graduate-student-budget project could produce a multimodal assistant with meaningful capabilities.
The downstream impact was rapid and broad. LLaVA-Med applied the framework to medical imaging. LLaVA-Interactive extended it to image editing and generation. Video-LLaVA adapted the architecture for video understanding. The linear-projection-plus-instruction-tuning recipe became the default starting point for open multimodal LLMs, influencing models like InternVL. Qwen-VL, by contrast, bridges vision and language with a cross-attention adapter rather than a projection.
The GPT-4-based data generation pipeline was equally influential. The idea of using a strong language model to synthesize training data for a weaker multimodal model has been adopted across the field, from SVIT to ShareGPT4V to ALLaVA. This "self-improvement via distillation" pattern now underpins much of multimodal data creation.
The architectural pattern also proved durable. The "vision encoder + projection + LLM" template that LLaVA popularized has been widely adopted by open multimodal models such as LLaVA-OneVision (SigLIP, a 2-layer MLP and Qwen-2), InternVL 1.5 (InternViT-6B, an MLP projector and InternLM2-20B) and Molmo, each of which cites LLaVA for the design. Unlike LLaVA, these later models also fine-tune the vision encoder instead of keeping it frozen, and variations exist in the projection module (linear, MLP, cross-attention, perceiver), the vision encoder (CLIP, SigLIP, InternViT), and the LLM backbone (LLaMA, Mistral, Qwen).
Perhaps most importantly, LLaVA established that the barrier to multimodal LLMs was not architectural innovation but data and alignment. The vision encoder and language model were both off-the-shelf; the contributions were the data pipeline and the training recipe. This reframing shifted research attention from architecture design toward data quality, instruction diversity, and training methodology — a shift that has persisted in the multimodal LLM community.
Related Reading
- Qwen2-VL — a modern open VLM in the LLaVA lineage, adding dynamic-resolution visual tokens and M-RoPE for any-resolution images and video
- Flamingo — the few-shot VLM whose frozen-backbone bridging LLaVA builds on, swapping gated cross-attention for a simple projection plus instruction tuning
- CLIP: Visual Models via Language Supervision — the vision encoder that provides LLaVA’s visual representations
- Vision Transformer — the ViT architecture underlying CLIP’s image encoder
- BLIP-2 — the Q-Former approach to vision-language bridging that LLaVA’s linear projection simplifies
- Attention Is All You Need — the transformer architecture underlying both the vision encoder and the language model
- DINO — self-supervised visual features that later multimodal models combined with CLIP features
- Vision-Language Alignment — compares LLaVA’s projector with CLIP, BLIP-2 and Flamingo, and what each trains or freezes in each stage