TL;DR
- Qwen2-VL introduces naive dynamic resolution: images keep a variable grid near their native aspect ratio, with dimensions rounded and resized within pixel bounds, producing a variable number of visual tokens rather than forcing every image into a fixed 224×224 or 448×448 grid — more image content means more tokens, less means fewer.
- M-RoPE (Multimodal Rotary Position Embedding) decomposes each token's position into three independent components — temporal, height, and width — so one rotary scheme handles text, images, and video frames without separate encoders or position tables.
- A single Qwen2 language model backbone processes all modalities: still images, sequences of images in one context, and long videos, all interleaved with text via the same token stream.
- Released in three sizes (2B, 7B, 72B), Qwen2-VL-72B matched or exceeded GPT-4o and Claude 3.5 Sonnet on several multimodal benchmarks at the time of publication, setting a strong open-VLM baseline.
Naive dynamic resolution
Most vision-language models resize every input image to a fixed resolution before encoding. A thumbnail and a 4K photograph both become the same 224×224 tensor, discarding detail in the larger image and wasting nothing useful in the smaller one. Qwen2-VL calls its alternative approach naive dynamic resolution: the processor rounds each dimension to a multiple of 28 and resizes within its configured pixel bounds before the ViT encodes the resulting grid by splitting it into 14×14 pixel patches, then merging every 2×2 group of adjacent patches into a single visual token. A 448×448 image produces 32×32=1,024 patch features, merged into 16×16=256 tokens. A 1,568×1,568 processed grid produces 3,136 merged tokens if its budget permits; the release-era processor defaults instead resize that input to 980×980, yielding 1,225 tokens. The token count is proportional to the image area, scaled down by a factor of 4 from the raw patch count.
This matters because the resolution budget can preserve more detail than a single fixed grid. A dense document, a high-resolution photograph, or a wide-angle video frame can contribute enough tokens to convey its content. At the same time, a small thumbnail contributes only a handful of tokens, keeping inference efficient. The LLM receives however many visual tokens the image warrants, not a one-size-fits-all block.
M-RoPE: multimodal position
Standard rotary position embeddings assign each token a single 1D position index. That works for text, but images have two spatial dimensions and video has a third temporal dimension. Qwen2-VL introduces M-RoPE, which decomposes every token's position into three independent components: temporal t, height h, and width w.
For text tokens, all three components share the same incrementing integer — effectively recovering 1D RoPE. For image tokens, t is fixed at a constant (e.g., 0) and (h, w) vary across the patch grid, encoding 2D spatial position. For video, t increments with each depth-two tubelet while (h, w) cycle through the patch grid within each tubelet. The three components are encoded in separate subsets of the attention head's rotary dimensions to select axis-specific cosine and sine rotations — the attention mechanism therefore sees how far apart two tokens are along each axis independently. Text, images, and video share one unified position scheme with no modality-specific position tables, and the same rotary frequency convention applies across all three.
Implementation
The position scheme becomes executable only when pixel order, spatial merging, and sequence offsets agree. The following reduced reference traces that contract before returning to how a single language model consumes the resulting stream.
Authored offline CPU packing and rotary mechanics with synthetic pixels, random reduced modules and seven executed operations. No pretrained weights, ViT attention, LLM inference, native GPU performance or benchmark reproduction. The release-era Transformers v4.45.2 library integration and matching image processor are pinned to 53fad641cfdb. The original Qwen2-VL repository now redirects to a successor; this reference uses the pinned Qwen2-VL code, without successor recipes. Equations below are authored explanations of the cited operations.
1. Bound the resolution and align the merge grid
Original paper
Variable visual length still needs a bounded, divisible grid.
Math and tensor shapes
The release uses patch p=14, merge m=2, factor f=28 and processor defaults of 3,136 to 1,003,520 pixels. H_0 and W_0 are the initial factor-rounded dimensions; H_r and W_r denote the final bounded output after the following budget adjustment. If the initial rounded area exceeds its budget, rescale by the square root of the area ratio and floor to the factor. Below the minimum, rescale upward and ceil. Reject dimensions below the factor or aspect ratios above 200. This reference additionally rejects infeasible custom budgets after rounding. Reduced smoke inputs use p=2 and bounds 16 to 256. Native dimensions are not always retained: 256 × 256 becomes 252 × 252; 1,568 × 1,568 becomes 980 × 980 under these processor defaults.
Reduced image [1, 3, 9, 17] → spatial size [8, 16]
Python implementation
def smart_resize(height, width, patch_size=14, merge_size=2,
min_pixels=56 * 56, max_pixels=14 * 14 * 4 * 1280):
"""Return the pinned processor's bounded, factor-aligned spatial size."""
values = (height, width, patch_size, merge_size, min_pixels, max_pixels)
if any(type(value) is not int or value <= 0 for value in values):
raise ValueError("Sizes and bounds must be positive integers.")
factor = patch_size * merge_size
if min_pixels > max_pixels or min_pixels < factor ** 2:
raise ValueError("Bounds must be ordered and allow a factor cell.")
if min(height, width) < factor:
raise ValueError("Each input dimension must be at least the factor.")
if max(height, width) / min(height, width) > 200:
raise ValueError("Absolute aspect ratio must not exceed 200.")
h_bar = round(height / factor) * factor
w_bar = round(width / factor) * factor
if h_bar * w_bar > max_pixels:
beta = math.sqrt(height * width / max_pixels)
h_bar = math.floor(height / beta / factor) * factor
w_bar = math.floor(width / beta / factor) * factor
elif h_bar * w_bar < min_pixels:
beta = math.sqrt(min_pixels / (height * width))
h_bar = math.ceil(height * beta / factor) * factor
w_bar = math.ceil(width * beta / factor) * factor
# Extreme custom budgets can make the upstream rounding infeasible.
if min(h_bar, w_bar) == 0 or not min_pixels <= h_bar * w_bar <= max_pixels:
raise ValueError("Rounded dimensions cannot satisfy these bounds.")
return h_bar, w_bar2. Pair frames before extracting depth-two tubelets
Original paper
The paper treats each still image as two identical frames.
Math and tensor shapes
Depth-two tubelets replace separate per-frame patches. The v4.45.2 processor duplicates a single image but requires paired video frames: its reshape rejects an odd clip with more than one frame. Last-frame duplication for odd video is an explicit authored input-preparation extension here, not v4.45.2 padding behavior. Tests contrast that rejection with the exact even-frame ordering after preparation. Synthetic floating pixels bypass RGB normalization, and PyTorch bicubic interpolation is a resize stand-in for PIL rather than a pixel-identical processor reproduction.
Still [1, 3, 8, 16] → [2, 3, 8, 16]; clip F=3 → F=4
Python implementation
def prepare_tubelets(frames, size):
"""Resize float [F,C,H,W] pixels and pair frames, duplicating if odd."""
if (frames.ndim != 4 or min(frames.shape) <= 0
or not frames.is_floating_point()):
raise ValueError("Expected nonempty floating [F,C,H,W] pixels.")
if not torch.isfinite(frames).all():
raise ValueError("Pixels must be finite.")
if len(size) != 2 or any(type(x) is not int or x <= 0 for x in size):
raise ValueError("Resize dimensions must be positive integers.")
resized = F.interpolate(
frames, size=size, mode="bicubic", align_corners=False
)
if resized.shape[0] % 2:
# F=1 is the paper's still-image duplication. For odd videos this
# is explicit authored preparation, not v4.45.2 processor behavior.
resized = torch.cat((resized, resized[-1:]), dim=0)
return resized3. Flatten patches in spatial merge order
Original paper
Adjacent groups must remain adjacent when the sequence is flattened.
Math and tensor shapes
Match the processor transpose (0,3,6,4,7,2,1,5,8): time, outer height, outer width, inner height, inner width, channel, two-frame depth, pixel height and pixel width. Every four consecutive rows therefore describe one 2 × 2 neighborhood, ordered top-left, top-right, bottom-left, bottom-right. A labeled 4 × 4 patch oracle produces [0,1,4,5], then [2,3,6,7], not naive row-major groups. A separate nested-loop pixel oracle checks channel, temporal and within-patch order. The pinned bias-free Conv3d reads each row as [C,2,p,p]. We omit ViT attention between embedding and merging, so these random features carry no pretrained semantics.
Grid [1, 4, 8] → 32 patch rows × 24 pixel values
Python implementation
def flatten_patches(frames, patch_size=14, merge_size=2):
"""Match processor transpose: t, group-h/w, inner-h/w, C, T, pixels."""
if frames.ndim != 4 or min(frames.shape) <= 0:
raise ValueError("Expected nonempty [F,C,H,W].")
if any(type(x) is not int or x <= 0 for x in (patch_size, merge_size)):
raise ValueError("Patch and merge sizes must be positive integers.")
count, channels, height, width = frames.shape
factor = patch_size * merge_size
if count % 2 or height % factor or width % factor:
raise ValueError("Frames must pair; spatial sizes must divide factor.")
grid_t = count // 2
grid_h, grid_w = height // patch_size, width // patch_size
patches = frames.reshape(
grid_t, 2, channels, grid_h // merge_size, merge_size, patch_size,
grid_w // merge_size, merge_size, patch_size
).permute(0, 3, 6, 4, 7, 2, 1, 5, 8)
patches = patches.reshape(
grid_t * grid_h * grid_w, channels * 2 * patch_size ** 2
)
return patches, (grid_t, grid_h, grid_w)
class PatchEmbed(nn.Module):
"""Pinned depth-two Conv3d over each processor-ordered patch row."""
def __init__(self, channels=3, patch_size=2, width=6):
super().__init__()
self.channels, self.patch_size = channels, patch_size
kernel = (2, patch_size, patch_size)
self.proj = nn.Conv3d(
channels, width, kernel, stride=kernel, bias=False
)
def forward(self, patches):
pixels = patches.reshape(
-1, self.channels, 2, self.patch_size, self.patch_size
)
return self.proj(pixels).flatten(1)4. Concatenate neighbors and project to language width
Original paper
A simple MLP compresses four adjacent vision tokens into one.
Math and tensor shapes
The release applies LayerNorm with epsilon 1e-6 to each patch before reshaping four neighbors into one vector. Two affine layers with GELU map that concatenation to the LLM width. This is concatenation and learned projection, not averaging. Six vision channels and twelve output channels replace the full model widths; independent convolution and scalar LayerNorm/MLP oracles check both values and adjacency. Backward verifies finite nonzero gradients in the random embedding and merger. No optimizer or model training is performed.
Features [32, 6] → normalized neighbor rows [8, 24] → [8, 12]
Python implementation
class PatchMerger(nn.Module):
"""Normalize each patch, concatenate 2x2 neighbors, then apply an MLP."""
def __init__(self, context_width=6, output_width=12):
super().__init__()
self.context_width = context_width
self.ln_q = nn.LayerNorm(context_width, eps=1e-6)
self.mlp = nn.Sequential(
nn.Linear(4 * context_width, 4 * context_width), nn.GELU(),
nn.Linear(4 * context_width, output_width)
)
def forward(self, features):
if (features.ndim != 2 or features.shape[1] != self.context_width
or features.shape[0] == 0 or features.shape[0] % 4):
raise ValueError("Expected four adjacent patch rows per token.")
neighbors = self.ln_q(features).reshape(-1, 4 * self.context_width)
return self.mlp(neighbors)5. Continue grid positions from the prior maximum
Original paper
Each new modality starts one past the preceding maximum axis ID.
Math and tensor shapes
Text and vision sentinels use the same integer on all three axes. Visual IDs enumerate the merged grid with width varying fastest, then height, then tubelet time. Still-image temporal IDs are constant at the current offset, which need not be zero. Video t is an integer tubelet index, not a timestamp or Qwen2.5-VL time-scaled coordinate. The next text ID follows the maximum across all axes, not the number of preceding visual tokens. Strip masked padding before numbering; padded slots retain the release value 1. Left and right padding preserve every valid position, while the generation delta changes with padded sequence length. Here P denotes valid slots when visual grids are present and all slots for pure text. The pinned pure-text branch takes its maximum over all slots, including padding IDs of 1: a single valid token in length three has delta −1. When visual grids are present, the release uses only valid positions for every sample in that batch. The authored parser also checks complete visual blocks.
Merged grid [T, Gh/2, Gw/2] → positions [3, B, L]
Python implementation
def position_ids(input_ids, attention_mask, grids):
"""Assign [3,B,L] IDs; grids follow visual order, ignoring padding."""
if input_ids.ndim != 2 or attention_mask.shape != input_ids.shape:
raise ValueError("Token IDs and mask must have the same [B,L] shape.")
if input_ids.dtype != torch.long or len(grids) != input_ids.shape[0]:
raise ValueError("Use integer tokens and one grid list per sample.")
if not ((attention_mask == 0) | (attention_mask == 1)).all():
raise ValueError("Attention mask must be binary.")
positions = torch.ones(3, *input_ids.shape, dtype=torch.long)
deltas = []
pure_text = not any(grids)
for batch, row in enumerate(input_ids):
valid = attention_mask[batch].bool()
tokens = row[valid].tolist()
if not tokens:
raise ValueError("Each sample needs at least one valid token.")
chunks, cursor, next_id, visual_index = [], 0, 0, 0
while cursor < len(tokens):
token = tokens[cursor]
if token == VISION_START:
chunks.append(torch.full((3, 1), next_id))
next_id += 1
if visual_index >= len(grids[batch]):
raise ValueError("Missing visual grid.")
kind, grid = grids[batch][visual_index]
if kind not in ("image", "video"):
raise ValueError("Unknown modality.")
if kind == "image" and grid[0] != 1:
raise ValueError("A still image has one temporal tubelet.")
count = validate_grid(grid)
marker = IMAGE if kind == "image" else VIDEO
end = cursor + 1 + count
if (tokens[cursor + 1:end] != [marker] * count
or end >= len(tokens) or tokens[end] != VISION_END):
raise ValueError("Visual count or sentinels disagree.")
t, h, w = grid[0], grid[1] // 2, grid[2] // 2
axes = torch.meshgrid(
torch.arange(t), torch.arange(h), torch.arange(w),
indexing="ij"
)
visual = torch.stack([axis.flatten() for axis in axes])
visual = visual + next_id
chunks.append(visual)
next_id = int(visual.max()) + 1
# The end sentinel is a text position, handled next.
cursor = end
visual_index += 1
else:
if token in (IMAGE, VIDEO, PAD):
raise ValueError("Unexpected visual or padding token.")
unmatched = (
cursor == 0 or tokens[cursor - 1] not in (IMAGE, VIDEO)
)
if token == VISION_END and unmatched:
raise ValueError("Unmatched end sentinel.")
chunks.append(torch.full((3, 1), next_id))
next_id += 1
cursor += 1
if visual_index != len(grids[batch]):
raise ValueError("Unused visual grids.")
positions[:, batch, valid] = torch.cat(chunks, dim=1)
# The pure-text release branch includes padding IDs in its maximum.
# Mixed visual streams retain the valid-position maximum convention.
maximum = int(positions[:, batch].max()) if pure_text else next_id - 1
deltas.append(maximum + 1 - input_ids.shape[1])
return positions, torch.tensor(deltas).unsqueeze(1)6. Select an axis for each rotary channel pair
Original paper
Temporal, height and width positions select separate rotary sections.
Math and tensor shapes
Build three cosine/sine tables, then choose t, h or w for each channel section. Python list multiplication repeats [1,2,3] to [1,2,3,1,2,3]; it does not double the widths to [2,4,6]. Both halves therefore select the same axis for each rotate_half pair. Rotate q and k independently, supporting different attention-head counts. Positions are not three encoded vectors added together. A scalar pair-rotation oracle checks the exact output and gradient; norms remain unchanged. Equal t/h/w positions reproduce ordinary 1D RoPE. Changing one axis changes only its assigned channels in both halves.
q [1, 2, L, 12], k [1, 1, L, 12]; sections [1, 2, 3]
Python implementation
def apply_mrope(query, key, positions, sections=(1, 2, 3), base=10000.0):
"""Select t/h/w channel rotations in each half; rotate [B,H,L,D] q/k."""
if (query.ndim != 4 or key.ndim != 4
or query.shape[0] != key.shape[0]
or query.shape[2:] != key.shape[2:]
or positions.shape != (3, query.shape[0], query.shape[2])):
raise ValueError("Expected compatible q/k and [3,B,L] positions.")
width = query.shape[-1]
invalid_sections = (
len(sections) != 3
or any(type(x) is not int or x <= 0 for x in sections)
)
if invalid_sections or 2 * sum(sections) != width or base <= 1:
raise ValueError("Three sections must sum to half the rotary width.")
frequency = base ** (
-torch.arange(0, width, 2, dtype=query.dtype) / width
)
angles = positions.to(query.dtype).unsqueeze(-1) * frequency
angles = torch.cat((angles, angles), dim=-1)
# Python list repetition gives [t,h,w,t,h,w], not doubled section widths.
sizes = list(sections) * 2
cosine = torch.cat(
[block[i % 3]
for i, block in enumerate(angles.cos().split(sizes, -1))],
dim=-1
).unsqueeze(1)
sine = torch.cat(
[block[i % 3]
for i, block in enumerate(angles.sin().split(sizes, -1))],
dim=-1
).unsqueeze(1)
def rotate_half(value):
first, second = value.chunk(2, dim=-1)
return torch.cat((-second, first), dim=-1)
return (query * cosine + rotate_half(query) * sine,
key * cosine + rotate_half(key) * sine)7. Reconcile sentinels, patches and merged placeholders
Original paper
Vision boundaries surround the compressed feature sequence.
Math and tensor shapes
Each image or video block contains one vision_start, exactly N_v repeated image/video placeholders, and one vision_end. Sentinels are vocabulary tokens, not visual features; 224 × 224 yields 256 patches, 64 merged features and 66 total block tokens. One video block contains all of its tubelets, rather than a separate sentinel pair for each frame. The authored assembly uses synthetic reserved IDs and checks modality, grids, sentinel order and feature counts before returning the stream. It executes one rectangular image, two images, and interleaved text/image/video. It does not run the tokenizer, substitute features into an LLM, generate answers or reproduce benchmark accuracy.
Rectangular image: 32 patches, 8 merged placeholders, two sentinels
Python implementation
def pack_stream(parts, embed, merger, padding=0, pad_left=False):
"""Pack synthetic text IDs and actual reduced image/video tensors."""
if type(padding) is not int or padding < 0:
raise ValueError("Padding must be a nonnegative integer.")
ids, features, grids, records = [], [], [], []
for kind, payload in parts:
if kind == "text":
if not payload or any(type(x) is not int or not 0 < x < 90
for x in payload):
raise ValueError("Synthetic text IDs must be in [1,89].")
ids.extend(payload)
continue
if kind not in ("image", "video"):
raise ValueError("Unknown modality.")
if (not isinstance(payload, torch.Tensor) or payload.ndim != 4
or payload.shape[1] != embed.channels):
raise ValueError("Visual tensors must match embedding channels.")
if kind == "image" and payload.shape[0] != 1:
raise ValueError("A still image must supply one frame.")
height, width = payload.shape[-2:]
size = smart_resize(height, width, 2, 2, 16, 256)
paired = prepare_tubelets(payload, size)
patches, grid = flatten_patches(paired, 2, 2)
merged = merger(embed(patches))
count = validate_grid(grid)
if patches.shape[0] != 4 * count or merged.shape[0] != count:
raise ValueError("Patch, feature and placeholder counts differ.")
marker = IMAGE if kind == "image" else VIDEO
ids.extend([VISION_START] + [marker] * count + [VISION_END])
features.append(merged)
grids.append((kind, grid))
records.append({
"kind": kind, "input_shape": list(payload.shape),
"paired_shape": list(paired.shape), "grid_thw": list(grid),
"patch_shape": list(patches.shape),
"merged_shape": list(merged.shape), "sentinels": 2,
})
padded = [PAD] * padding + ids if pad_left else ids + [PAD] * padding
mask = ([0] * padding + [1] * len(ids) if pad_left
else [1] * len(ids) + [0] * padding)
tokens = torch.tensor([padded], dtype=torch.long)
attention = torch.tensor([mask], dtype=torch.long)
positions, deltas = position_ids(tokens, attention, [grids])
return {
"input_ids": tokens, "attention_mask": attention,
"positions": positions, "deltas": deltas,
"features": features, "records": records,
}Smoke JSON records the source SHA-256, seed, reductions, actual input, tubelet, patch and merged shapes, three sequence position traces, padding stability, rotary norm error, finite diagnostic loss and module gradients. It runs entirely offline and performs zero optimizer steps.
qwen2vl.pyrequirements-qwen2vl.txt
python -m pip install -r requirements-qwen2vl.txt python qwen2vl.py --smoke --seed 7
One model for image, multi-image, and video
Dynamic resolution and M-RoPE are what let a single Qwen2 LLM handle every visual input type without routing logic. A single image becomes a block of visual tokens preceded by a <|vision_start|> sentinel and followed by <|vision_end|>, interleaved inline with the surrounding text prompt. A multi-image conversation simply appends additional vision blocks in sequence within the same context window. A video pairs frames into depth-two tubelets and flattens their merged grids into one vision block, with the temporal M-RoPE axis distinguishing tubelet indices. A still image is duplicated into two identical frames to use the same patch embedding.
The LLM sees one flat token sequence: text tokens, vision tokens, text tokens, vision tokens — all carrying consistent M-RoPE position IDs. There is no separate video encoder, no modality-specific MLP head, and no special routing at inference time. The model can therefore handle arbitrary interleavings of text and visual content, including referring back to an earlier image while answering a question about a later one.
Why it mattered
Qwen2-VL set a high open-VLM bar at release. The 72B model matched GPT-4o on DocVQA, outperformed it on MathVista, and rivaled Claude 3.5 Sonnet on several other benchmarks — a rare achievement for an openly released model. But the architectural contributions are arguably more durable than the benchmark numbers.
Dynamic resolution lets token counts vary with the processed grid, subject to configured pixel and context budgets. Larger budgets can retain more detail, while rounding and resizing keep the packed input feasible. M-RoPE makes position information genuinely multimodal: rather than stitching together separate 1D text positions and 2D image positions with hacks, every token in the model carries a coherent (t, h, w) address. Future models extending to 3D volumes, point clouds, or other spatiotemporal data have a clean position framework to build on.
Together, these two ideas — token counts that scale with content, and positions that decompose across modalities — form a principled foundation for any-resolution, any-modality perception in a single language model.
Related Reading
- LLaVA: Visual Instruction Tuning — the seminal open VLM recipe of frozen vision encoder + linear projection + instruction tuning that Qwen2-VL builds on
- Flamingo — the pioneering few-shot VLM that introduced interleaved image–text sequences and cross-attention bridging
- CLIP: Visual Models via Language Supervision — the contrastive vision encoder whose ViT architecture underlies most modern VLM image encoders