VideoMamba: State Space Model for Efficient Video Understanding

A video is a sequence before it is a state space problem

Recognizing a person opening a drawer requires both appearance and change. A single frame may show the drawer, but the action depends on what came before and after. Feeding more frames to a model supplies more evidence while multiplying the number of spatial patches it must process. A useful video backbone therefore needs a way to exchange information across a long clip without making every patch compare directly with every other patch.

VideoMamba makes that exchange a bidirectional selective scan over one spatiotemporal sequence. It keeps the isotropic, patch-based organization of a vision transformer, but replaces attention-based token mixing with B-Mamba blocks. The central design question becomes concrete: in what order should the model read the video? That order affects both the running state and the local convolution that precedes it. The model is described in §3 of the March 2024 v2 paper; its official repository identifies the publication as ECCV 2024.

The figures below are authored illustrations and calculated arithmetic, not pretrained model outputs. Published benchmark values are explicitly labeled and linked to their tables. The downloadable Python checks small reference calculations; it does not reproduce VideoMamba training, accuracy or runtime.

First decide what one token contains

Start with a tensor shaped B\times3\times T\times H\times W. A 3D convolution uses a temporal kernel \tau and spatial patch size p, with strides equal to those sizes. Each output location becomes a learned feature vector. The resulting grid has temporal, row and column coordinates; flattening it creates a sequence but does not remove the need to represent those coordinates.

The released patch projection and forward pass use a default temporal kernel of 1, so the default token is a spatial patch within a frame. Video input alone does not imply that every patch embedding already mixes multiple frames. A larger temporal kernel in the explorer illustrates tubelet geometry; it is not a claim about the chosen paper recipe.

For divisible inputs, patch count is

The additional position is one global class token. The implementation adds spatial position embeddings per frame and temporal embeddings per spatial location, then constructs a frame-major sequence and prepends the class token. Its final normalized class feature feeds the classifier. At 16 frames, 224-square resolution, patch size 16 and temporal kernel 1, this gives 3,136 patch tokens and 3,137 sequence positions. These are calculated dimensions, not operation counts.

The scan order is part of the inductive bias

Once tokens have coordinates, there are several plausible traversals. Spatial-first reads all patches of one frame, then the next frame. Temporal-first follows a fixed spatial location through time before moving to the next location. Both enumerate exactly the same tokens, but they present different neighbors to a sequence model.

This matters in two ways. First, information from a particular patch crosses different numbers of state updates before reaching another patch. Second, the short 1D convolution inside Mamba now sees different local patterns. Spatial-first favors contiguous appearance neighborhoods and preserves the order an image-pretrained model already encountered. Temporal-first makes consecutive frames at one location adjacent, but also changes the meaning of every local convolution neighborhood. This explanation is an architectural interpretation; the measured comparison is the paper's scan ablation in Figure 7.

That ablation reports 65.1% SSV2 top-1 for spatial-first bidirectional versus 62.4% for temporal-first bidirectional, using the Tiny model fine-tuned from ImageNet pretraining. Mixed spatiotemporal variants do not improve on the simple choice in that experiment; alternating spatial and temporal layers also reaches 65.1%. The result supports the selected order under that transfer protocol. It does not prove that spatial-first is optimal for every task, sampling strategy or from-scratch training regime.

Spatial-first also introduces artificial boundaries: the last patch of a row precedes the first patch of the next row, and the last patch of a frame precedes the next frame's first patch. Positional embeddings preserve coordinate information, but they do not make those transitions equivalent to physical adjacency. VideoMamba accepts this sequence bias rather than constructing a full 3D neighborhood graph.

Two directions let a prefix token read the entire clip

A forward-only recurrence cannot make a class token at the beginning depend on tokens it has not visited yet. Bidirectionality supplies a second information path. The projected sequence is scanned in its original order and in reverse order; the reverse branch's output is restored to the original coordinate system before the branches are combined.

A scalar state illustrates the arithmetic:

In Mamba, the input controls the selection parameters, including the step size and the input/output state projections. This makes retention and writing content-dependent. The scalar explorer deliberately supplies those coefficients directly so the reader can separate directional reachability from the much larger learned module. It does not claim that a fixed exponential kernel is VideoMamba's learned behavior. See Equations 1–6 and the selective-SSM description.

The pinned fast bidirectional implementation has separate forward and backward convolution and selective parameters. It flips the projected sequence for the backward branch, flips that output back, sums the two outputs, and applies the output projection. The branch internals also contain gating and a skip term. Around this mixer, normalization, residual accumulation and stochastic depth form the block. None of those details are replaced by the scalar diagram.

The backward path is also a deployment limitation. Classification can use the entire sampled clip, including later frames. This is not a causal streaming architecture merely because each directional core is recurrent. A persistent forward cache cannot reproduce the full bidirectional result before future inputs arrive. Any online deployment needs a changed context policy and its own evaluation.

Scaling the backbone requires a training argument

An inexpensive mixer does not guarantee that a larger visual model will optimize or generalize better. VideoMamba keeps one embedding width through the stack instead of progressively downsampling as a hierarchical backbone does. The reported family uses Tiny (24 blocks, width 192), Small (24, 384) and Middle (32, 576). A Base configuration (24, 768) appears in the architecture table but is excluded from the final family because of suboptimization. Table 1 and §3.3 make that limitation part of the scaling story.

The remedy is self-distillation across model sizes: a trained Small model guides Middle through feature alignment while the student also learns the classification task. Here “self” refers to the model family; it is not an exponential-moving-average copy of the same student, and the teacher is smaller rather than larger.

The paper describes L2 alignment of final feature maps. The distillation model supplies a projection to teacher feature width and normalization, while the training engine adds the mean normalized feature distance to classification loss. The teacher runs without gradient tracking. This is feature supervision, distinct from matching softmax class probabilities with a temperature-scaled KL divergence.

For nonzero normalized vectors,

The identity explains the explorer's scale invariance and range of zero to four. It does not establish training stability by itself. The evidence for the recipe is the convergence ablation in Figure 6 and §4.1. That section uses ImageNet-1K training for 300 epochs, AdamW and BF16; extending those results to substantially larger models remains a separate empirical question. The paper's early-stopping experiment does not improve the teacher-guided result, so “stop distilling earlier” is not the reported solution.

Masked video alignment is a different pretraining protocol

The image self-distillation experiment should not be conflated with masked video pretraining. The latter follows UMT-style alignment: a video student learns from a frozen CLIP-ViT-B teacher, with visible tokens aligned at the final output. This imports knowledge from a teacher pretrained on CLIP's image-text corpus. Calling the student stage self-supervised does not erase that upstream supervision or data. The paper explicitly records the CLIP-400M teacher provenance in Tables 3–4.

Mask choice matters because deleting tokens changes the stream seen by the pre-SSM convolution. A scattered mask can turn formerly distant patches into adjacent sequence positions. Keeping runs preserves more of the original local structure. That motivates row-based masking, alongside random, tube and attention-based alternatives.

The distinction between clip-row and frame-row is their sampling scope: clip-row masks the flattened clip, whereas frame-row masks frames individually. The explorer isolates the simpler adjacency issue with deterministic fixtures; it does not implement either sampler's full randomness or the teacher's attention mask. The source is §3.4 and Figure 5.

The most informative ablation is Table 5, which uses 200 epochs, not the 800-epoch recipe for the main masked-pretraining results. Its mask-type comparison reports SSV2 top-1 of 67.4 for random, 66.3 for tube, 68.2 for clip-row, 67.8 for frame-row and 68.5 for attention masking. Last-output alignment performs best among the tested alignment choices, and the mask-ratio comparison peaks at 80%. Thus preserving runs is a useful motivation, but the best measured option is not simply “keep rows.” Semantic mask selection also matters, and these are local ablations under one recipe.

Multimodal training introduces another stage: the visual backbone is combined with a text encoder and cross-modal decoder, trained using contrastive, matching, masked-language and unmasked-token alignment objectives. The paper adds an unmasked tuning phase after masked training. Consequently, a retrieval result tests an entire alignment pipeline rather than a visual backbone with a text head attached after training. See §4.4 and Table 8.

Linear token mixing changes what can fit, not what a benchmark proves

At fixed model width and state size, scanning a sequence twice still has linear dependence on its length. Increasing frames therefore grows the sequence proportionally. Increasing both spatial dimensions multiplies the patch count quadratically in image side length; the mixer is linear in tokens, not linear in resolution. Learned projections, local convolutions, activations and all layers still contribute to memory and runtime.

The benchmark selector intentionally preserves pretraining and evaluation-view counts. For K400, the 16-frame Middle rows have the same 224-square input and 202G reported per-view FLOPs: ImageNet-based supervised pretraining gives 81.9% top-1, and the masked recipe gives 83.4%, both with 3 spatial crops and 4 temporal clips. That is evidence about two complete training recipes, not proof that a single loss term caused the difference. For SSV2, the 16-frame rows are 68.3% and 71.0%, but the supervised row uses 3×4 views while the masked row uses 3×2. Those view counts must remain visible. Tables 3–4 are the sources; the page's multiplied totals are arithmetic, not new profiling.

Nor does VideoMamba uniformly dominate contemporary masked video transformers. Table 3 reports UMT-B at 85.7% K400 top-1 with an 8-frame input and CLIP teacher provenance, versus 82.0% for the corresponding 8-frame masked VideoMamba-M row. The models have different parameter counts and compute, so this is a tradeoff rather than a controlled substitution of one mixer. Likewise, “more frames helps” depends on the task: Figure 7 shows a more consistent benefit on K400 than on short SSV2 clips, and higher resolution is not uniformly better.

The paper's Figure 1 reports substantial speed and GPU-memory advantages for its long-input comparison. Those are measurements for the authors' compared workloads, not consequences that can be read directly from the two complexity curves above. Reproducing a speedup requires hardware, precision, batch size, implementation, clip shape and measurement boundary. This review has not rerun that benchmark and does not assign a runtime to its operation-count illustrations.

Long videos still arrive as sampled evidence

The long-term evaluation is valuable because it trains directly from video instead of relying on a separately extracted feature bank. Yet an end-to-end 32- or 64-frame model does not read every frame of a multi-minute recording. Temporal sampling remains a bottleneck: an event absent from the sampled frames cannot be recovered by a better scan.

Table 6 also provides a useful counterexample to monotonic scaling. VideoMamba-S improves from 95.3 to 97.4 on Breakfast when moving from 32 to 64 frames, whereas masked-pretrained Middle moves from 97.9 to 96.9 on that dataset while improving from 89.6 to 90.4 on COIN. The input length, model size and pretraining interact; “longer always wins” would misread the table. Comparisons against feature-based methods also change the training boundary and upstream pretraining datasets, so they do not isolate the state space operator.

The paper's limitations leave larger-scale backbones, audio and hour-level video understanding with language models unvalidated. That is the appropriate boundary for the result: an efficient clip backbone with evidence across several video tasks, not a demonstrated general-purpose memory for arbitrary-length footage.

From the equations to a runnable reference

The five rows below retain the connection between source, shapes and executable arithmetic. They are deliberately smaller than the network: counting a valid patch grid, enumerating scan permutations, restoring reverse states, computing normalized feature distance and inspecting masked adjacency. They make hidden indexing assumptions testable without suggesting that a scalar script reproduces a trained video system.

These five rows display generated excerpts from examples/videomamba.py. They implement small deterministic arithmetic problems using Python’s standard library. Official source links are pinned to revision 37355c2.

1. Count the sequence before choosing a mixer

Original paper

Enlarge figure

Math and tensor shapes

\begin{aligned}T^\prime&=\lfloor T/\tau\rfloor\\N&=T^\prime\lfloor H/p\rfloor\lfloor W/p\rfloor\\L&=N+1\end{aligned}

[B,3,T,H,W] → [B,D,T′,H′,W′] → [B,N+1,D]

The released PatchEmbed uses matching kernel and stride without padding. For divisible inputs this is the usual tubelet formula. Our helper also makes discarded border regions explicit through floor division, rather than silently padding. It counts shapes only, not learned Conv3d projections.

Inspect pinned official source

Python implementation

python

def token_geometry(frames, height, width, tubelet=1, patch=16):
    """Count valid non-overlapping tubelets and the one global class token."""
    dimensions = (frames, height, width, tubelet, patch)
    if any(type(value) is not int or value <= 0 for value in dimensions):
        raise ValueError("Dimensions must be positive integers")
    time, rows, columns = frames // tubelet, height // patch, width // patch
    if not min(time, rows, columns):
        raise ValueError("Kernel must fit input")
    tokens = time * rows * columns
    return {"time": time, "rows": rows, "columns": columns,
            "tokens": tokens, "sequence": tokens + 1}

2. A reshape is an architectural decision

Original paper

Enlarge figure

Math and tensor shapes

\begin{aligned}i_{\rm SF}(t,r,c)&=tH^\prime W^\prime\\&\quad+rW^\prime+c\end{aligned}

3 frames × 2 rows × 3 columns → a permutation of IDs 0…17

IDs are original coordinates, not embeddings. Reordering changes the locality of the pre-SSM convolution and the path through the state. Spatial position embeddings are added per frame; temporal embeddings are added per spatial coordinate; the final rearrange restores frame-major order before the global CLS token is concatenated.

Inspect pinned official source

Python implementation

python

def scan_order(time, rows, columns, temporal_first=False):
    """List spatial-first IDs in the selected scan's traversal order."""
    if any(type(value) is not int or value <= 0
           for value in (time, rows, columns)):
        raise ValueError("Grid must be positive")
    spatial = rows * columns
    if temporal_first:
        return [position + frame * spatial
                for position in range(spatial) for frame in range(time)]
    return list(range(time * spatial))

3. Restore reverse outputs to their original positions

Original paper

Enlarge figure

Math and tensor shapes

\begin{aligned}h_i&=a_i h_{i-1}+b_i x_i\\\tilde y_i&=\operatorname{reverse}(y^{\leftarrow})_i\\y_i&=y_i^{\rightarrow}+\tilde y_i\end{aligned}

input [L] → forward [L], restored backward [L] → sum [L]

The scalar helper takes arbitrary per-token decay/write coefficients, so its tests can expose reversal mistakes even when coefficients vary. The UI uses a constant coefficient fixture. The real fast path learns separate branch convolution and selective parameters, gates branch outputs and applies a shared output projection. There is no division by two in that pinned sum.

Inspect pinned official source

Python implementation

python

def recurrence(values, decay, write):
    """Evaluate a scalar recurrence with explicitly supplied coefficients."""
    if not len(values) == len(decay) == len(write):
        raise ValueError("Equal-length arrays required")
    if not all(math.isfinite(v) for v in [*values, *decay, *write]):
        raise ValueError("Finite arrays required")
    state, output = 0.0, []
    for value, retention, gain in zip(values, decay, write):
        state = retention * state + gain * value
        output.append(state)
    return output


def bidirectional(values, decay, write):
    """Align a reversed scan before summing; weights are tied for this demo."""
    forward = recurrence(values, decay, write)
    backward = recurrence(values[::-1], decay[::-1], write[::-1])[::-1]
    return {"forward": forward, "backward": backward,
            "merged": [left + right for left, right in zip(forward, backward)]}

4. Align normalized features, not class probabilities

Original paper

Enlarge figure

Math and tensor shapes

\begin{aligned}\mathcal L_{\rm feat}&=\|\hat s-\hat t\|_2^2\\&=2-2\hat s^\top\hat t\\\mathcal L&=\mathcal L_{\rm cls}+\mathcal L_{\rm feat}\end{aligned}

student [B,N+1,576] → projected [B,N+1,384]; teacher [B,N+1,384]

All final positions, including CLS, are aligned. The distillation model projects student features into teacher width and normalizes along the feature dimension. The training engine averages 2−2·dot across its feature positions, with the teacher evaluated under no_grad. The helper computes the nonzero single-vector forward loss. It does not implement the projection, classification loss, gradients or an optimizer; zero vectors are rejected as an explicit teaching boundary.

Inspect pinned official source

Python implementation

python

def feature_loss(student, teacher):
    """Squared L2 distance after normalization; zero vectors are rejected."""
    if not student or len(student) != len(teacher):
        raise ValueError("Matching nonempty features required")
    if not all(math.isfinite(v) for v in [*student, *teacher]):
        raise ValueError("Finite features required")
    student_norm, teacher_norm = math.hypot(*student), math.hypot(*teacher)
    if not student_norm or not teacher_norm:
        raise ValueError("Feature vectors must be nonzero")
    return sum((left / student_norm - right / teacher_norm) ** 2
               for left, right in zip(student, teacher))

5. Count the adjacency a mask leaves behind

Original paper

Enlarge figure

Math and tensor shapes

\begin{aligned}A&=\sum_{j=1}^{K-1}\mathbf1[k_j=k_{j-1}+1]\\q&=A/(K-1)\end{aligned}

mask [L] → retained original IDs [K] → preserved neighbor count

Here k is the increasing list of retained IDs. When fewer than two survive we report zero rather than divide by zero. This metric deliberately tests scan adjacency, not 2D distance or semantics. The fixtures retain 50% for readability; they neither sample the published attention masks nor reproduce the preferred 80% masking protocol.

Inspect pinned official source

Python implementation

python

def retained_mask(time, rows, columns, mode="row"):
    """Make authored 50% retention fixtures, not upstream sampled masks."""
    length = len(scan_order(time, rows, columns))
    if mode not in ("row", "scattered"):
        raise ValueError("Unknown mask")
    return [(index // columns) % 2 == 0 if mode == "row" else index % 2 == 0
            for index in range(length)]


def adjacency(mask):
    """Count neighbors in the compacted stream that were already adjacent."""
    kept = [index for index, retain in enumerate(mask) if retain]
    pairs = max(0, len(kept) - 1)
    original = sum(right == left + 1 for left, right in zip(kept, kept[1:]))
    return {"kept": kept, "pairs": pairs, "original": original,
            "fraction": original / pairs if pairs else 0.0}

Original paper crops: Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, Yu Qiao. v2 source · License. Cropped/rasterized from the original PDF; colors unchanged.

Download the reference and run python3 videomamba.py --smoke. It checks invariants and prints deterministic JSON. No training, checkpoint inference or timing measurement is performed.

Python referenceRequirementsREADME and limitations

What to carry into the next model

VideoMamba's strongest architectural lesson is that serialization is a modeling choice. Patch geometry decides what information each token contains. Scan order decides which information is local. Bidirectionality decides whether a token can depend on later inputs. Training protocols then determine whether these choices transfer, scale and align with other modalities. Linear sequence complexity widens the feasible input budget, but cannot settle those other questions.

For a new application, evaluate the temporal sampler, causality requirements and training provenance together with throughput. A scene-heavy classification task, a fine-grained motion task and continuous online recognition impose different constraints. The published experiments justify investigating the backbone across those settings; they do not remove the need for task-specific tests.

Related reading

• Mamba explains input-dependent selective state updates and why a state can compress a long sequence without materializing a dense attention matrix.
• Vision Mamba supplies the bidirectional visual lineage; compare its position and class-token choices with VideoMamba's simpler frame-major sequence.
• VMamba explores a different answer to image traversal: cross-scans and a hierarchical visual backbone rather than VideoMamba's isotropic clip sequence.
• CLIP explains the image-text teacher provenance behind masked alignment. Its supervision should remain visible when comparing downstream “self-supervised” recipes.
• The VideoMamba paper, v2 and official implementation pinned at 37355c2 are the primary sources for mechanisms, protocols and published values on this page.