Vision Mamba: Bidirectional State Space Models for Images

An image has two sides; a causal sequence has a past

An image patch becomes easier to interpret with context: a small curved edge might belong to a cup, an eye, or a wheel. A model needs a way to bring information from other patches to that location. Self-attention compares token pairs directly. Vision Mamba, or Vim, passes a compact running state through the patch sequence in both directions.

First, look at the order introduced by flattening an image. These twelve numbers identify patch positions in an illustrative 3×4 grid; they are not pixel values or learned features.

Read row by row: 0, 1, 2, 3, 4, 5, …, 11. A forward causal scan at patch 5 has seen patch 1 above it, but has not seen patch 9 below it. The transition from 3 to 4 also crosses the image from one row's right edge to the next row's left edge. Sequence order supplies a notion of past; it does not preserve every spatial adjacency.

Vim keeps this sequence representation and gives each token a forward view and a backward view. To follow those views, we first need to know what enters the sequence and what a state carries.

From pixels to position-aware tokens

Patch extraction collects a local RGB window. A learned projection turns its values into a feature vector; its coordinates are learned channels, not a list of pixel locations. With non-overlapping windows, a convolution whose kernel and stride both equal the patch size performs the same operation as a shared linear projection of flattened patches.

The released Vim-Tiny middle-CLS baseline uses a 224×224 RGB image and 16×16 windows at stride 16. Its patch projection produces a 14×14 grid of 192-channel vectors. Flattening the two spatial axes gives 196×192 patch features for each image.

CLS is a learned classification token, not an extra image patch. The selected release inserts it at sequence index 98: patch IDs 0–97, CLS, then patch IDs 98–195. There are now 197 tokens of width 192. Adding a learned absolute position vector to every sequence slot preserves information about where the tokens belong. Changing CLS placement in the small demonstration illustrates assembly; it does not retrain those position embeddings. Pinned patch embedding and token assembly, middle-CLS insertion and positions

The paper's Equation 4 writes CLS at the prefix of the sequence. Figure 2 in v3 already depicts middle CLS, as does this released model. These placements describe different sequence access at CLS; we will use middle CLS throughout the released-model walkthrough.

Each assembled token is still just a vector. The next operation builds a running summary from a stream of those vectors.

A state retains, writes, and reads

Imagine carrying a small notebook along the sequence. At each token, keep some of the previous note, write something from the current input, then read an output. The notebook has a fixed number of coordinates even if the sequence becomes longer. It compresses history rather than keeping a separate entry for every previous token.

For a scalar illustration, let u_t be the current input and h_{t-1} the previous state. The retention factor a_t preserves history; the write coefficient b_t scales new input. The read coefficient c_t extracts an output, and d supplies a direct input skip:

For example, previous state 2, retention 0.5, input 3, and write coefficient 0.2 give a new state of 1.6. The controls below compute this kind of update. Their values are illustrative mechanics, not activations or predictions from a trained model.

The learned model makes the update selective. From the current directional content, projections derive Δ (the update interval), B (the input write), and C (the output read). A (state decay) and D (the direct skip) are learned parameters independent of the current token. B and C can be signed. Δ is made positive with softplus, and affects both retention and writing; it is not an attention weight or a probability gate.

Now index an expanded channel by j and a state coordinate by n. The learned parameter A^{\log}_{j,n} stores the logarithm of the decay magnitude. \mathrm{raw}\Delta_{t,j} is the current content's projected update value before softplus, and \mathrm{bias}_j is its learned channel bias. For one image and one direction, the released numerical reference uses:

Tiny has 16 state coordinates for each of its 384 expanded channels in each direction. B and C vary by token and direction and are shared across channels within that direction; Δ varies by channel too. Each directional scan starts from zero state. A state of fixed size can lose information, and many histories can lead to the same summary. The equations describe available operations, not guaranteed retention of every detail.

There is a discretization detail worth keeping visible. Paper Equation 1 gives exact zero-order hold for a continuous system. With A, B, and input held constant during one interval, its scalar input contribution is \frac{\exp(\Delta A)-1}{A}Bu. The released selective_scan_ref instead uses \Delta Bu, while retaining \exp(\Delta A) for decay. The input expressions agree to first order as Δ tends to zero, but they are not identical. Our teaching code follows the released recurrence. Paper v3, §3.1, pinned selective_scan_ref

A selective state can now summarize a prefix. The missing context is the other side of the token.

Two scans meet at the same token

The forward scan processes the full sequence in its original order. The backward scan processes the entire reversed sequence, including CLS. Neither resets at a row boundary. Each direction has its own convolution and state-space parameters, so the backward branch is more than a reversed copy of the forward output.

At patch 5 in our small grid, the forward path can receive information from patches 0–5; the backward path can receive information from patches 11–5. Their union reaches every patch. With middle CLS between patches 5 and 6, its two views cover opposite halves of the image. With prefix CLS, its forward view initially includes no image patches, while its backward view includes them all.

These highlighted paths show structural reachability. They do not measure saliency, learned influence, or an effective receptive field. A distant input's contribution depends on the sequence of decay factors, writes, reads, and features.

Backward results are initially stored in reversed order. Reverse them again before combining, so original token t meets original token t. Let L be the assembled sequence length, y_t^{\rightarrow} the forward result, and \widetilde y_s^{\leftarrow} the result at reverse-stream index s. Alignment and the selected release's average are:

Without that inverse reversal, the two branches would mix different original positions. The divide by two comes from if_divide_out=True in the selected factory; Paper Algorithm 1 shows an addition. We now have aligned contextual features, ready to become a block's contribution to the residual stream.

Inside the released Vim block

Normalize the incoming residual features with RMSNorm, then use one shared input projection to form content x and gate z. Each has width 384. The forward content passes through its depthwise kernel-four causal Conv1d, then SiLU, then its selective scan. The backward branch reverses the sequence and runs its own convolution, projections, and scan. In each branch, Δ is projected through a rank-12 intermediate representation before expanding to 384 channels.

The convolution mixes nearby sequence positions, not a two-dimensional 4×4 image neighborhood. It crosses row boundaries just as the sequence does. The SSM adds longer context. Both paths include their own direct D skip inside the scan.

The gate comes from the shared projection of the normalized input, without that directional convolution. Define g_t=\operatorname{SiLU}(z_t) and let W_{\mathrm{out}} project 384 channels back to 192. The mixer contribution is:

Here \odot means elementwise multiplication and \sigma is the sigmoid function. SiLU is signed and unbounded above: \operatorname{SiLU}(z)=z\,\sigma(z). A negative z can produce a negative gate, and a positive gate can exceed one. Calling it a probability would describe the wrong operation.

The pinned v2 code gates each direction before reversing the backward output and averaging. It reverses the entire shared x/z projection for that branch, so its gate returns to the same original token after alignment. Multiplication by this shared per-position gate distributes over addition; the after-average diagram above is algebraically equivalent. Pinned v2 mixer, scan gate

Let R_\ell be the residual features entering block ℓ and v_\ell its mixer contribution. At inference, a complete residual update can be written R_{\ell+1}=R_\ell+v_\ell. The direct D skip is inside each SSM; this residual addition carries the full token features around the mixer. There is no separate Transformer feed-forward MLP sublayer after the mixer in this Vim block.

The original figure puts the two directions and residual path together:

The mixer keeps the token count and width unchanged. Stacking it therefore builds deeper features without a spatial pyramid.

A constant-resolution backbone and a middle-token readout

Vim-Tiny stacks 24 blocks, each operating on 197×192 tokens. Internally, the mixer expands channels to 384 and contracts them again. Patch resolution remains 14×14 throughout; no stage downsamples the image grid.

The official implementation carries two tensors between blocks: the accumulated residual and the latest mixer contribution. A block first adds the previous contribution to the carried residual, normalizes that sum, then produces the next contribution. The first block starts from the assembled tokens. After block 24, a final addition incorporates the last contribution, and a final RMSNorm prepares the readout. This deferred addition permits fused add-and-normalize execution. It represents the same residual chain at inference, but a raw block's hidden_states output alone is not the complete residual feature tensor. Recorded stages must distinguish those two quantities. Training additionally applies stochastic depth on the residual contributions. Pinned Block, final addition and normalization

Select the normalized vector at CLS index 98, giving 192 image features, then apply the released linear head to produce 1,000 ImageNet logits. Softmax converts these scores into class probabilities.

The factory's long name contains final_pool_mean, but forward_features returns CLS before reaching the pooling switch whenever CLS is enabled. This model does not mean-pool all 197 tokens. The paper calls its prediction head an MLP; the selected release uses a single linear layer. Pinned CLS readout, Tiny factory, linear head

We can now connect the image, running state, directional alignment, residual stream, and readout to executable tensor operations.

Implementation: paper, math, and Python

Read the eight rows in order: patches, tokens, selectivity, recurrence, bidirectional mixing, the block, readout, and long sequences. Each row places the paper reference beside explained symbols, shapes, and Python extracted from the runnable reference. Algorithm 1 is architectural pseudocode; it is not a complete executable implementation. The source below makes the released exponential retention, ΔBu input write, signed gate, and direction average explicit.

Follow a small runnable CPU PyTorch model. It preserves the selected block’s operations and makes the state updates readable.

Download the Python reference · Pinned requirements · Pinned official model source

python vision_mamba.py --smoke --backward

The smoke run uses random weights to check finite forward and backward passes. The pretrained example below uses a separate exporter and released weights.

1. Turn image windows into a patch sequence

Original paper

Enlarge figure

Math and tensor shapes

P=H_pW_p,\quad X_p\in\mathbb{R}^{b\times P\times d}

H_p=\left\lfloor\frac{H-k}{s}\right\rfloor+1

A Conv2d with kernel and stride 16 performs the patch projection. Flatten only the spatial axes and transpose channels last. Rows change slowly and columns change quickly. This is the baseline checkpoint’s nonoverlapping patch geometry.

224 × 224 RGB → 14 × 14 windows → [b, 196, 192]

Python implementation

python

def patch_tokens(image, projection):
    """Project RGB windows and flatten H then W, with columns changing fastest."""
    grid = projection(image)  # [batch, width, rows, columns]
    return grid.flatten(2).transpose(1, 2)  # [batch, patches, width]

2. Insert the class token and add image positions

Original paper

Enlarge figure

Math and tensor shapes

m=\lfloor P/2\rfloor

X_0=[X_p[:m],x_{cls},X_p[m:]]+E_{pos}

CLS is a learned vector, not an image patch. Its position lets both directional scans arrive with information from half the image. Adding a position vector changes the features without changing sequence length. The small teaching model checks the same assembly on a rectangular grid.

[b, 196, 192] → CLS at index 98 → [b, 197, 192]

Python implementation

python

def assemble_tokens(patches, cls_token, positions):
    """Insert the learned CLS at floor(patch_count / 2), then add positions."""
    midpoint = patches.shape[1] // 2
    cls = cls_token.expand(patches.shape[0], -1, -1)
    tokens = torch.cat(
        (patches[:, :midpoint], cls, patches[:, midpoint:]), dim=1
    )
    if tokens.shape[1:] != positions.shape[1:]:
        raise ValueError("Position embeddings must match this token grid")
    return tokens + positions, midpoint

3. Choose the state update from the current content

Original paper

Enlarge figure

Math and tensor shapes

\Delta_t=\operatorname{softplus}(W_\Delta r_t+b_\Delta)

A=-\exp(A_{log})

A depthwise kernel-four convolution processes the scan order. The projection supplies low-rank r, write coefficients B and read coefficients C; another projection expands r to a step size for each channel. The learned negative diagonal A stays input-independent. A positive step size controls retention and writing; it is not a probability.

Per direction: expanded width 384 · state 16 · Δ rank 12

Python implementation

python

def forward(self, content):
    """Build token-dependent delta/B/C after this direction's causal conv."""
    u = causal_convolution(content, self.conv.weight, self.conv.bias)
    projected = self.x_proj(u.transpose(1, 2))
    dt, b, c = projected.split(
        [self.delta_rank, self.state_size, self.state_size], dim=-1
    )
    delta = self.dt_proj(dt).transpose(1, 2)
    a = -self.a_log.exp()  # Learned, input-independent negative dynamics.
    return selective_scan(
        u, delta, a, b.transpose(1, 2), c.transpose(1, 2), self.d
    )

4. Keep a running state, then read it

Original paper

Enlarge figure

Math and tensor shapes

h_t=\exp(\Delta_t A)\odot h_{t-1}+\Delta_t B_tu_t

y_t=\sum_n C_{t,n}h_{t,n}+D u_t

The running state compresses earlier content. B writes the current input into that state; C reads it into one output per channel. D supplies a direct input skip. This ΔBu update matches the reference implementation, rather than claiming the exact zero-order-hold input integral. The loop favors clarity; parallel CUDA kernels compute it differently.

Running state [b, 384, 16] · output [b, 384, 197]

Python implementation

python

def selective_scan(
    u, delta, a, b, c, d, z=None, delta_bias=None, delta_softplus=True
):
    """Scan a real diagonal state; B/C vary per token and share channels.

    Args:
        u: Content, shape [batch, channels, length].
        delta: Step-size preactivation with the same shape as u.
        a: Negative learned dynamics, shape [channels, state].
        b: Token-dependent input write, shape [batch, state, length].
        c: Token-dependent output read, shape [batch, state, length].
        d: Learned direct skip, shape [channels].
        z: Optional signed SiLU gate, with the same shape as u.
        delta_bias: Optional learned bias, shape [channels].
        delta_softplus: Whether to make step sizes positive with softplus.

    Returns:
        A tensor with u's shape. This uses exp(delta*A) retention and the
        delta*B*u input approximation, not the exact ZOH input integral.
    """
    if delta_bias is not None:
        delta = delta + delta_bias[None, :, None]
    if delta_softplus:
        delta = F.softplus(delta)
    state = u.new_zeros(u.shape[0], u.shape[1], a.shape[1])
    outputs = []
    for index in range(u.shape[-1]):
        step = delta[:, :, index, None]
        retention = torch.exp(step * a[None])
        write = step * b[:, None, :, index] * u[:, :, index, None]
        state = retention * state + write
        read = (state * c[:, None, :, index]).sum(dim=-1)
        outputs.append(read + d[None] * u[:, :, index])
    output = torch.stack(outputs, dim=-1)
    return output if z is None else output * F.silu(z)

5. Align the backward scan before combining

Original paper

Enlarge figure

Math and tensor shapes

Y_{bi}=\tfrac12(Y_f+\operatorname{flip}(Y_b))

Y=W_{out}(Y_{bi}\odot\operatorname{SiLU}(Z))

Flip the complete sequence, including CLS, before the backward convolution and scan; flip its outputs back afterward. The selected div2 release averages the aligned outputs. A shared SiLU gate is signed and unbounded, so it can suppress or reverse an output. Applying the aligned shared gate after the average is algebraically equivalent to the official direction-wise gating.

[b, 197, 192] → content/gate width 384 → [b, 197, 192]

Python implementation

python

def align_average_gate(forward, reverse_order, gate):
    """Align the reverse scan, average both directions and apply signed SiLU."""
    aligned = reverse_order.flip(-1)
    return (forward + aligned) * 0.5 * F.silu(gate)


class BidirectionalMixer(nn.Module):
    """Share the content/gate projection; keep the two directional SSMs separate."""

    def __init__(self, width, state_size=16):
        super().__init__()
        channels = 2 * width
        rank = math.ceil(width / 16)
        self.in_proj = nn.Linear(width, 2 * channels, bias=False)
        self.forward_direction = Direction(channels, state_size, rank)
        self.backward_direction = Direction(channels, state_size, rank)
        self.out_proj = nn.Linear(channels, width, bias=False)

    def forward(self, tokens):
        content, gate = self.in_proj(tokens).transpose(1, 2).chunk(2, dim=1)
        forward = self.forward_direction(content)
        backward = self.backward_direction(content.flip(-1))
        mixed = align_average_gate(forward, backward, gate)
        # Official code gates each direction before averaging. The shared
        # aligned gate makes this after-average form algebraically equivalent.
        return self.out_proj(mixed.transpose(1, 2))

6. Carry the residual alongside the mixer output

Original paper

Enlarge figure

Math and tensor shapes

R_l=R_{l-1}+H_{l-1}

H_l=\operatorname{BiMamba}(\operatorname{RMSNorm}(R_l))

At the first block, the residual starts as the embedded tokens. Later blocks first add the preceding mixer contribution. RMSNorm scales channels by their root mean square and does not subtract the mean. The mixer returns a new contribution. There is no separate Transformer feed-forward sublayer in this chosen Vim block. The teaching evaluation omits training DropPath.

Mixer contribution and residual: both [b, 197, 192]

Python implementation

python

class RMSNorm(nn.Module):
    """Normalize the root mean square across channels, without mean removal."""

    def __init__(self, width, eps=1e-5):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(width))
        self.eps = eps

    def forward(self, x):
        return (
            x
            * torch.rsqrt(x.square().mean(dim=-1, keepdim=True) + self.eps)
            * self.weight
        )


class VimBlock(nn.Module):
    """Carry residual separately, matching the released add/norm/mixer order."""

    def __init__(self, width, state_size):
        super().__init__()
        self.norm = RMSNorm(width)
        self.mixer = BidirectionalMixer(width, state_size)

    def forward(self, hidden, residual=None):
        residual = hidden if residual is None else residual + hidden
        contribution = self.mixer(self.norm(residual))
        return contribution, residual

7. Finish the residual addition and read the middle CLS

Original paper

Enlarge figure

Math and tensor shapes

F=\operatorname{RMSNorm}(R_{24}+H_{24})

\ell=W_{head}F[:,98]+b_{head}

The last mixer contribution still needs its residual addition before the final norm. The released factory name contains final_pool_mean, but the class-token branch takes precedence. Our tiny random-weight smoke classifier has five classes; the separate official pretrained exporter uses the real 1000-class head.

[b, 197, 192] → [b, 192] CLS features → [b, 1000] logits

Python implementation

python

def forward(self, image):
    patches = patch_tokens(image, self.projection)
    hidden, cls_index = assemble_tokens(
        patches, self.cls_token, self.positions
    )
    residual = None
    for block in self.blocks:
        hidden, residual = block(hidden, residual)
    # The last mixer contribution has not yet entered the carried residual.
    features = self.norm(hidden if residual is None else residual + hidden)
    cls_feature = features[:, cls_index]
    return self.head(cls_feature)  # CLS takes precedence over mean pooling.

8. Make overlapping patches a separate model choice

Original paper

Enlarge figure

Math and tensor shapes

P=(\lfloor(224-16)/s\rfloor+1)^2

s=16: L=197;\qquad s=8: L=730

Overlapping windows sample the same image more densely. They are not obtained by replacing 14 with 28 or by silently reusing the baseline predictions. Two scan directions visit 2L token positions; dense self-attention forms L² pair entries. This compares token mixing at fixed width and state size, not whole-model runtime or total training memory.

14 × 14 + CLS → 27 × 27 + CLS

Python implementation

python

def sequence_length(image_size, patch_size, stride):
    """Count actual overlapping windows, plus CLS; this does not resize weights."""
    windows_per_side = (image_size - patch_size) // stride + 1
    return windows_per_side**2 + 1


# 224px, kernel 16: stride 16 -> 197 tokens; stride 8 -> 730 tokens.
# Stride 8 needs separately fine-tuned weights and position embeddings.

The CPU teaching reference exposes the recurrence as a differentiable loop. Its small smoke model has random weights and checks finite forward/backward computation. It does not reproduce pretrained ImageNet results or the optimized GPU kernels. Download the runnable Python reference and its requirements, then run:

python vision_mamba.py --smoke --backward

The displayed excerpts and download come from the same source. The official release at the pinned revision supplies the complete model and fused kernels.

The mechanics reference shows what the operator computes. A released checkpoint lets us inspect what learned weights actually produce.

A recorded prediction from the released Tiny model

Choose a photograph below to see the actual 224×224 model crop, the recorded top five ImageNet probabilities, and the tensor stages from patch projection to the classifier. These outputs come from the released Vim-Tiny middle-CLS baseline, rather than the random-weight teaching model. The displayed probabilities use a fixed 0–100% scale; they describe this classifier's distribution over its 1,000 labels, not calibrated certainty about the scene.

One photograph through the released Vim-Tiny

Measured pretrained example

These are saved outputs from the official model on CPU, using PyTorch reference backends and the released checkpoint’s model weights. Select a photograph to inspect the actual 224 × 224 crop and its five highest ImageNet probabilities.

PhotographChelsea the catCoffee cupAstronaut Eileen Collins

Showing recorded Vim-T predictions for Chelsea the cat.

Vim-T (middle CLS, baseline)

7,148,008 parameters · state size 16 · input 224 × 224

Softmax probability · 0–100% scale

1. Egyptian cat46.35%
2. tabby19.55%
3. tiger cat12.92%
4. lynx0.37%
5. Persian cat0.31%

A distribution over the 1,000 ImageNet categories. These probabilities are not calibrated certainty. Three example photographs do not estimate ImageNet accuracy.

Recorded tensor stagePatch tokensPosition-aware tokensBlock 1 · mixer contributionBlock 1 · full residual featuresBlock 12 · mixer contributionBlock 12 · full residual featuresBlock 24 · mixer contributionBlock 24 · full residual featuresNormalized middle CLSClass logits

Middle CLS inserted at 98; learned absolute positions added

197 tokens × 192 channels

Position-aware tokens

Input · batch, tokens, channels; CLS/head are vectors

1 × 197 × 192

Output

1 × 197 × 192

Tensor shapes were traced during this model run. The colored cells show schematic geometry, not measured activations, attention, saliency, or object masks.

How these predictions were produced

The official middle-CLS bidirectional-v2 Tiny factory loaded the checkpoint strictly, with no missing or unexpected tensors. Evaluation and inference mode use float32 on CPU with 2 threads and the official PyTorch scan reference.

Resize the short edge to 256 using bicubic PIL interpolation, center-crop to 224, convert RGB to float32 in [0,1], then normalize with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. Every normalized crop matched the pinned official evaluation transform exactly.

The teaching scan and mixer also passed an independent output and gradient comparison against the pinned reference. The pretrained classifier here executes the official model directly; it is separate from the random-weight teaching program.

CPU backend substitutions

• mamba_inner_fn_no_out_proj -> unchanged mamba_inner_ref with identity out projection then BLD-to-BDL transpose
• selective_scan_fn -> unchanged official selective_scan_ref
• causal_conv1d_fn -> torch.nn.functional.conv1d with left padding, groups=channels, then SiLU
• rms_norm_fn -> unchanged official rms_norm_ref; residual_in_fp32 maps to upcast=True
• AST extraction avoids CUDA/Triton imports; original class/factory/function bodies and decorators are preserved
• register_model is an identity decorator to avoid global timm registry mutation; real timm initialization, DropPath and _cfg are supplied

Official code revision

dd0358ad1e42701f22afbefa0717cc8825cf9f45

Checkpoint SHA256 · selected key: model

fedb1e60267f3bc7b2a8443f2ddead8d39e019d10247a87252162b3789801fd5

Input crop SHA256

ff9de1b8e229782de3f5f15371421046ae5d01fd55b480c708c55edf555fd34f

Execution

PyTorch 2.12.0 · torchvision 0.27.0 · Pillow 12.2.0 · exported 2026-10-10

Full measurements and provenance · Reproduce the export · Pinned requirements

The tensor planes illustrate shape and stage, while the inspector reports measured tensor summaries. They are not spatial saliency maps. Mixer outputs and accumulated residual features are identified separately, including the final residual addition and normalization before CLS readout.

These three photographs are examples of inference, not an accuracy benchmark. The pinned checkpoint, strict loading, preprocessing, numerical reference comparison, and all 1,000 outputs are recorded in the full export manifest. The repository-scoped exporter reproduces that export using CPU reference backends in place of the official CUDA/Triton kernels. It preserves the bidirectional v2 computation; it does not benchmark the fused implementation.

Having inspected the baseline at one resolution, we can ask what changes when we give it more tokens from the same image.

More tokens from overlapping patches

Keep the image at 224×224 and each patch at 16×16, but reduce extraction stride from 16 to 8. Neighboring windows now share pixels. The projection is still applied to a full 16×16 RGB window; we have changed where those windows begin.

Let P be the patch width and s the stride. Without padding, each spatial dimension contributes \lfloor(224-P)/s\rfloor+1 windows. Stride 16 gives 14×14 = 196 patches; stride 8 gives 27×27 = 729 patches. Including CLS means 197 versus 730 tokens. The latter places middle CLS at index 364.

For sequence length L, two scans make 2L directional visits. A full pairwise attention matrix has L² entries per head. At fixed channel width and state size, extending 197 tokens to 730 increases these counts by about 3.71× and 13.73×, respectively. The diagram compares those mechanisms; projections, convolutions, kernel execution, and hardware also affect end-to-end time and memory.

This is a separately fine-tuned finer-granularity variant, not a free stride switch on the baseline checkpoint. More windows change the sequence geometry and position embedding length. The paper fine-tunes for 30 additional epochs with stride 8 while retaining patch size 16. Paper v3, §4.1, separate stride-eight Tiny factory

Those mechanics explain why longer sequences are a useful design direction. The published experiments tell us what was measured.

Published results and their limits

The table below reproduces Vim rows from paper v3 Table 1, rather than combining numbers from different releases. All rows use 224×224 evaluation on ImageNet-1K validation; † denotes the stride-eight long-sequence fine-tuning setting.

The paper trains baselines for 300 epochs with AdamW, batch size 1,024, a cosine learning-rate schedule, EMA, and image augmentations; validation uses center-cropped 224×224 inputs. The † models continue for 30 epochs with learning rate 10^{-5} and weight decay 10^{-8}. These are the authors' benchmark results, not measurements from this page. Table 1 and evaluation protocol

The pinned README lists 80.5% and 81.6% for Small and Small+, while Table 1 reports 80.3% and 81.4%. The values above consistently follow the paper. Rounded parameter counts describe the published variants; they are not an exact parameter audit of every checkpoint. Pinned model table

The paper reports 2.8× faster feature extraction and 86.8% lower GPU memory than DeiT-Ti at 1248×1248, in its batch-inference comparison. These figures belong to that model pair, resolution, and implementation. They do not imply every Vim workload is faster, nor that the inspectable CPU loop achieves the same memory behavior. A fixed-size recurrence still compresses information; scan order still changes path lengths; overlapping patches still require additional work. Figure 1 and §4.1

Vim and VMamba make different spatial choices

Vim and VMamba are concurrent approaches to visual selective state spaces. Their shared recurrence idea does not make their image traversal or backbone interchangeable:

Vim keeps a plain sequence throughout the backbone. VMamba builds a hierarchy of spatial feature maps. This comparison is about the selected released architectures, not an accuracy ranking across unmatched experiments. See the original Vim architecture and the final VMamba architecture.

Related reading

• Mamba: follow the input-dependent recurrence, complete causal block and persistent decode buffers before adapting the operator to images.
• Mamba-2: derive the scalar-transition SSD operator, its structured matrix form and chunked computation.
• Vision Transformer: follow the patch and CLS representation that Vim retains while changing the mixing operator.
• VMamba: follow four scan orders into a hierarchical visual backbone.
• Self-attention in ViT: compare direct token-pair mixing with a recurrent summary.
• Positional embeddings in ViT: understand why sequence slots need position information.
• Convolution operation: connect patch projection and local sequence convolution to kernel, stride, and weight sharing.
• Mamba paper: read the selective state-space mechanism and hardware-aware implementation that Vim builds upon.