MambaVision: A Hybrid Mamba-Transformer Vision Backbone

Which information should a vision backbone carry forward?

A photograph gives a network two different problems. Nearby pixels describe edges and texture. Distant regions help decide whether an edge belongs to a wheel, a window or the background. A useful backbone must preserve local evidence while building representations that connect larger parts of the image.

MambaVision assigns different operators to different resolutions: convolution early, then a mixture of state-space blocks and attention later. Its mixer also retains a separate convolutional path beside the state-space path. That makes it a useful architecture to study when asking where recurrence helps and where direct token interaction remains valuable. This review follows the released classification implementation, then separates that mechanism from the experiments used to evaluate it.

Ali Hatamizadeh and Jan Kautz first posted the paper in July 2024; the revised version is a CVPR 2025 paper. The header uses the conference year. The review date above is independent of both. Publication record, official repository.

All six interactive figures below are authored illustrations or deterministic calculations. None shows a checkpoint prediction, a learned attention map or a speed measurement. The versioned paper and pinned source at revision 7860a50 are the primary references.

Start with the hierarchy

Before inspecting a mixer, locate the sequence it receives. The released Tiny factory sets base width 80 and stage depths [1, 3, 8, 4]. Two stride-two convolutions form the stem. Each stage preserves its incoming spatial dimensions; an outgoing downsampler halves them and doubles channels before the next stage. These details give a concrete interpretation to the four boxes in an architecture diagram. Pinned stem, layers and Tiny factory.

At 224×224, calculate the first map as 56×56×80. The next maps are 28×28×160, 14×14×320 and 7×7×640. Spatial positions decrease by four at each transition while channel width doubles. This is a change in representation: the network spends early work on many narrow feature vectors and later work on fewer, wider vectors.

The first two stages use residual convolution blocks. In the last two, each block has a normalized mixer residual followed by a normalized MLP residual. Attention and MambaVision therefore occupy the same block role, with different token-mixing mechanisms. They are not two complete backbones whose final predictions are averaged. Released ConvBlock, Block and MambaVisionLayer.

This equation omits training-time stochastic depth and optional layer scales. It exposes two independent skip routes: the input survives the mixer, and the intermediate representation survives the MLP. A scan’s direct D·u term, which we will meet below, sits inside the mixer and is a third, different mechanism.

For an odd stage depth, the released constructor places the extra block in the MambaVision prefix. T2 has eleven blocks in stage 3: six mixers followed by five attention blocks. Small has seven: four followed by three. The overview’s “half and half” is a useful design description, but the integer boundary matters when instantiating a real model. Pinned block allocation.

With the stage sequence established, the next question is what makes the MambaVision mixer different from a standard gated Mamba mixer.

Two parallel paths, one recurrent state

The projected input splits into an x path and a z path. Each has its own ordinary depthwise one-dimensional convolution and SiLU activation. The x path then enters a selective scan. The z path bypasses that scan. Concatenating their outputs preserves both feature groups until the final channel projection combines them. Paper Algorithm 1 and Figure 3, released mixer.

“Symmetric” refers to the parallel branch layout. It does not mean forward and backward scans. The two convolutions have separate weights, and only one branch maintains a state. It also does not mean the z output multiplies the scan output: the official selective-scan call passes z=None, then concatenates the independently computed z features afterward.

There is a useful paper-to-code width distinction. Algorithm 1 projects to width C and divides that into C/2 paths. The released mixer defaults to expand=2, yielding inner width 2C and two paths of width C each. Its output projection maps 2C back to C. The figure lets you switch between these conventions without silently treating their parameter counts as identical. Equation 7’s printed output projection also needs to be read in light of concatenation: concatenating two C/2 paths produces C input channels for that projection. Algorithm 1 and Eq. 7, released projection dimensions.

Why preserve a separate path? Consider a state that compresses a long prefix into a small vector. Some distinctions in the individual inputs can disappear in that compression. A separate feature path gives the output projection another representation to use. This is a structural opportunity, not a guarantee that a particular object detail survives. Establishing that would require a trained-model intervention or measurement.

Before following that compressed state, examine the local operation that feeds it. Its padding changes which inputs can affect a token even before recurrence begins.

Noncausal convolution changes local access

In an autoregressive language model, a token must not read future inputs. A fixed image does not impose that prediction constraint. Ordinary same-padded convolution can therefore include neighbors on both sides of a token in the flattened sequence.

The impulse makes this visible. Under a three-tap regular filter, an output immediately before the impulse can already depend on it. Under a causal filter it cannot. With the released default four-tap kernel, same padding uses offsets −1, 0, +1 and +2. The asymmetry is a consequence of the even kernel width and the padding convention; it does not imply a reverse recurrent scan. Pinned convolution defaults and forward path.

A second distinction matters for images: this is Conv1d over the partitioned token sequence, not a new 3×3 spatial filter. Adjacent flattened indices can straddle a row boundary. The earlier two-dimensional convolutions and downsamplers provide spatial locality with different geometry. Keeping the two meanings of “local” separate prevents a misleading receptive-field diagram.

Ordinary convolution relaxes causality in the local preprocessing. It does not erase the scan’s ordering: after convolution, the state still updates one sequence position after another.

Selectivity controls retention and writing

A scalar state is enough to inspect the arithmetic. Let A be negative so the exponential retention factor lies between zero and one for a nonnegative effective step size. Each update retains some old state and adds a new input contribution. The readout uses C, while D supplies a direct path from the current input.

This is the selective kernel convention used by the teaching reference, with a ΔBu injection. The paper also introduces exact zero-order-hold discretization of a continuous system. Its general integrated B term is not numerically identical to ΔB. The reference deliberately does not claim that equivalence. The official mixer delegates recurrence to mamba_ssm’s selective scan. Official call and parameter preparation.

In the real mixer, Δ, B and C come from the convolved x features, rather than being fixed scalars. A is parameterized through a negative exponential, and the scan applies softplus to the effective delta input. The figure freezes B and C at one and lets you supply Δ directly so that retention and writing remain inspectable. A larger Δ simultaneously changes the retained fraction and the new contribution; calling it only a “forget gate” would hide half the update.

The freeze option uses Δ=0 after the first write. This is an analytic boundary: the state stays constant and new inputs add nothing. Finite softplus inputs produce strictly positive values, so the option is a mathematical limit rather than a claim about an exactly observed checkpoint state.

For an impulse followed by zeros and constant Δ, the state has a closed form:

This identity is one of the numeric tests. It checks that the implementation retains state through the expected number of updates, including the first step. The bars illustrate a computed decay, not a measured effective receptive field. A trained model may have many state components with different dynamics and input-dependent writes.

Recurrence can carry information through the sequence it is given. To understand the scope of that sequence, return to the layer wrapper around both mixer types.

Attention placement and the boundary of global context

The released layer pads its feature map, partitions it into windows, runs all blocks in that stage on those window sequences, reverses the partition and crops the padding. Both MambaVision and attention blocks therefore operate inside the same partition. At 224×224 input, stage 3’s 14×14 map fits one 14×14 window, and stage 4’s 7×7 map fits one 7×7 window. At larger input sizes, a stage can contain multiple windows. Pinned MambaVisionLayer.forward.

Within a window, attention directly combines values using query–key compatibility weights:

Here d_{\rm head} means the dimension of one head, which is the scale used by the released code. This notation avoids confusing head width with the number of heads in the paper’s accompanying prose. The softmax example in the figure is a separately authored four-key calculation; it is not an extracted attention map. Official attention implementation.

For window side w, one head computes w⁴ pairs per window. A 14×14 map in one window has 38,416 pairs. A 28×28 map with four such windows has 153,664 pairs, while full-map dense attention would have 614,656. These are derived pair counts. They exclude channel projection, value aggregation costs, MLPs, data movement and all other blocks, so they cannot establish a latency advantage.

The padding case deserves equal care. A 225×225 input yields a 15×15 stage-3 map, padded to 28×28 for 14×14 windows. That is 559 padded positions. The official attention call supplies no explicit padding mask. Cropping afterward removes padded output locations; it does not mean padded tokens had no influence on retained outputs. The figure counts the padded work rather than pretending only real pixels participated.

Late attention supplies direct token interaction within its window after the mixer prefix. It does not instantly create all-to-all communication across every window of a large image. Convolution and downsampling connect spatial neighborhoods elsewhere in the hierarchy; repeated unshifted windows alone should not be drawn as a full-map global graph.

Read the mechanism as executable arithmetic

The five rows below connect source references, equations, tensor shapes and code extracted from one canonical Python file. The reference uses only the standard library. It checks mechanics on small deterministic inputs, with fixed scalar weights where a learned projection would otherwise obscure the operation.

1. Resolve shapes and allocate the late blocks

Original paper

Enlarge figure

Math and tensor shapes

H_i=\lceil H/2^{i+2}\rceil,\quad C_i=2^iC_0,\quad n_M=\lceil n/2\rceil

224×224 → 56×56×80 → 28×28×160 → 14×14×320 → 7×7×640

These are pre-downsample Tiny maps, with depths [1,3,8,4]. M denotes mixer, A denotes attention and C denotes convolution. Stage downsampling occurs after its blocks in this code. The reference omits convolution weights and normalization, and computes shape arithmetic only.

Pinned upstream implementation · 7860a50

Python implementation

python

def hierarchy(height, width, dim=80, depths=(1, 3, 8, 4)):
    """Returns pre-downsample stage shapes and released block ordering."""
    if min(height, width, dim, *depths) < 1 or len(depths) != 4:
        raise ValueError("Positive dimensions and four stage depths required")
    height, width = (height + 3) // 4, (width + 3) // 4
    stages = []
    for stage, depth in enumerate(depths):
        split = (depth + 1) // 2
        blocks = "C" * depth if stage < 2 else "M" * split + "A" * (depth - split)
        stages.append((height, width, dim * 2 ** stage, blocks))
        height, width = (height + 1) // 2, (width + 1) // 2
    return stages

2. Give a token access to neighboring future inputs

Original paper

Enlarge figure

Math and tensor shapes

v_t=\sum_{j=0}^{k-1}w_j u_{t+j-\lfloor(k-1)/2\rfloor}

One scalar channel [L] → [L]; real model applies one filter per channel

Out-of-range inputs are zero. This is cross-correlation, matching neural-network Conv1d conventions. For k=4, the offsets are −1,0,+1,+2. A row-major sequence makes the end of one row adjacent to the start of the next. The causal switch is a comparison baseline, not a second MambaVision branch.

Pinned upstream implementation · 7860a50

Python implementation

python

def same_conv(sequence, kernel, causal=False):
    """Computes zero-padded cross-correlation, as Conv1d does.

    Args:
        sequence: Scalar token sequence.
        kernel: Ordered tap weights; even sizes pad more on the right.
        causal: Whether to place all padding on the left instead.

    Returns:
        A list with the same length as the input.
    """
    if not sequence or not kernel:
        raise ValueError("Nonempty input and kernel required")
    left = len(kernel) - 1 if causal else (len(kernel) - 1) // 2
    return [sum(weight * sequence[t + k - left]
                for k, weight in enumerate(kernel)
                if 0 <= t + k - left < len(sequence))
            for t in range(len(sequence))]

3. Separate exponential retention from a new write

Original paper

Enlarge figure

Math and tensor shapes

h_t=e^{\Delta_t A}h_{t-1}+\Delta_t B_tu_t,\qquad y_t=C_th_t+Du_t

One channel, one state, supplied Δ[L] → scalar outputs [L]

The real kernel has multiple states, learned A and D, and input-dependent Δ, B and C. Here B and C are held fixed. No learned projection or softplus is simulated. In particular ΔBu is not the exact zero-order-hold integral for a generic continuous B. Supplying an effective positive Δ keeps the numerical scope explicit.

Pinned upstream implementation · 7860a50

Python implementation

python

def selective_scan(inputs, deltas, a=-1.0, b=1.0, c=1.0, direct=0.0):
    """Runs one channel, one state with supplied positive effective deltas.

    Uses the selective kernel's delta*B*u injection, not exact ZOH B.
    Learned delta projection/softplus and input-dependent B/C are omitted.
    """
    if len(inputs) != len(deltas) or a > 0:
        raise ValueError("Matching sequence lengths and nonpositive A required")
    state, outputs = 0.0, []
    for value, delta in zip(inputs, deltas):
        if not math.isfinite(delta) or delta < 0:
            raise ValueError("Finite nonnegative deltas required")
        state = math.exp(delta * a) * state + delta * b * value
        outputs.append(c * state + direct * value)
    return outputs

4. Concatenate two features before mixing channels

Original paper

Enlarge figure

Math and tensor shapes

y=W_o[\operatorname{Scan}(\operatorname{SiLU}(K_x*x));\operatorname{SiLU}(K_z*z)]

Released default C → 2C → (C,C) → 2C → C

This runnable scalar example uses fixed distinct filters and a fixed 2-to-1 output projection. It demonstrates concatenation followed by linear combination, without implementing the learned channel projections, residual MLP or training. The expansion argument records the paper/code width distinction; it does not change this scalar example’s channel count. The full block separately adds mixer and MLP residuals.

Pinned upstream implementation · 7860a50

Python implementation

python

def mixer(sequence, expansion=2):
    """Illustrates two branches with fixed scalar weights and a 2-to-1 output.

    Expansion documents width accounting only. This scalar demo is not a
    learned channel projection or an implementation of the full MLP block.
    """
    if expansion not in (1, 2):
        raise ValueError("Choose paper width 1 or released default 2")
    def silu(value):
        return value / (1 + math.exp(-value))
    x = [silu(v) for v in same_conv(sequence, (0.2, 0.6, 0.2))]
    z = [silu(v) for v in same_conv(sequence, (-0.1, 0.8, 0.3))]
    scanned = selective_scan(x, [0.5] * len(x))
    concatenated = list(zip(scanned, z))
    return [0.7 * state + 0.3 * local for state, local in concatenated]

5. Partition a map before either late-stage mixer

Original paper

Enlarge figure

Math and tensor shapes

N_w=\lceil H/w\rceil\lceil W/w\rceil,\quad P=N_ww^4,\quad L_{scan}=N_ww^2

3×5 real grid, w=2 → six windows, nine padding cells

The IDs preserve row-major position within each window. None marks padded cells; the actual network uses zeros. They are not an attention mask. P counts query–key pairs for one attention head in one block. Window reversal restores the padded grid, then crops to the original size before downsampling. This reference enumerates geometry, not the attention or scan output.

Pinned upstream implementation · 7860a50

Python implementation

python

def window_layout(height, width, size):
    """Partitions row-major IDs, retaining None for right/bottom padding."""
    if min(height, width, size) < 1:
        raise ValueError("Positive sizes required")
    hp = math.ceil(height / size) * size
    wp = math.ceil(width / size) * size
    windows = []
    for top in range(0, hp, size):
        for left in range(0, wp, size):
            windows.append([
                row * width + column if row < height and column < width else None
                for row in range(top, top + size)
                for column in range(left, left + size)])
    return windows

Original-paper crops: Ali Hatamizadeh and Jan Kautz, MambaVision, arXiv v2 (2025). Cropped and rasterized under CC BY-SA 4.0. The paper illustrations and released-code details are distinguished above.

Run python examples/mambavision.py in a checkout, or use the downloadable reference, requirements note and scope README. Running the file executes its assertions and prints JSON. It does not train a backbone, compute gradients, fetch a checkpoint or reproduce the paper’s accuracy. Source-region synchronization keeps the displayed excerpts aligned with that runnable file.

These checks establish useful invariants: every real grid position appears in exactly one window; zero-delta updates preserve state; the impulse response matches its closed form; concatenated branch widths match the output projection’s input; and odd stage depths keep the intended block order. None of these invariants substitutes for evaluating learned representations.

What changes for dense prediction?

Classification reduces the final map to a vector and produces class scores. A location-sensitive task must retain spatial maps and attach a separately trained head. The hierarchy makes this interface natural: early maps offer more locations, later maps offer coarser representations.

The task diagram distinguishes output structure. Classification answers which category is present. Instance segmentation associates a separate mask with each detected instance. Semantic segmentation assigns categories to pixels without requiring an identity for every instance. The colored cells are authored regions, not network predictions.

Published evaluations attach Cascade Mask R-CNN for COCO and UPerNet for ADE20K. These are separate systems and fine-tuning protocols. A classification checkpoint is a starting backbone, not a ready-made detector or segmenter. Official detection implementation and configs, official semantic segmentation implementation.

Published evidence and what it can support

The selected values below come from the revised paper’s Tables 1–3. They are cited results, independent of the toy Python program and browser figures.

Source: Table 1. These use 224×224 ImageNet-1K evaluation; throughput uses batch 128 on an A100. Appendix I describes 300-epoch classification training with LAMB. These batched rates do not establish single-image latency, data-loading time or performance on another accelerator. The table does not provide enough measurement-boundary and precision detail to reconstruct a deployment benchmark from the values alone.

Sources: COCO Table 2, ADE20K Table 3. COCO uses Cascade Mask R-CNN, a 3× schedule and 1280×800 crops; ADE20K uses UPerNet and 512×512 crops. The detection and segmentation systems include their heads: their parameters and FLOPs must not be equated with the classification backbone column above. Appendix G’s Mask R-CNN 1× window ablation is another protocol and has different AP values.

One controlled experiment is especially relevant to the architecture argument: the reported final-half attention pattern reaches 82.3% top-1, versus 81.5% for first-half attention. This supports operator placement within that experiment. It does not isolate every possible cause of the full model’s comparison against other backbone families. Hybrid-pattern ablation.

For a practical reading of those results, keep three boundaries visible:

• Architecture versus implementation. A faster full model can benefit from early convolutions, channel widths, fewer sequence rearrangements, attention kernels and different depths. The complete system’s throughput is not a measurement of the scan alone.
• Training regime versus model name. ImageNet-21K pretraining, higher-resolution fine-tuning and larger variants change the experimental question. Their scores should not silently replace a 224×224 ImageNet-1K baseline.
• Connectivity versus evidence of use. A graph can permit a distant influence without establishing that the learned network uses it reliably. A qualitative attention visualization also does not establish robustness, calibration or causality.

The useful design lesson is a division of labor. A spatial hierarchy reduces the number of positions before more expensive interactions. A recurrent path compresses ordered context, a separate path retains local features, and later attention gives tokens direct interaction within the current window. Understanding those boundaries makes the reported tradeoffs interpretable and suggests concrete experiments: move the attention boundary, change window size under a fixed recipe, or measure kernels under the actual target workload.

Related reading

• Mamba develops selectivity and the state-space recurrence underlying the x path.
• VMamba uses multiple spatial scan orders, offering a different response to image ordering.
• Vision Mamba explores bidirectional state-space modeling for visual sequences.
• Self-attention in Vision Transformers explains the query–key–value operation used in late blocks.
• Convolution operation connects local receptive fields, filters and stride to the early hierarchy.
• Feature pyramid networks explains why dense heads consume multiple spatial scales.
• Versioned paper and pinned official code provide the architecture and experimental definitions behind this review.