EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba

A visual state-space model can carry information across an image without constructing a full attention matrix. But linear work still has a constant: VMamba’s cross-scan sends the same feature map through four complete traversals. EfficientVMamba asks whether those traversals need to revisit every position.

Its answer is atrous selective scanning. Partition the spatial grid into four interleaved subsets, scan each subset once, then put the outputs back at their original coordinates. The subsets span the image while each sequence is shorter. A parallel convolution branch supplies nearby interactions that the partition separates. The resulting network spends state-space computation early, at high spatial resolution, and uses inverted residual convolution blocks later. Paper v1, §§4.1–4.3

This review follows the March 15, 2024 preprint by Xiaohuan Pei, Tao Huang and Chang Xu, which the official repository cites as an arXiv article. The date above is the write-up date. Implementation details refer to official code revision 0bc5ee2. Figures below contain authored diagrams and calculated teaching values; benchmark tables contain cited paper results. There are no recorded checkpoint predictions or locally measured GPU timings here.

Four shorter sequences, one dense output

Start with a feature map, not raw RGB pixels. Each cell contains a channel vector produced by earlier layers. With skipping step p = 2, its row and column parities define four disjoint groups: even/even, odd/even, even/odd and odd/odd. Every real position belongs to one group. This is a partition rather than dropping three quarters of the output pixels.

A 4×6 map contains 24 positions. Full SS2D builds four sequences of length 24. EfficientVMamba builds four sequences of length 6, covering the same 24 positions once in total. “Atrous” describes the gaps in spatial sampling: consecutive scan elements can be two columns or two rows apart in the original map. The state update itself still processes consecutive sequence indices. Paper v1, Figure 2 and Eq. 5

The official gather order makes an important detail concrete. Group 0 is even-row/even-column in row-major order. Group 1 is odd-row/even-column in column-major order. Group 2 is even-row/odd-column in row-major order. Group 3 is odd-row/odd-column in column-major order. The two column streams come from transposing the spatial axes before slicing. There are no backward reversals in this released gather. Reusing VMamba’s full forward/reverse diagram would teach the wrong routing. Pinned EfficientScan.forward

The paper writes a general skipping step p, but the released four-offset gather/scatter is safely interpreted here at p = 2. Four offsets cover all residue classes only for that setting. A naive change to p = 3 would leave spatial classes unassigned. The explorers deliberately expose shapes and traversal positions without pretending that an arbitrary dilation slider is a supported complete implementation.

Odd spatial sizes require another distinction. The code pads the right and bottom to a multiple of two before gathering. A 3×5 input becomes a 4×6 working grid; all four sequences have length 6, with 9 padded positions across them. Those positions are zero-valued inputs to the scan, not absent time steps. The eventual crop removes their output locations, but a state can still decay while traversing padding before a later valid token.

These streams contain features in different orders. To recover an image, their outputs must return to the right coordinates.

Merge restores coordinates instead of adding directions

In full VMamba SS2D, each pixel has an output from every direction, and aligned directional outputs are summed. In the p = 2 EfficientVMamba gather, a pixel appears in one stream. EfficientMerge reverses the routing: reshape, undo the column ordering where needed, write into the corresponding spatial slice, and crop padding. It does not add four outputs at each pixel. Pinned EfficientMerge.forward

Let \pi_k(t) name the spatial coordinate visited by token t of group k. For real positions, the merge is simply:

The inverse can be checked without learning anything: replace the sequence operator with the identity. Gather followed by merge must return the original map exactly. It must not return four times the map. Rectangular tests are particularly useful because swapping height and width in a column-order reshape can hide behind square examples.

Restoring a dense output does not restore the connectivity of four full-image scans. A state in the even/even group cannot directly read an odd/odd feature. The routing saves work partly by giving up redundant directional coverage and cross-group recurrence within that operator. To see what remains, follow one of the shorter state sequences.

What the selective state carries

Each group has an independent state trajectory. For a single scalar state, the update used in our readable reference is:

Here u is the input feature, A controls decay, Δ is a positive time scale, B writes information into the state, C reads it out, and D provides a direct input term. Production code applies channel-specific transitions and a state dimension, with token-derived Δ, B and C; the scalar explorer isolates the recurrence by supplying coefficients directly. The exponential transition is distinct from the approximate input update ΔBu. Exact zero-order-hold integration would use a different input coefficient. Paper v1, §§3.1–3.2, pinned scan wrapper

Increasing Δ simultaneously changes how quickly the old state decays and how strongly the new input is written. It is not merely a probability of keeping the old feature. A trained model derives these updates from the input, which is what makes the operator selective. A fixed decay slider demonstrates the arithmetic without claiming to display that learned selection.

A sparse group can still span the image, but each token reads a prefix in its own traversal order. “Global” therefore means a sequence can connect spatially distant positions, not that every output has a direct bidirectional relationship with every pixel. Nor does a possible path establish that a trained model assigns meaningful weight to all distant content. Compression into finite state, decay and routing bias all remain.

The missing nearby interactions motivate the next branch.

Local convolution fills a different role

Two adjacent pixels can belong to different residue groups. A 3×3 convolution can combine them immediately, even when one ES2D recurrence cannot. The paper’s Efficient Visual State Space block, or EVSS, therefore combines a state-space branch with a stride-one local convolution branch and channel recalibration:

Each branch retains the spatial shape. Squeeze-and-excitation uses a global summary to gate channels, allowing the branch features to be recalibrated before addition. These gates are not attention weights over pixel pairs, and the equation does not impose a two-branch softmax. Paper v1, §4.2, Eq. 6

The impulse experiment makes the division of labor visible. Sparse scan spreads the impulse downstream within one residue class. The local stencil reaches adjacent classes. Its fixed average and adjustable scalar fusion gates are authored surrogates, not the released learned convolution or SE weights. They demonstrate complementary connectivity; they do not measure an accuracy improvement.

The released implementation is more specific than the paper’s compact equation. Its convolution branch is 3×3 depthwise convolution → BatchNorm → GELU → 1×1 convolution. The normalized input supplies both branches. A shared BiAttn module is called separately on the scan and convolution outputs; it normalizes features, averages spatial positions, applies a reduced channel projection and GELU, then a sigmoid channel gate. The two recalibrated outputs are added, followed by an input residual and, when configured, an MLP residual. Thus the reference code shares recalibration parameters while applying them to different branch features. Pinned BiAttn and VSSBlock

Local mixing and the rest of the block can bridge residue groups. This supports a network with wider contextual influence than an isolated sparse scan. It does not make that sparse scan algebraically equivalent to four complete SS2D traversals.

Invert the usual local-then-global allocation

Many efficient attention backbones put convolutions early and global operators late, after reducing spatial resolution. EfficientVMamba uses the reverse allocation: EVSS in stages 1 and 2, inverted residual blocks in stages 3 and 4. Linear scan work makes early contextual mixing feasible without a full attention matrix; later local blocks process features that already contain context. Paper v1, §4.3

At 224×224 input resolution, the four stage scales are 56×56, 28×28, 14×14 and 7×7. Tiny uses channel widths [48, 96, 192, 384]; Small and Base use [96, 192, 384, 768]. The released EfficientVSSM uses a kernel-four, stride-four patch projection before the stages, followed by hierarchical downsampling. The late inverted residual blocks use expansion ratio 4 and SE ratio 0.125. The classifier normalizes, globally averages and projects to class logits. Paper v1, Table 1, pinned backbone builder

Table 6 compares three Tiny allocations: EVSS everywhere gives 75.1%, local-first gives 75.6%, and EVSS-first gives 76.5% ImageNet top-1. All have 0.8G rounded FLOPs, but the final design has 6M parameters versus 5M for the other two. This supports the chosen allocation under the reported configurations; it does not isolate stage order at exactly equal parameter capacity. Paper v1, Table 6

With routing, state updates, fusion and stage allocation defined, the implementation can now be read as explicit operations rather than a box labeled “efficient.”

Paper, equations and executable Python

The five rows below connect the original mechanism to tensor shapes and the canonical, deterministic reference. The Python uses only the standard library. It implements the released p = 2 spatial ordering and a scalar recurrence, then uses a local mean as a deliberately simplified convolution branch. It omits training, learned projections, channel normalization, gradients and optimized GPU execution.

1. Gather spatial residues into ordered streams

Original paper

Enlarge figure

Math and tensor shapes

O_k[t]=X[2r_t+m_k,2c_t+n_k]

[H,W] → four lists of ceil(H/2) × ceil(W/2)

Groups 0 and 2 are row-major; groups 1 and 3 are column-major. Offsets in original row/column coordinates are (0,0), (1,0), (0,1), (1,1). Right and bottom padding supply zeros. The helper supports odd rectangles without quietly dropping border pixels.

Pinned official implementation

Python implementation

python

def routes(height, width):
    """Return padded coordinates in the four released p=2 traversal orders."""
    if any(not isinstance(v, int) or isinstance(v, bool) or v <= 0
           for v in (height, width)):
        raise ValueError("Spatial dimensions must be positive integers")
    rows, cols = (height + 1) // 2, (width + 1) // 2
    paths = []
    for group, (dy, dx) in enumerate(((0, 0), (1, 0), (0, 1), (1, 1))):
        path = []
        outer, inner = (cols, rows) if group % 2 else (rows, cols)
        for a in range(outer):
            for b in range(inner):
                row, col = (b, a) if group % 2 else (a, b)
                path.append((2 * row + dy, 2 * col + dx))
        paths.append(path)
    return paths


def gather(values, height, width):
    """Gather valid features, inserting zero for right/bottom padding."""
    if len(values) != height * width:
        raise ValueError("Input shape mismatch")
    return [[values[y * width + x] if y < height and x < width else 0.0
             for y, x in path] for path in routes(height, width)]

2. Update an independent state inside each group

Original paper

Enlarge figure

Math and tensor shapes

h_t=e^{\Delta_t A}h_{t-1}+\Delta_t B_tu_t,\qquad y_t=C_th_t+D u_t

one scalar state per illustrative stream; production: channel × state dimension

This loop accepts a per-token Δ list and scalar B/C coefficients. Learned input projections, multiple channels and optimized kernels are omitted. Its ΔBu write term follows the implementation convention, rather than claiming exact zero-order-hold integration. Reset the state for each group. Padding is a real recurrence step and can affect later valid outputs in a column-major stream.

Pinned official implementation

Python implementation

python

def selective_scan(values, delta=0.5, a=-1.0, b=1.0, c=1.0, d=0.0):
    """Compute a scalar selective recurrence with supplied coefficients.

    Delta may be a per-token list. Learned projections are intentionally absent.
    The input update uses delta * B * u, as in the released scan kernel family,
    rather than the exact zero-order-hold input integral.
    """
    steps = delta if isinstance(delta, (list, tuple)) else [delta] * len(values)
    if len(steps) != len(values) or a > 0:
        raise ValueError("Invalid time scales or unstable A")
    if not all(math.isfinite(v) for v in [a, b, c, d, *values, *steps]):
        raise ValueError("All values must be finite")
    if any(v < 0 for v in steps):
        raise ValueError("Time scales must be nonnegative")
    state, outputs = 0.0, []
    for value, dt in zip(values, steps):
        state = math.exp(dt * a) * state + dt * b * value
        outputs.append(c * state + d * value)
    return outputs

3. Invert the routing, then crop the padding

Original paper

Enlarge figure

Math and tensor shapes

Y[2r_t+m_k,2c_t+n_k]=\widetilde O_k[t]

four padded streams → [H,W]

Every valid coordinate has exactly one owner at p=2. Scatter is therefore an inverse permutation plus a crop, not the four-direction summation used in full SS2D. An identity sequence operator gives Y=X. A wrong column reshape can pass square examples; our tests use rectangular and singleton maps.

Pinned official implementation

Python implementation

python

def merge(streams, height, width):
    """Scatter one output per coordinate and crop padding; do not sum streams."""
    paths = routes(height, width)
    if len(streams) != 4 or any(len(s) != len(p)
                               for s, p in zip(streams, paths)):
        raise ValueError("Expected four padded streams")
    output = [0.0] * (height * width)
    for stream, path in zip(streams, paths):
        for value, (y, x) in zip(stream, path):
            if y < height and x < width:
                output[y * width + x] = value
    return output

4. Add complementary global and local features

Original paper

Enlarge figure

Math and tensor shapes

F=\operatorname{SE}(\operatorname{ES2D}(X))+\operatorname{SE}(\operatorname{Conv}(X))

two shape-preserving maps → one map

The executable surrogate combines a scalar scan and a zero-padded local mean with supplied gates. It does not implement learned SE, BatchNorm, residual normalization or the MLP. The pinned code uses a shared BiAttn module on each branch, followed by an input residual and an optional MLP residual. Its local branch is depthwise 3×3, BatchNorm, GELU, then pointwise 1×1.

Pinned official implementation

Python implementation

python

def local_mean(values, height, width):
    """Apply a zero-padded 3x3 mean, standing in for learned local mixing."""
    routes(height, width)
    if len(values) != height * width:
        raise ValueError("Input shape mismatch")
    output = []
    for y in range(height):
        for x in range(width):
            neighbors = [values[(y + dy) * width + x + dx]
                         for dy in (-1, 0, 1) for dx in (-1, 0, 1)
                         if 0 <= y + dy < height and 0 <= x + dx < width]
            output.append(sum(neighbors) / 9.0)
    return output


def illustrative_fusion(values, height, width, scan_gate=0.5, local_gate=0.5):
    """Combine branches with supplied scalar gates, without learned SE or MLP."""
    scanned = [selective_scan(s) for s in gather(values, height, width)]
    global_values = merge(scanned, height, width)
    local_values = local_mean(values, height, width)
    return [scan_gate * g + local_gate * loc
            for g, loc in zip(global_values, local_values)]

5. Count scan positions without inventing speedups

Original paper

Enlarge figure

Math and tensor shapes

V_{\rm full}=4HW,\qquad V_{\rm sparse}=4\lceil H/2\rceil\lceil W/2\rceil

4 × 6: 96 → 24 visits; 3 × 5: 60 → 24 visits

This counts recurrence positions at fixed channel/state sizes, not total model FLOPs. Padding weakens the ideal fourfold reduction on small odd grids. Gather/scatter, projections, channel recalibration, local branches, MLPs and downstream blocks still cost work. A measured latency claim needs hardware, precision, batch size and timing boundaries.

Pinned official implementation

Python implementation

python

def scan_work(height, width):
    """Count visits, including padding, not FLOPs or measured latency."""
    paths = routes(height, width)
    dense = 4 * height * width
    sparse = sum(len(path) for path in paths)
    return {"dense": dense, "sparse": sparse,
            "ratio": dense / sparse, "padding": sparse - height * width}

Original paper crops: Xiaohuan Pei, Tao Huang, Chang Xu. v1 source · License. Cropped/rasterized from the original PDF; colors unchanged.

Run python examples/efficientvmamba.py in the repository checkout. The script checks rectangular and odd-size gather/scatter identities, analytic impulse decay, cross-group locality and padded visit counts. Download the reference and the comment-only requirements file from the synchronized example mirrors. The canonical source is examples/efficientvmamba.py; excerpt regions and downloads are owned by the central synchronization workflow.

Correctness of a scalar teaching reference establishes neither pretrained accuracy nor deployment speed. The next comparison keeps those evidence boundaries explicit.

What gets cheaper, and what does not follow

For a feature map of height H and width W, full four-direction scanning visits 4HW positions. Four sparse streams at p = 2 visit:

When both dimensions are even, that becomes HW, a fourfold reduction in scan positions. For a 3×5 grid it is 24 visits versus 60, a 2.5× reduction. For a 1×1 map the four padded streams remove the apparent visit advantage entirely. Channel width and state size are held fixed in this count. Paper v1, §4.1, pinned padding implementation

Both dense SS2D and ES2D remain linear in spatial token count at fixed channel/state dimensions. This is a constant-factor reduction in the sequence component, not a change from quadratic to linear complexity. Dense self-attention is a separate quadratic comparison; windowed or sparse attention has different scaling.

Gather/scatter memory movement, channel projections, convolution, recalibration, MLPs and later backbone stages remain. A fourfold visit reduction therefore cannot be read as a fourfold model FLOP reduction, training-memory reduction or wall-clock speedup. Shorter sequences can also change kernel utilization and launch overhead. The cited classification and dense-prediction tables do not provide a hardware/precision/timing protocol that would justify a local millisecond claim here.

Published accuracy and benchmark boundaries

The paper reports ImageNet-1K classification at 224×224, trained for 300 epochs with AdamW, batch size 1,024, an initial learning rate of 0.001, cosine decay and 20 warmup epochs. Its recipe includes augmentation, MixUp and EMA. Those training choices belong with the accuracy numbers. Paper v1, §5.1 and Table 2

Source: paper v1, Table 2. EfficientVMamba-S exceeds the listed Vim-Ti by 5.6 percentage points, while using more parameters and fewer reported FLOPs. EfficientVMamba-B uses fewer FLOPs than the listed VMamba-T but has more parameters and 0.4 points lower top-1. These are tradeoffs, not universal dominance. The VMamba values are the contemporary comparison in this 2024 paper; they should not be silently replaced by a later VMamba revision.

Dense prediction attaches different heads and changes the evaluation protocol:

Sources: Table 3, RetinaNet and Table 4, UperNet. RetinaNet uses the reported 1× detection schedule with ImageNet-pretrained backbones. The UperNet setup uses 160k fine-tuning iterations and batch size 16; SS and MS mean single-scale and multi-scale testing. Its table lists 512×512 image size while measuring FLOPs at 512×2048. Classification FLOPs and segmentation-system FLOPs therefore have different workloads and cannot be directly compared.

Table 4 also needs a careful reading: the prose in §5.3 says Tiny surpasses ResNet-50 despite listing 38.9 versus 42.1 SS mIoU, and says Base outperforms VMamba-S despite listing 46.5 versus 49.5. The table supports lower-cost operating points with an accuracy tradeoff, not those higher-accuracy claims. Likewise, Small’s 41.5 SS mIoU is below the table’s 43.8 for DeiT-S + MLN. The review follows the tabulated values and does not repeat the contradictory prose. Paper v1, Table 4 and §5.3

What the ablations establish

The Tiny progression in Table 5 is 72.1 → 73.6 → 75.1 → 76.5%: a resized VMamba baseline, ES2D, branch fusion, then inverted residual insertion. However, parameters change from 2.4M → 5M → 5M → 6M and rounded FLOPs from 0.9G → 0.8G → 0.8G → 0.8G. The paper explicitly resizes the baseline to approximately match compute. Its 72.1% result is therefore not the standard VMamba-T in Table 2. Paper v1, Table 5

The progression supports the combined design under the tested budgets. It does not isolate ES2D as the sole causal source of the full accuracy gain: width, depth, parameter count and block type also change. The local branch’s gain is consistent with the connectivity argument, but an impulse diagram is not empirical evidence that a trained model learned that exact behavior. The reported differences also lack a multi-seed uncertainty estimate in these tables.

EfficientVMamba’s useful contribution is a precise allocation of work. It preserves a dense spatial output by routing every real position into one sparse stream, accepts less redundant directional context, supplements that scan with local mixing, and places those global/local blocks early in the hierarchy. The mechanism explains where savings can come from. The benchmark protocols determine whether the resulting accuracy, memory and latency are appropriate for a particular deployment.

Related reading

• VMamba: the four full-image traversals and aligned summation that ES2D changes.
• Mamba: input-dependent state updates and the distinction between recurrence equations and efficient scan kernels.
• EfficientNet: channel recalibration and the interaction between depth, width and resolution.
• Convolution operation: local receptive fields and the stencil behind the branch comparison.
• Self-attention in Vision Transformers: direct pairwise interactions, distinct from ordered state compression.
• Original preprint and pinned official implementation: the primary sources for mechanisms and reported results.